DynamoDB Streams: Capturing Table Changes

• DynamoDB Streams captures changes to items in DynamoDB tables as a time-ordered sequence.
• Changes such as inserts, updates, and deletes are captured, allowing programmatic responses.
• Enabling DynamoDB Streams creates a log of events that facilitate triggering actions, syncing data, and updating search indexes.

Key Features of DynamoDB Streams

• DynamoDB Streams captures insertions, updates, and deletions made to a DynamoDB table.
• Primary key values, attributes changed, and old values (for updates and deletes) are captured in the stream.
• Events are ordered by the time they were applied to the table, enabling sequential processing.
• INSERT events indicate a new item was added.
• MODIFY events indicate an existing item was changed.
• REMOVE events indicate an item was deleted.
• Each event is stored as a stream record with metadata, accessible and processable by a stream consumer.
• DynamoDB Streams retains records for 24 hours after they are written before automatic deletion.
• DynamoDB Streams operates asynchronously without affecting DynamoDB table performance.

Enabling DynamoDB Streams

• DynamoDB Streams can be set up when creating a table or by modifying an existing one.
• Stream view types determine the level of detail captured in stream records.

Stream View Types

• KEYS_ONLY captures only the primary key attributes of the changed item which is useful for analytics or audits.
• NEW_IMAGE captures the entire item's new state after the operation which is useful for capturing fully updated data.
• OLD_IMAGE captures the entire item's previous state before the operation which is useful for "undo" features or auditing changes.
• NEW_AND_OLD_IMAGES captures the item's full state both before and after the change which is useful for synchronizing systems.
• To enable DynamoDB Streams for an existing table, use the following AWS CLI command: aws dynamodb update-table --table-name MyTable --stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES.

Reading from DynamoDB Streams

• The DynamoDB Streams API or AWS Lambda integration can be used to read stream records.

DynamoDB Streams API

• The GetShardIterator and GetRecords API calls allow direct interaction with the stream to fetch records.
• Streams are divided into shards that can be read sequentially.

AWS Lambda Integration

• AWS Lambda can be directly integrated with DynamoDB Streams to automatically trigger functions in response to table changes.
• Lambda functions can process stream records, update search indexes, or replicate changes to other systems.
• Lambda functions process records from the stream as they arrive and can be configured to process records in batches.
• To set up Lambda with DynamoDB Streams, associate a Lambda function to trigger upon table changes.
• A sample Lambda function triggered by DynamoDB Streams can process INSERT, MODIFY, and REMOVE events.

Stream Processing Best Practices

• Process stream records in batches for optimal performance.
• Implement error handling in Lambda functions to manage failure cases, using retries or dead-letter queues.
• Properly handle concurrency when reading from multiple shards, as DynamoDB Streams allows parallel processing.

Use Cases for DynamoDB Streams

• Enabling real-time analytics through change capture and processing for real-time reporting.
• Enabling data replication by syncing data between DynamoDB and other systems in real time.
• Enabling audit logging to track changes to tables for compliance/security monitoring.
• Automatically triggering actions, like notifications or updating other systems, upon specific changes in DynamoDB.
• Supporting event-driven architectures where table changes trigger workflows in other services such as Lambda or Step Functions.

Stream Record Example

• A stream record example for an INSERT event with the NEW_IMAGE stream view type includes details such as UserID, UserName, Email, and RegistrationDate.

DynamoDB Streams and Consistency

• DynamoDB Streams delivers records with eventual consistency.
• Stream records are durable within the 24-hour retention window, but are lost if not processed within that time.

Conclusion

• DynamoDB Streams enables real-time processing of DynamoDB table changes.
• Integration with AWS Lambda or the Streams API allows responding to changes in real time.
• The ability to capture new and old images of items, along with different event types, provides a flexible way to manage and track data changes.

DynamoDB Streams captures changes to DynamoDB items in a time-ordered sequence. These changes, including inserts, updates, and deletes, allow triggering actions and syncing data. Stream records retain data for 24 hours, enabling sequential processing of database events.