Streams & Global Tables
Change data capture for event-driven architectures and multi-region active-active replication with automatic conflict resolution.
Table of Contents
What DynamoDB Streams Are
DynamoDB Streams is an ordered log of all item-level changes in a table. Every INSERT, MODIFY, or REMOVE generates a stream record that downstream consumers can process.
The Security Camera Analogy
DynamoDB Streams is like a security camera recording every change to your table. It captures what happened (INSERT/MODIFY/REMOVE), when it happened, and what the item looked like before and after. The footage is kept for 24 hours. Multiple viewers (Lambda functions) can watch the same footage independently without affecting each other.
DynamoDB Streams Characteristics
- ✅Ordered log of all item-level changes (INSERT, MODIFY, REMOVE)
- ✅Records available for 24 hours (then automatically deleted)
- ✅Each stream record contains: event type, item keys, before/after images
- ✅Exactly-once delivery per shard — within a shard, records are ordered
- ✅Records within a partition key are ordered (cross-partition ordering not guaranteed)
- ✅Near real-time — records available within milliseconds of the write
Stream View Types
| View Type | What's Captured | Use Case |
|---|---|---|
| KEYS_ONLY | Only key attributes of modified item | Trigger processing that fetches fresh data |
| NEW_IMAGE | Entire item after modification | Sync to search index, cache warming |
| OLD_IMAGE | Entire item before modification | Audit logging, undo operations |
| NEW_AND_OLD_IMAGES | Both before and after | Diff detection, complex event processing |
Choose Wisely — Cannot Change Later
The stream view type is set when enabling streams on a table. Changing it requires disabling and re-enabling streams, which creates a new stream ARN. NEW_AND_OLD_IMAGES is the most flexible but generates the most data. Choose based on your consumer needs.
// Stream record structure (NEW_AND_OLD_IMAGES) { eventID: "abc123", eventName: "MODIFY", // INSERT | MODIFY | REMOVE dynamodb: { Keys: { PK: { S: "USER#123" }, SK: { S: "PROFILE" } }, OldImage: { PK: { S: "USER#123" }, name: { S: "Old Name" }, version: { N: "5" } }, NewImage: { PK: { S: "USER#123" }, name: { S: "New Name" }, version: { N: "6" } }, SequenceNumber: "111222333", SizeBytes: 256 }, eventSourceARN: "arn:aws:dynamodb:us-east-1:123456:table/Users/stream/..." }
Processing Streams with Lambda
The most common pattern: Lambda polls the stream automatically when configured as an event source trigger. No infrastructure to manage — fully serverless CDC.
Item Written
Application writes to DynamoDB table
Stream Record
DynamoDB creates stream record (within milliseconds)
Lambda Polls
Lambda service polls stream shards for new records
Batch Invocation
Lambda invoked with batch of records (configurable batch size)
Processing
Lambda processes records — sync to ES, invalidate cache, send notification
Lambda Trigger Configuration
| Setting | Purpose | Recommendation |
|---|---|---|
| BatchSize | Records per Lambda invocation | Start with 100, tune based on processing time |
| MaximumBatchingWindowInSeconds | Wait time to fill batch | 0 for low latency, 5-60 for throughput |
| BisectBatchOnFunctionError | Split failing batch to find poison pill | Always enable |
| MaximumRetryAttempts | Retries before sending to DLQ | 3-5 attempts |
| DestinationConfig.OnFailure | Dead letter queue for failed batches | Always configure (SQS or SNS) |
| FilterCriteria | Only invoke for matching events | Filter by eventName or item attributes |
Poison Pill Records
If one record in a batch causes Lambda to fail, the entire batch retries indefinitely (blocking the shard). Enable BisectBatchOnFunctionError to split the batch and isolate the failing record. Always configure a dead letter queue for unprocessable records.
Event-Driven Patterns
Common Stream Processing Patterns
| Pattern | How It Works | Example |
|---|---|---|
| Search sync | Stream → Lambda → Elasticsearch index | Near-real-time search of DynamoDB data |
| Cache invalidation | Stream → Lambda → delete Redis key | Invalidate cache when source data changes |
| Audit logging | Stream → Lambda → write to audit table | Immutable record of every change |
| Aggregation | Stream → Lambda → update counters/sums | Maintain materialized views |
| Notification | Stream → Lambda → SNS/SES | Send email when order status changes |
| Cross-region sync | Stream → Lambda → write to other region | Before Global Tables existed |
Outbox Pattern with DynamoDB
Problem: Dual-write — write to DynamoDB AND publish to SNS/EventBridge If DynamoDB write succeeds but SNS publish fails → inconsistency Solution: Transactional Outbox 1. Write business item + event record in same DynamoDB transaction 2. Stream processor reads event record from stream 3. Stream processor publishes to SNS/EventBridge 4. Guarantee: if item write succeeds, event WILL eventually be published TransactWriteItems([ { Put: { item: orderItem } }, // business data { Put: { item: orderCreatedEvent } } // outbox event ]) → Stream captures both → Lambda publishes event to EventBridge
DynamoDB as Event Store
DynamoDB as Event Store
- ✅Append-only writes — never update past events
- ✅PK = AGGREGATE#123, SK = EVENT#0000001 (sequence number)
- ✅Rebuild state by reading all events for an aggregate (Query)
- ✅Stream as the publish mechanism — new events stream to consumers
- ✅Snapshot optimization: periodically write current state to avoid full replay
Global Tables Overview
Global Tables replicate a DynamoDB table across multiple AWS regions. Active-active: reads and writes accepted in any region. Replication is asynchronous with sub-second lag typically.
Global Tables Characteristics
- ✅Active-active: reads and writes accepted in any region
- ✅Replication lag typically < 1 second (asynchronous)
- ✅DynamoDB Streams powers replication internally
- ✅Each region is fully independent — regional failure doesn't affect others
- ✅Conflict resolution: last writer wins (based on timestamp)
- ✅All replicas must use same capacity mode (provisioned or on-demand)
The Branch Office Analogy
Global Tables are like branch offices that each maintain their own copy of the company ledger. Any branch can accept new entries. Changes propagate to all other branches within seconds. If two branches modify the same entry simultaneously, the most recent change wins. If a branch goes offline, the others continue operating — when it comes back, it catches up automatically.
Write in us-east-1
Application writes item to US East region
Local Commit
Item committed locally, acknowledged to client
Stream Replication
DynamoDB Streams propagates change to other regions
Remote Apply
eu-west-1 and ap-southeast-1 receive and apply the change
Globally Consistent
All regions have the item (typically < 1 second total)
Conflict Resolution
When the same item is modified in multiple regions simultaneously, DynamoDB uses last-writer-wins based on the item's timestamp. There is no merge — the entire item is replaced by the winner.
No Merge, No Custom Resolution
DynamoDB does not support custom conflict resolution strategies. Last-writer-wins is the only option. If two regions update different attributes of the same item simultaneously, one update is lost entirely. Design your application to avoid concurrent writes to the same item from multiple regions.
Strategies to Avoid Conflicts
Strategies to Avoid Conflicts
- ✅Route writes to user's home region (user affinity)
- ✅Partition ownership: each region 'owns' specific partition keys
- ✅Append-only patterns: never update, only insert new items
- ✅Use conditional writes with version numbers (will fail in non-owner region)
- ✅Accept eventual consistency — design for idempotent operations
Global Tables Use Cases & Limitations
✅ Use Cases
- Low-latency reads globally (nearest region)
- Disaster recovery (automatic failover)
- Data locality compliance (write to specific region)
- Active-active multi-region applications
- Gaming leaderboards (regional + global)
⚠️ Limitations
- Eventually consistent only across regions
- Transactions cannot span regions
- TTL deletions replicate (may cause surprises)
- All replicas must use same capacity mode
- Last-writer-wins only (no custom conflict resolution)
Interview Questions
Q:What are DynamoDB Streams and how do they differ from Kinesis Data Streams?
A: DynamoDB Streams is a change data capture log specific to DynamoDB — 24-hour retention, ordered per partition key, integrated with Lambda triggers. Kinesis Data Streams is a general-purpose streaming service with 7-day retention (extendable to 365), multiple consumers via KCL, and higher throughput. You can enable Kinesis adapter on DynamoDB for longer retention and fan-out.
Q:How would you implement near-real-time search on DynamoDB data?
A: Enable DynamoDB Streams with NEW_IMAGE view type. Configure Lambda trigger that reads stream records and indexes them into Elasticsearch/OpenSearch. On INSERT/MODIFY: upsert document in ES. On REMOVE: delete document from ES. Result: search index stays within seconds of DynamoDB, no polling needed.
Q:How does conflict resolution work in Global Tables?
A: Last-writer-wins based on timestamp. If the same item is modified in us-east-1 and eu-west-1 simultaneously, the write with the later timestamp survives. The entire item is replaced — no attribute-level merge. To avoid conflicts: route writes to user's home region, use append-only patterns, or implement application-level conflict detection.
Q:What is the outbox pattern and why use it with DynamoDB?
A: The outbox pattern solves the dual-write problem: writing to DynamoDB AND publishing an event must both succeed or both fail. Solution: write the business item and an 'event' item in the same DynamoDB transaction. A stream processor reads the event item from the stream and publishes it to SNS/EventBridge. Guarantees: if the write succeeds, the event will eventually be published.
Q:What happens to a Lambda stream processor when it encounters a bad record?
A: Without BisectBatchOnFunctionError, the entire batch retries indefinitely, blocking the shard. With bisect enabled, DynamoDB splits the batch in half recursively until the failing record is isolated. After MaximumRetryAttempts, the record is sent to the configured dead letter queue (OnFailure destination). Always enable both settings.
Common Mistakes
Not configuring a dead letter queue for stream processors
Without a DLQ, a poison pill record blocks the entire shard indefinitely. All subsequent records in that shard are stuck. Always configure OnFailure destination and BisectBatchOnFunctionError.
✅Configure an SQS dead letter queue as the OnFailure destination and enable BisectBatchOnFunctionError on the Lambda event source mapping.
Assuming Global Tables provide strong consistency across regions
Global Tables are eventually consistent across regions (typically < 1 second). If you read in eu-west-1 immediately after writing in us-east-1, you may get stale data. Design for eventual consistency or route reads to the write region.
✅Route reads to the same region where the write occurred, or design your application to tolerate eventual consistency across regions.
Writing to the same item from multiple regions simultaneously
Last-writer-wins means one update is silently lost. If two regions update different attributes of the same item, the entire item from the 'loser' is discarded. Route writes for the same entity to one region, or use append-only patterns.
✅Implement region affinity for writes (route each entity's writes to a single region) or use append-only patterns to avoid conflicts.
Using KEYS_ONLY stream view when you need item data
With KEYS_ONLY, your Lambda must do a GetItem to fetch the actual data — adding latency, cost, and the risk of reading a newer version than what triggered the stream. Use NEW_IMAGE or NEW_AND_OLD_IMAGES if your processor needs item attributes.
✅Enable NEW_IMAGE or NEW_AND_OLD_IMAGES stream view type when your processor needs to access item attributes.
Not handling stream record ordering correctly
Records are ordered within a partition key, but NOT across partition keys. If your processor assumes global ordering, it will produce incorrect results. Process each partition key's records independently and handle out-of-order cross-partition events.
✅Process records per partition key independently and use sequence numbers or timestamps to handle cross-partition ordering.