Fan-Out with DynamoDB Streams and Lambda To Build Event-Driven Designs

Designing systems that are reactive, scalable and decoupled used to be complicated.

Today, cloud providers (AWS) allow you to do build these system quickly and with ease. This can be achieved with a few services only and all without spending a penny.

One such powerful design pattern is “fan-out processing” and we can design this with DynamoDB Streams and a Lambda function.

This pattern will enable us to automatically respond to changes in your DynamoDB table by triggering multiple downstream workflows without complex orchestration.

Let’s break down how this works.

What is Fan-out?

Fan out is a design pattern that takes a single event (e.g. a DynamoDB write) and distributes it to multiple independent consumers (e.g. search index, message queue, etc).

In DynamoDB, when an item is added, updated or deleted, a record of that change is captured in a DynamoDB Stream. You can then attach a Lambda function to this stream and react to the write operation.

The Lambda can either:

• handle the event directly (like update a cache)

• push the item into multiple async processing pipelines (like SQS, SNS, EventBridge, etc)

Fan-out Use Cases

Imagine you are building a collaborative editing application. When a document is edited you can:

1. Update a search index (OpenSearch).

2. Send a real-time Websocket notification to subscribed collaborators.

3. Log the changes for auditing

4. Trigger a text analysis job (like AI processing)

Now, instead of embedding all of this logic into a single monolithic Lambda function, you use DynamoDB streams and break down each of these elements into separate functions or services.

Implementing Fan-out designs

Step 1: Enable DynamoDB Streams

In your DynamoDB table, you can head over to the Exports and streams section.

If you scroll down a little you will see the DynamoDB stream details section. Click to enable Streams. Select the new and old images option.

You can also update the Streams settings with the code below:

import { UpdateTableCommand, DynamoDBClient } from '@aws-sdk/client-dynamodb'

const client = new DynamoDBClient()

await client.send(new UpdateTableCommand({
  TableName: 'documents',
  StreamSpecification: {
    StreamEnabled: true,
    StreamViewType: 'NEW_AND_OLD_IMAGES'
  }
}))

Step 2: Attach a Lambda to the Stream

In the Triggers section, create a new trigger and attach a Lambda function to it.

Your Lambda function will be invoked automatically when an item is written, updated or deleted from your table:

export const handler = async (event) => {
  for (const record of event.Records) {
    const data = record.dynamodb.NewImage

    await Promise.all([
      updateSearchIndex(data)
      sendToSQS(data),
      publishToWebsocket(data),
      logToCloudWatch(data),
    ])
  }
}

The Lambda function loops through the items changed on your DynamoDB table and can then fan-out that data to the downstream services.

Best practices

Using this fan-out process offers several benefits:

• Decoupling your processes

• Scales independently

• More resilient (if a downstream process is overloaded, you can retry the logic without touching the core system)

Some best practices to implement:

1. Use batch size and parallelization factor in the Lambda event source mapping to control throughput. (for DynamoDB its batch size

2. Implement idempotency to avoid processing events twice.

3. Use DLQs (dead-letter queues) for failed records

4. Consider using EventBridge as an intermediate fan-out bus if you want more control or filtering.

Conclusion

Using the fan-out design pattern with DynamoDB Streams and Lambda is a clean, scalable and serverless way to build event-driven systems. It helps you turn your database changes into real-time workflows without much complexity.

Every modern application has use cases where it needs to respond to changes in your data, and this is a cost-effective and non-complex way to design it.

👋 My name is Uriel Bitton and I hope you learned something in this edition of Excelling With DynamoDB.

📅 If you're looking for help with DynamoDB, let's have a quick chat.

🙌 I hope to see you in next week's edition!