sailfork / t3code-github / .repos/ alchemy-effect/ website/ src/ content/ docs/ aws/ messaging/ dynamodb-streams.mdx

The DynamoDB table from the previous part stores records, but writes happen in the dark — there's no way for downstream systems to react. DynamoDB Streams fix that: every change to the table emits a change record (insert / modify / remove) you can stream into a Lambda. In Alchemy this is one helper call.

Subscribe to the stream

Start with the smallest possible subscription: log just the event name (INSERT / MODIFY / REMOVE) for each change. Add it alongside the existing bindings:

// src/api.ts
import * as AWS from "@alchemy.run/aws";
import * as DynamoDB from "@alchemy.run/aws/DynamoDB";
+import * as Console from "effect/Console";
import * as Effect from "effect/Effect";
import * as Layer from "effect/Layer";
// ...

const table = yield* DynamoDB.Table("Items", { /* ... */ });
const putItem = yield* DynamoDB.PutItem(table);
const getItem = yield* DynamoDB.GetItem(table);

+yield* DynamoDB.consumeTableChanges(
+  table,
+  Effect.fn(function* (event) {
+    for (const record of event.Records) {
+      yield* Console.log(`${record.eventName}: ${record.dynamodb.Keys?.id?.S}`);
+    }
+  }),
+);

That single consumeTableChanges(...) call enables a stream on the table, creates a Lambda EventSourceMapping that polls it, and adds the IAM grants for dynamodb:GetRecords, GetShardIterator, DescribeStream, and ListStreams — all from the call site. The handler gets each batch Lambda delivers; when it succeeds, Lambda checkpoints past the batch.

Pick the view type

By default the stream emits keys only. Most handlers want the full row before and after the change — set streamViewType to NEW_AND_OLD_IMAGES:

yield* DynamoDB.consumeTableChanges(
  table,
+  { streamViewType: "NEW_AND_OLD_IMAGES" },
  Effect.fn(function* (event) {
    for (const record of event.Records) {
      yield* Console.log(`${record.eventName}: ${record.dynamodb.Keys?.id?.S}`);
    }
  }),
);

The other options (KEYS_ONLY, NEW_IMAGE, OLD_IMAGE) trade payload size for less detail — pick the smallest one your handler actually needs.

Control where consumption starts

startingPosition: "LATEST" is the default and ignores history; the consumer only sees records that arrive after the event-source mapping is enabled. Switch to "TRIM_HORIZON" to replay everything currently in the stream's 24-hour retention window:

yield* DynamoDB.consumeTableChanges(
  table,
  {
    streamViewType: "NEW_AND_OLD_IMAGES",
+    startingPosition: "TRIM_HORIZON",
+    batchSize: 10,
  },
  Effect.fn(function* (event) {
    for (const record of event.Records) {
      yield* Console.log(`${record.eventName}: ${record.dynamodb.Keys?.id?.S}`);
    }
  }),
);

batchSize caps the number of records the function sees per invocation. Larger batches are more efficient; smaller batches fail in smaller blast radii.

Type the row payload

DynamoDB stream records use the same attribute-value shape as GetItem/PutItem. Annotate the handler's event with the row shape and TypeScript will help you destructure NewImage/OldImage:

yield* DynamoDB.consumeTableChanges(
  table,
  {
    streamViewType: "NEW_AND_OLD_IMAGES",
    startingPosition: "TRIM_HORIZON",
    batchSize: 10,
  },
-  Effect.fn(function* (event) {
+  Effect.fn(function* (
+    event: DynamoDB.StreamEvent<{ id: { S: string }; content: { S: string } }>,
+  ) {
    for (const record of event.Records) {
-      yield* Console.log(`${record.eventName}: ${record.dynamodb.Keys?.id?.S}`);
+      yield* Console.log(
+        `${record.eventName}: ${record.dynamodb.Keys?.id?.S} -> ` +
+          `${record.dynamodb.NewImage?.content.S ?? "(deleted)"}`,
+      );
    }
  }),
);

Provide the runtime layer

The Lambda-specific implementation of TableEventSource is in Lambda.TableEventSource. Add it to the merged layer:

}).pipe(
  Effect.provide(
    Layer.mergeAll(
      AWS.Lambda.BucketEventSource,
+      AWS.Lambda.TableEventSource,
      DynamoDB.GetItemHttp,
      DynamoDB.PutItemHttp,
      S3.PutObjectHttp,
      S3.GetObjectHttp,
    ),
  ),
),

TableEventSource is the binding that mutates the table's stream configuration during deploy and creates the event source mapping. You don't have to think about either — they're both side effects of consumeTableChanges(...).

Deploy

bun alchemy deploy

Alchemy enables the stream on the table (takes a few seconds for the table to settle) then creates the event source mapping. Once the mapping is Enabled, every PutItem/UpdateItem/DeleteItem shows up in your function's CloudWatch logs.

Verify

Reuse the PUT /items/:id route to publish changes, then tail the function logs to confirm they arrive:

bun alchemy logs Api --tail

In another terminal:

curl -X PUT --data 'first' "$URL/items/one"
curl -X PUT --data 'second' "$URL/items/two"
curl -X PUT --data 'updated' "$URL/items/one"

You'll see three MODIFY/INSERT log lines stream past — one per write. Stream records typically arrive within 1–2 seconds of the originating change.

Checkpoint at the first failure

Change records in a shard are ordered, so when one fails, report it. Lambda checkpoints every record before it and retries from there, instead of replaying the whole batch:

yield* DynamoDB.consumeTableChanges(
  table,
  { streamViewType: "NEW_AND_OLD_IMAGES" },
  Effect.fn(function* (event) {
    for (const record of event.Records) {
      const ok = yield* writeToWarehouse(record).pipe(
        Effect.as(true),
        Effect.catchCause(() => Effect.succeed(false)),
      );
      if (!ok) {
        return { batchItemFailures: [{ itemIdentifier: record.dynamodb.SequenceNumber! }] };
      }
    }
  }),
);

If the handler fails instead, Lambda retries the whole batch.

Next we'll add an SQS Queue so the Lambda can fan changes (or anything else) out to a different consumer process and pick them up downstream.