Designing Reliable CDC Pipelines on AWS

Change data capture looks simple on an architecture diagram: read database changes, land them somewhere durable, transform them, and publish the latest state. The difficult part begins when the pipeline has to survive retries, late events, deletes, schema changes, and operational failures without quietly corrupting downstream data.

This article describes the design principles I use when thinking about a reliable CDC pipeline on AWS. The examples are intentionally generic: the goal is to explain the engineering model, not a particular company’s implementation.

Start with the contract, not the tools

Before choosing services, define what the pipeline promises.

A useful CDC contract answers five questions:

1. What is the source of truth? A transactional database, an event log, or both?
2. What does “current” mean? Latest event by source timestamp, ingestion order, or commit sequence?
3. How are deletes represented? Tombstones, operation codes, or physical removal?
4. Can processing be replayed? If a transformation changes, can a deterministic result be rebuilt from durable inputs?
5. What is the acceptable freshness window? Seconds, minutes, or hours?

Without those answers, it is easy to build a technically impressive pipeline whose semantics are ambiguous.

Separate capture from processing

One of the strongest reliability patterns is to make capture durable and processing repeatable.

Source database
      │
      ▼
Change capture
      │
      ▼
Durable object storage
      │
      ├── raw/full-load records
      └── raw/change records
              │
              ▼
      incremental processing
              │
              ▼
       curated datasets

Object storage creates a boundary between “did we capture the change?” and “did we process the change correctly?” If a transformation fails, the source does not need to be queried again. If business logic changes, the captured history can be replayed.

That separation is especially useful when AWS DMS, S3, Glue or Spark, and analytical stores are involved. The individual services matter less than the contract between stages.

Treat inserts, updates, and deletes as first-class events

Many CDC implementations are designed around inserts and then bolt on updates and deletes later. That usually creates special-case logic.

A cleaner model is to normalize every captured record into an operation-aware structure:

primary_key | operation | source_change_time | payload
-------------------------------------------------------
42          | I         | ...                | {...}
42          | U         | ...                | {...}
42          | D         | ...                | {...}

The processing layer can then reason explicitly about state transitions. The exact operation field may come from the CDC technology or may be normalized during ingestion.

The important point is that a delete is not “missing data.” It is an event with meaning.

Idempotency is more important than exactly-once slogans

Distributed pipelines retry. Jobs are restarted. Workers fail after writing some output but before recording success. “Exactly once” is often used as shorthand for a much more useful property: reprocessing the same input should not create an incorrect result.

Common ways to achieve this include:

• deterministic merge keys;
• source sequence numbers or commit timestamps;
• checkpointed input ranges;
• transactional table formats;
• write-then-swap patterns;
• deduplication using stable event identities.

If a job processing changes from 10:00–11:00 runs twice, the second execution should converge to the same state rather than doubling records.

Late-arriving changes need a policy

A dangerous assumption is that yesterday’s data is immutable.

Operational systems can emit updates for records created days or months earlier. Corrections, delayed transactions, backfills, and source-system repairs all violate a strict “new partition only” model.

There are several strategies:

Reprocess a moving window

Every run revisits a recent range of source partitions. This is simple and often adequate when late changes are bounded.

Maintain change-aware keys

Use CDC metadata to identify exactly which business keys changed and update only those records.

Use a table format designed for incremental updates

Formats such as Apache Iceberg provide table-level metadata and operations that make incremental maintenance easier than manually rewriting directory structures.

The correct approach depends on change volume and storage layout. What matters is that lateness is modeled deliberately rather than treated as an anomaly.

Data quality should compare behavior over time

A pipeline can complete successfully and still produce bad data.

Basic checks such as row counts are useful, but production validation should look for multiple signals:

• freshness;
• count changes relative to previous runs;
• null-rate changes;
• duplicate rates;
• domain or range violations;
• referential-integrity breaks;
• unexpected distribution shifts.

The most useful question is often not “is today’s count nonzero?” but “is today’s behavior plausible compared with recent history?”

For example, a 40% drop in daily records may be technically valid, but it deserves investigation when the previous seven days were stable.

Make replay an intentional capability

A CDC pipeline is easier to operate when replay is part of the design.

I like to distinguish three layers:

Raw — immutable captured source data.

Processed — normalized, deduplicated, operation-aware data.

Published — consumer-ready tables, files, or APIs.

If the published layer is wrong, rebuilding it should not require re-extracting the source database. If processing logic changes, the raw layer should be sufficient to reproduce the result.

That architecture costs some storage, but object storage is often inexpensive compared with the operational cost of an unrecoverable pipeline.

Observability should answer “where did the data stop?”

A useful monitoring system should let an engineer determine whether a freshness problem occurred during:

1. source capture;
2. object-storage delivery;
3. transformation;
4. publication;
5. downstream consumption.

For each stage, record timestamps and counts that can be compared. A single “job succeeded” metric does not tell you whether the source itself stopped producing changes.

Optimize after correctness is measurable

Once the semantics are stable, cost and performance become much easier to improve.

Typical opportunities include:

• processing only changed partitions or keys;
• compacting small files;
• selecting appropriate Spark executor sizes;
• pruning columns early;
• using columnar storage;
• reducing unnecessary full-table scans;
• tuning analytical warehouse workloads independently from ingestion.

Cost optimization without reliable measurements often shifts expense from compute into engineering time.

The architecture principle I keep coming back to

Reliable CDC systems are not primarily about moving data quickly. They are about maintaining a defensible history of state changes and being able to explain how the current dataset was produced.

If capture is durable, processing is idempotent, deletes and late updates have explicit semantics, and replay is possible, many operational problems become recoverable rather than catastrophic.

That is the foundation I would choose before optimizing for latency, throughput, or cost.