Change Data Capture: Real-Time Patterns for Modern Pipelines

What CDC solves that batch ETL doesn't

A nightly batch job that reloads a table works fine until the business needs to know about a change within seconds rather than by tomorrow morning — a fraud flag, an inventory update, a price change that needs to propagate to a live storefront. Change Data Capture (CDC) is the family of techniques for streaming just the changes out of a source system as they happen, instead of periodically re-reading the whole table.

Log-based vs. query-based CDC

Query-based CDC: periodically query the source for rows where an updated_at column is newer than the last run. Simple to implement, but it misses hard deletes (a row that's gone leaves no trace to query for) and adds repeated read load on the source database.

Log-based CDC: read the database's own transaction log (the same mechanism it uses for replication) to capture every insert, update, and delete as it's committed — including deletes — without querying the source tables directly. Tools like Debezium do this for Postgres, MySQL, and SQL Server.

A typical log-based CDC architecture

A common, production-proven pattern: source database → Debezium connector → Kafka topic → stream processor → Delta Lake / Iceberg table. Debezium turns each committed row change into a structured event; Kafka buffers and distributes those events reliably; the stream processor (Spark Structured Streaming, Flink) applies them to the target table.


{
  "op": "u",
  "before": { "id": 482, "status": "pending" },
  "after":  { "id": 482, "status": "shipped" },
  "ts_ms": 1758312044000
}

That single event carries everything a downstream job needs to apply the update: which row, what it was, what it became, and exactly when.

When batch is still the right call

CDC infrastructure is real operational overhead — a connector to run, a Kafka cluster to maintain, exactly-once semantics to get right. For a dimension table that changes a handful of times a day, a nightly full load (as covered in our earlier article on Slowly Changing Dimensions) is simpler, cheaper, and just as correct. CDC earns its complexity when the business genuinely needs sub-hour latency or when hard deletes matter — not by default.

Data Engineering