Change Data Capture (CDC)

CDC is "tail the database's WAL / binlog and emit a stream of inserts/updates/deletes." It's the lowest-latency way to get operational data into a warehouse, search index, or other service. Sits underneath ETL / ELT & Ingestion and adjacent to Message Brokers.

Postgres CDC

• ★ Debezium — the incumbent. Java-based Kafka Connect plugin reading Postgres logical replication, MySQL binlog, MongoDB oplog, and more. Apache 2.0. Battle-tested at large scale; ops-heavy but stable.
• ★ PeerDB — Postgres-first; replicates Postgres → Snowflake / BigQuery / ClickHouse / Postgres / S3 / Kafka with very low latency. Acquired by ClickHouse Inc. in 2024; OSS edition (Elastic License v2) remains usable.
• Sequin — Postgres → Kafka / Redis / SQS / HTTP webhooks; Elixir/Go; MIT. Lightweight; great for app-tier "react to row changes."
• Materialize Source / RisingWave Source — both stream processors can ingest Postgres logical replication directly; see Stream Processing.
• pg_logical / wal2json — Postgres extensions; the building blocks under most CDC tools.
• Supabase Realtime — bundled with Supabase; broadcasts row changes to clients.

MySQL CDC

• ★ Debezium MySQL connector — same project, MySQL binlog source.
• Maxwell's Daemon — MySQL → JSON → Kafka / Kinesis / SQS / SNS; Apache 2.0; lighter than Debezium.
• Canal (Alibaba) — Apache 2.0; widely deployed in Asia.

SaaS / multi-source CDC

• Estuary Flow — multi-source CDC + transforms; commercial with a free tier; see ETL / ELT.
• Striim, HVR (Fivetran), Qlik Replicate, Oracle GoldenGate — enterprise CDC; paid; sometimes the only option for old DBs (Oracle, DB2).
• Airbyte CDC — Airbyte sources support CDC for major DBs; simpler than running Debezium yourself.
• Snowflake Streams, BigQuery Change Streams, Postgres LISTEN/NOTIFY — the warehouse / DB-internal version.

Targets / sinks

• Iceberg / Delta / Hudi — increasingly the default landing target for CDC (MERGE INTO); see Lakehouse Table Formats.
• ClickHouse — ReplacingMergeTree table engine handles upserts well.
• Apache Kafka / Redpanda — durable buffer between source and consumer.
• Search indexes — Meilisearch / Typesense / OpenSearch via Kafka Connect or Debezium sinks; see Search.

Patterns to know

• Outbox pattern. App writes a domain event into an outbox table in the same transaction as the business write; Debezium tails the table. Avoids dual-writes.
• Snapshot + stream. Initial snapshot of every row, then incremental WAL-based updates; Debezium's "incremental snapshot" handles this.
• Idempotent consumers. Same change can arrive twice; consumers must dedupe by (pk, lsn) or (pk, op_ts).
• Schema evolution. Producers add columns; CDC must propagate them. Use a schema registry (see Message Brokers).
• Replication slot management. Postgres logical slots accumulate WAL if nothing consumes them — monitor and alarm.

Pick this if…

• Default Postgres CDC, large scale: Debezium.
• Postgres → cloud warehouse, low ops: PeerDB.
• Postgres CDC for app-tier consumers: Sequin.
• MySQL, lightweight: Maxwell or Debezium.
• Avoid running CDC infra entirely: Airbyte CDC connectors or Estuary Flow.
• Outbox pattern for microservices: Debezium + a domain_events table.