Change data capture is how a pipeline notices that a row changed without re-reading the whole table. There are three ways to do it, they are not equivalent, and the diagram below shows what each one watches and what it costs the source system.
The three methods
- 1Query-based CDC polls a timestamp column. Select everything where updated_at is newer than the last run. It works on almost any source and needs no special access, which is why it is so common. It also cannot see deletes, and it misses changes that do not update the timestamp.
- 2Trigger-based CDC writes every change to a shadow table. A database trigger records inserts, updates and deletes as they happen, so nothing is missed. The cost is that the trigger runs inside the source transaction, adding write load to the production system your business depends on.
- 3Log-based CDC reads the database’s own write-ahead log. The log already exists for crash recovery, so reading it adds no load to the tables and captures every change in commit order, deletes included. It needs elevated database access, which is the main reason teams do not start here.
Why deletes are the deciding factor
A cancelled order that is deleted in the source but never deleted downstream keeps counting toward revenue, quietly, for as long as that table exists. Query-based CDC cannot see it. Teams usually discover this during a reconciliation, months after the gap opened, which is why ‘does it capture deletes’ is the first question to ask of any replication setup.
Where CDC is actually used
- Keeping a warehouse in sync with production without a nightly full reload.
- Migrating a database with minimal downtime, by replaying changes while the old system still runs.
- Feeding streaming pipelines and event-driven systems from an existing relational database.
- Maintaining an audit history of what changed and when, which the source table itself usually does not keep.
How CDC relates to ELT
CDC is the extract step done incrementally. The transform still happens afterwards, in the warehouse, which is the pattern drawn on our ETL vs ELT diagram. Where it fits in the wider flow is on the data pipeline architecture diagram.
What this looks like in Peliqan
Peliqan syncs incrementally from 300+ sources, using log-based capture where the source database supports it and the API’s own change endpoints where it does not, so you get incremental loads without maintaining capture logic per source.
Use this diagram wherever you like
The diagram is free to use, including commercially, as long as there is a visible link back to this page. Download the PNG for slides and documents, or the SVG if you want to edit the labels. No email required. You can browse the rest of the set in the Peliqan diagram library.
Ready to build this on your own data? Get started with Peliqan.