A data pipeline moves data from where it is produced to where it is analysed. There are two ways it can arrive – in batches, or as it happens – and the diagram below shows both landing in the same modelled layer, with the obligations that apply either way.
The two paths
- 1Batch runs on a schedule. Extract on an interval, land the raw records, then transform them into models. Simpler to reason about and to re-run, and correct for the large majority of reporting. Most teams should start here and stay here longer than they expect to.
- 2Streaming reacts as events occur. Capture changes continuously, move them through a log, process them on the way. Worth its extra operational cost when a decision genuinely cannot wait for the next batch – fraud checks, stock levels, live personalisation.
The obligations that apply either way
- Schedule or trigger: something has to decide when work runs and what it depends on.
- Schema drift: the source will add, rename and drop fields without warning you.
- Retries: transient failures are normal, so the pipeline must resume rather than start again from nothing.
- Idempotent writes: running the same load twice must not create the data twice – this is the one that quietly corrupts numbers.
- Freshness alerts: a pipeline that silently stops is worse than one that fails loudly, because nobody stops trusting the dashboard.
Why replayability is the design constraint
Every serious pipeline decision follows from one question: can you run it again? Keeping raw data as it arrived means a modelling mistake is fixed by rebuilding, not by re-extracting from a source that may no longer hold the history. This is the practical reason ELT overtook ETL for most workloads.
How this relates to ETL and ELT
Batch and streaming describe when data moves. ETL and ELT describe where it is transformed. They are independent choices, and the transform position is drawn out on our ETL vs ELT diagram; the full stage-by-stage view is on the ETL architecture diagram.
What this looks like in Peliqan
Peliqan runs managed incremental pipelines from 300+ sources into your warehouse, with scheduling, retries and schema handling included, so the obligations above are the platform’s job rather than a set of scripts somebody has to maintain.
Use this diagram wherever you like
The diagram is free to use, including commercially, as long as there is a visible link back to this page. Download the PNG for slides and documents, or the SVG if you want to edit the labels. No email required. You can browse the rest of the set in the Peliqan diagram library.
Ready to build this on your own data? Get started with Peliqan.
