Data observability is continuous monitoring of the signals a table gives off, so a problem is caught before somebody spots a wrong number in a meeting. There are five signals, and the diagram below shows what each one alerts on.
The five pillars
- 1Freshness – how recently it updated. The single most valuable signal, because the most common failure is a job that stopped running while the dashboard carried on showing yesterday’s number as though it were today’s.
- 2Volume – how many rows arrived. Catches the partial load that reported success. A table that normally gains 40,000 rows and gains 400 is broken even though nothing errored.
- 3Schema – what the columns are. Upstream teams rename and drop fields without telling you. Schema monitoring turns that from a silent break into an alert.
- 4Distribution – whether the values look normal. Null rates, ranges and category mixes shifting suddenly. This is the pillar that catches the problems no one wrote a test for.
- 5Lineage – what is upstream and downstream. Not a health signal itself, but the one that turns an alert into an action, because it tells you what else is affected.
Observability or data quality?
They are complementary, and the difference is who decides what to check. Quality is the standard you set deliberately: six dimensions, each with a test you write. Observability is the watch that runs regardless, catching anomalies nobody predicted. A team with tests and no observability finds out about the failures it imagined; a team with observability and no tests has no agreed definition of correct.
What to instrument first
- Freshness on every table a dashboard or report depends on.
- Row-count deltas against the same weekday, not against yesterday.
- Schema change alerts on the tables you do not control.
- Null-rate monitoring on the handful of fields your key metrics use.
How it relates to quality
The standard itself is on our six dimensions of data quality diagram, and the graph that makes an alert actionable is on the data lineage diagram.
What this looks like in Peliqan
Pipelines from 300+ sources run in the same place as the models built on them, so freshness and row counts are known per table rather than reconstructed from separate tools’ logs, and a failure can alert before the number reaches a report.
Use this diagram wherever you like
The diagram is free to use, including commercially, as long as there is a visible link back to this page. Download the PNG for slides and documents, or the SVG if you want to edit the labels. No email required. You can browse the rest of the set in the Peliqan diagram library.
Ready to build this on your own data? Get started with Peliqan.
