In Peliqan, queries can build on other queries. That is useful until a chain gets long enough that nobody remembers what depends on what. Data lineage shows the chain for you: where a result comes from, and what breaks downstream if you change it.
How to use data lineage
- 1Write queries on top of queries. The demo uses a query called VIP Deals that references another query. Hover over the reference and Peliqan shows the details of that query, and you can open its source.
- 2Open the lineage view. With a query open, go to its details section and minimise the view. The lineage of that query appears – in the example, the current query uses another query, which itself depends on two source queries.
- 3Read provenance, then impact. Lineage shows where the data is coming from, which is what you want while troubleshooting. It also shows the other direction: change this query and you can see exactly which downstream queries are affected.
- 4Jump to a source query. From the lineage view, open any source query to view or run it, then look at its lineage too. You can walk the chain in either direction.
What you can do with data lineage
- Trace a number back through every query it passes through
- See which downstream queries a change will affect before you make it
- Open and run a source query from the lineage view while you troubleshoot
- Keep long chains of dependent queries maintainable as your models grow
How lineage is built
Peliqan builds the lineage from the queries themselves, so it stays current as you add and change queries. It covers queries built on synced tables in the data warehouse and queries built on other queries.