Data Pipeline Orchestration: Run pipelines in the right order

Peliqan’s scheduler runs each pipeline on its own clock. Data pipeline orchestration is the data app template you add when one pipeline has to finish before the next one starts.

Why you need this plugin

Two pipelines scheduled for nine o’clock both start at nine o’clock, independently. That is fine while they are unrelated. It stops being fine the moment one depends on the other – when a transformation needs its source data to have landed first, or a report needs the CRM sync to be complete.

You can stagger the schedules and hope the first finishes in time, but nothing enforces it. Orchestration replaces timing with sequence: run a pipeline, wait for it to actually finish, check the result, then start the next one.

What you can do with it

1. 1Run pipelines in a fixed order. The next pipeline starts only once the previous one has completed.
2. 2Wait for completion instead of guessing. Setting is_async = False holds the script until the run has finished.
3. 3Check the status before continuing. Read the run status back and decide what the next step should be.
4. 4Handle a pipeline that is already running. Catch it and exit, rather than firing a second run on top of the first.
5. 5Schedule the chain once. One schedule on the orchestration app, instead of a separate schedule per pipeline.

The core of it

connection_name = "Hubspot"
try:
    # is_async = False means run_pipeline waits for completion
    run_pipeline_result = pq.run_pipeline(connection_name = connection_name, is_async = False)
except:
    st.text("A pipeline for this connection is already running.")
    exit()

run_pipeline_details = pq.get_pipeline_runs(connection_name = connection_name, run_id = run_pipeline_result["run_id"])
status = run_pipeline_details["data"][0]["status"]
st.text(status)

# Run subsequent pipelines here

How to add it

Data pipeline orchestration is a data app template. Add it from the templates library in Peliqan, point it at your connections and extend the chain as far as you need – no separate orchestrator to run or maintain.

Want to try it on your own pipelines? Get started with Peliqan.