2. Set up ELT pipelines in Peliqan

Peliqan has 300+ connectors to ERP, CRM and accounting systems, cloud storage and external databases. This tutorial shows how a pipeline is created, scheduled and monitored, so data from every source you connect keeps arriving in the data warehouse on a schedule.

How to set up an ELT pipeline

1. 1Add a connection. Pick a connector from the list and sign in. Peliqan creates the pipeline for you and starts syncing in the background into the built-in data warehouse, or into your own Snowflake, Redshift, BigQuery or Azure warehouse.
2. 2Set the schedule. Each pipeline runs on the scheduler. Change it to run every six hours or more frequently, or disable it if you want to control runs yourself.
3. 3Check the runs. Open a connection to see its run history. The first run discovers and locks the schema, then each sync run fetches data. You can inspect the individual API calls, or let the AI data engineer summarise a whole run for you.
4. 4Change what gets synced. Open a connection, go to Advanced settings and select additional tables. Some connectors add their own options – with Odoo, for example, you can list extra models including custom ones.
5. 5Run a pipeline from an app. For a custom schedule or extra logic, create an app and call pq.run_pipeline with a connection and, optionally, only the tables you want. The run appears in the pipeline history like any other.

What you can do with ELT pipelines

• Sync 300+ sources into one warehouse without writing or maintaining API code
• Let pipelines handle reauthorization and rate limiting, backing off automatically when a source throttles them
• Sync incrementally, so each run only fetches new and updated records
• Trigger a pipeline for selected tables from a Python app on your own schedule

About incremental syncs

Peliqan keeps a state per table: the timestamp of the most recent record it received. Each incremental run picks up from there. Inside every table you will also find metadata columns that start with _SDC, at the end of the table. The useful one is _SDC_BATCHED_AT, the moment the pipeline received that record – it only changes when the record itself actually changed in the source. If the source schema changes, run a full resync: it drops the tables, clears the state and reloads everything from the start.

Ready to move your data? Get started with Peliqan.