A workflow orchestrator is the thing that decides what runs, when, and in what order. If your data stack is a kitchen, the orchestrator is the head chef holding the schedule on a clipboard: it knows that the soup needs stock before the stock can be made, that the dessert can wait until after dinner service, and that if the oven breaks the pastry chef should be paged at 2 a.m. Without an orchestrator, every pipeline is just a script someone runs by hand or a cron job that fails silently at 3 a.m.

In more technical terms: a workflow orchestrator schedules and executes pipelines — collections of tasks with dependencies between them — and gives you visibility into whether they ran, how long they took, and what failed. It is the connective tissue between the rest of the data stack: ingestion (Fivetran, Airbyte), transformation (dbt), reverse ETL, machine learning training, and notification systems all live as nodes in some orchestrator's graph.

Why Orchestration Exists

Before dedicated orchestrators, data teams used cron. Cron is fine for "run this script every hour" but breaks the moment you have dependencies. If job B needs the output of job A, and A is late, cron will happily run B against stale data and never tell you. If a job fails, cron forgets it ever happened. If you need to backfill three weeks of history because you found a bug, cron has no concept of "the run that should have happened last Tuesday."

Orchestrators exist to fix four specific problems that cron cannot:

1. Dependencies. Most data work is a chain. Extract from Salesforce, load into the warehouse, run dbt, refresh the BI cache, send the morning report. Each step needs the previous step to have actually succeeded. An orchestrator models these dependencies explicitly.

2. Retries and failure handling. Real systems fail. APIs rate-limit you, networks blip, warehouses hit query queues. An orchestrator retries with backoff, alerts a human if retries exhaust, and lets you resume from the failed step instead of restarting from scratch.

3. Observability. When the CEO asks "why is the dashboard wrong?" you need to know which pipeline failed, when, what error it threw, and what data it touched. An orchestrator gives you a UI with run history, logs, and lineage.

4. Backfills and reruns. Data pipelines exist in time. If you discover a bug in last week's transformation, you need to rerun the affected days, in order, without re-running everything. An orchestrator understands the concept of a "logical date" and can replay history.

The DAG: One Idea That Defined the Category

The single most important concept in orchestration is the Directed Acyclic Graph, or DAG. The name sounds intimidating but the idea is simple:

• Graph: a set of nodes (tasks) connected by arrows (dependencies).
• Directed: the arrows have a direction. Task A points to Task B means "A must finish before B starts."
• Acyclic: no loops. You can't have A depending on B depending on C depending on A. That would be impossible to run.

Kitchen analogy: A DAG is the recipe card the head chef tapes to the wall. "Chop onions" → "Sauté onions" → "Add stock" → "Simmer". You can't simmer before sautéing, and you can't sauté the same onions twice in a circle. Each arrow is a dependency; the whole picture is the DAG. Multiple parallel branches (the salad station, the dessert station) all converge at "plate the dish."