Data Orchestration

This is the data-engineering view of orchestration. For Kubernetes-native pipelines (Argo, Tekton) and ML workflows (Kubeflow, Flyte), see Workflows & Pipelines. For app-tier durable execution (Inngest, Temporal, Restate), see Workflow Engines.

Python-flavored orchestrators

• ★ Apache Airflow — the incumbent. DAGs as Python; massive operator ecosystem (Snowflake, BigQuery, dbt, Spark, …). Apache 2.0. Boring, mature, runs everywhere; Airflow 3 (2025) modernized the scheduler and UI.
• ★ Dagster — asset-centric (you declare the data, not just the task). Apache 2.0; paid Dagster+ Cloud. The most-recommended new pick for data teams in 2026, especially with dbt, Sling, or Polars.
• ★ Prefect — task-flow API in Python; lightweight and ergonomic. Apache 2.0; paid Prefect Cloud has a generous free tier.
• Mage — newer; combines orchestration with a notebook-style UI; Apache 2.0; pivoted toward AI in 2025.
• Flyte — type-safe, k8s-native; strong for ML. Apache 2.0.
• Kedro — pipeline-as-package opinionated framework from QuantumBlack; works well with Airflow / Argo as the runtime.

Polyglot / config-first

• ★ Kestra — YAML / declarative DAGs; Java engine; runs Python, Node, shell, dbt, Spark. Apache 2.0; paid Cloud. The default pick for non-Python-only teams.
• Argo Workflows — Kubernetes-native; container-per-step. Apache 2.0; see Workflows & Pipelines.
• Conductor (Netflix), Temporal — see Workflow Engines; usable for data but app-tier-shaped.

Lightweight / single-binary

• Windmill — open core; TypeScript / Python scripts orchestrated via UI; AGPLv3. Self-hostable; great for small teams.
• Pyrra, schedule (Python lib), croniter + your own runner — for tiny jobs, plain cron + a Postgres lock table beats running an orchestrator.
• GitHub Actions / GitLab CI — surprisingly viable for nightly ETL up to a few hundred jobs; see CI / CD.

Hosted / managed

• Astronomer (managed Airflow), Prefect Cloud, Dagster+ Cloud, Kestra Cloud — usually the "we don't want to run this" upgrade.
• AWS MWAA / GCP Cloud Composer — managed Airflow on cloud providers; pay-by-the-hour.
• Modal, Beam.cloud — serverless Python; often replace orchestrators for lightweight ETL.

Patterns to adopt

• Assets, not tasks. Dagster's model — "this DAG produces customers_daily" — generalizes; even in Airflow, treat outputs as the unit, not steps.
• Idempotent + partitioned. Re-running a day shouldn't double-insert. Partition on the ingest date and INSERT … ON CONFLICT or use MERGE.
• Don't put business logic in the orchestrator. Logic lives in dbt / Python packages / SQL; the orchestrator only schedules and observes.
• Observability per asset. Freshness, row-count, and schema checks per asset go into Data Observability and Data Quality.
• CI for DAGs. Lint with sqlfluff / ruff; test DAG imports in CI; roll forward via GitOps.

Pick this if…

• Default new data team in 2026: Dagster (asset-first) or Prefect (lighter).
• You have an existing Airflow team or many Airflow operators: stay on Airflow.
• Polyglot team / non-Python services: Kestra.
• k8s-native, container-per-step: Argo Workflows.
• ML-heavy, type-safe: Flyte.
• Small team, hosted: Prefect Cloud or Dagster+ free tiers.