Workflows & Pipelines (Infra)

Distinct from CI / CD (build & ship code) and Workflow Engines (durable app-tier business workflows). This page covers infra-level pipelines: ETL / ML / batch / scheduled jobs that run inside a cluster.

Kubernetes-native

• ★ Argo Workflows — DAG / step-based; CRD-defined; lives in your cluster. The default for k8s-native pipelines. Pairs with Argo CD / Argo Events.
• ★ Tekton (CNCF) — pipelines-as-CRDs; popular for k8s-native CI/CD; broad ecosystem.
• Argo Events — event-source CRDs that trigger Argo Workflows.
• Apache Airflow (now KubernetesExecutor / KubernetesPodOperator) — runs on k8s; classic DAG scheduler.
• Prefect — Python-native modern Airflow alternative.
• Dagster — newer; data-asset-centric; great for data engineering.
• Mage — newer Python data pipeline.
• Kubeflow Pipelines — for ML on k8s.
• Flyte — type-safe ML / data orchestrator.

Non-k8s

• Apache Airflow standalone — still huge in data engineering.
• Prefect Cloud — managed Prefect.
• Dagster Cloud — managed Dagster.
• n8n / Activepieces / Windmill — visual workflow tools; popular in self-host community.

CI-style alternatives

• See CI / CD — GitHub Actions / Buildkite / Dagger / Earthly. These can run pipelines too.
• Dagger — increasingly used for data and ML pipelines, not just CI.

Specialized

• Apache Beam — unified batch + streaming model; runs on Flink / Dataflow / Spark.
• Apache Flink — stream processing.
• Apache Spark — batch + stream; ML.
• Ray — distributed Python; ML / RL workloads.
• Modal / Beam.cloud / Replicate — serverless Python / ML compute.

What lives in this category

• ETL (Extract / Transform / Load) pipelines.
• ML training / inference orchestration.
• Periodic batch jobs (nightly reports, data warehouse loads).
• Multi-step infra workflows (provision → configure → test → release).
• Long-running data backfills.

When to use Argo Workflows specifically

• You're already on k8s and want to keep workflows there.
• DAGs of containers (each step is a container; great for polyglot pipelines).
• You want UI + CRDs + GitOps for workflows.
• Heavy parallelism (1000+ pods at once is fine).

When Airflow / Prefect / Dagster instead

• Data engineering team that already knows Python and DAG concepts.
• Need rich operators for data warehouses (Snowflake, BigQuery, Redshift, dbt).
• Less interested in container-per-step; want Python DAGs.
• Prefer hosted UIs / dashboards.

Patterns to adopt

• Idempotent steps. Re-running a workflow shouldn't double-charge / double-send.
• Retries with backoff in the workflow definition, not the app.
• Persistent storage for intermediate artifacts (S3 / R2 / Volume).
• Dependencies as DAGs, not implicit ordering.
• Versioned workflow definitions in git; deploy with GitOps.
• Observability — Argo / Airflow / Prefect / Dagster all have their own UIs; integrate with Grafana / Loki for unified logs.

Pick this if…

• k8s-native, container-per-step: Argo Workflows.
• k8s-native CI/CD-shape: Tekton.
• Data engineering, Python DAGs: Airflow (boring, mature) or Dagster (modern).
• ML on k8s: Kubeflow Pipelines or Flyte.
• Visual / no-code-ish: n8n or Windmill.
• Stream processing: Flink or Spark Streaming.