Stream Processing

Stream processing computes results over unbounded streams (events, CDC feeds, IoT) — joins, windowed aggregates, materialized views. Sits on top of Message Brokers, feeds Data Warehouses and Time-Series Databases. For batch DAGs, see Data Orchestration.

SQL-on-streams (streaming databases)

★ RisingWave — Postgres-wire-compatible streaming database; create materialized views over Kafka / Pulsar / Postgres CDC, query with SQL. Apache 2.0; managed Cloud is paid. The most-recommended new pick in 2026 for "I want a SQL view that updates in real time."
★ Materialize — same niche, Postgres-wire-compatible streaming SQL database. Was open core under BSL, but re-licensed to a closed-source SaaS in 2024 — only the older OSS releases remain Apache 2.0. Watch licensing carefully.
ksqlDB (Confluent) — SQL over Kafka topics; Confluent Community License (not OSI-approved). Less popular in 2026 than RisingWave for greenfield.
TimePlus / Proton — open-source streaming SQL on top of ClickHouse; Apache 2.0; small community.
Tinybird — hosted ClickHouse + ingest API; SQL-defined materialized streams; small free tier.

★ Apache Flink — the heavyweight; stateful, exactly-once, event-time-aware; Apache 2.0. Production default at large scale (Stripe, Uber, Pinterest). Ververica is the commercial sponsor.
Apache Spark Structured Streaming — micro-batch; Apache 2.0; tight integration with Spark batch + ML.
Apache Beam — unified batch/stream API; runs on Flink / Dataflow / Spark; Apache 2.0.
Kafka Streams — JVM library, runs in your app process; Apache 2.0; great for "modest stream processing inside a Java service."

★ Bytewax — Rust core + Python API; dataflow-style; Apache 2.0. Default for "I want Flink semantics, Python ergonomics."
★ Pathway — Python streaming framework; reactive; supports LLM-on-stream patterns; BSL-1.1 (free up to a generous size threshold).
Quix Streams — Python library for Kafka-based stream processing; Apache 2.0; pairs with Quix Cloud.
Faust-streaming — community-maintained fork of the original Robinhood Faust; Apache 2.0; less active.

Benthos / RedPanda Connect — Go-based stream-processor + connector toolkit; MIT; great for routing and small transforms on streams. Often replaces Kafka Streams for small jobs.
Vector (Datadog) — observability-focused but capable of general stream transforms; see Log Aggregation.
NATS JetStream consumers — for simple consume + process loops; see Message Brokers.

Event time vs. processing time. Late events, watermarks, allowed lateness — Flink and RisingWave handle this; many lightweight tools punt.
Exactly-once requires checkpoints + idempotent sinks. Flink + Iceberg sink, RisingWave's checkpoint store.
Joins over streams need state. RocksDB-backed state in Flink; storage planning matters.
Windowed aggregates. Tumbling, hopping, session — pick the right one for the metric.
Hot keys. A small number of partitions can hot-spot; pre-aggregate locally before keying.
Replay > live retry. Keep raw events durable in Kafka / Iceberg so you can re-run if the processor logic changes.

RisingWave — Apache 2.0 OSS; managed Cloud paid.
Materialize — closed-source SaaS now; OSS Apache 2.0 versions only up to v0.26; do not assume current builds are open.
ksqlDB — Confluent Community License (CCL); usable but not OSI-approved.
Pathway — BSL-1.1 with a usage threshold; review the threshold before betting on it for paid product.
Flink, Spark, Beam, Bytewax, Quix Streams, Kafka Streams, Benthos — all permissive (Apache 2.0 / MIT).