Message Brokers & Event Streaming

Distinct from Job Queues (which are app-tier "run this job later") and Workflow Engines (durable multi-step orchestration). This page is the layer underneath: the broker that delivers messages between services.

Streaming brokers (replayable log)

• ★ Apache Kafka — the incumbent. Distributed log, partitioned, retains everything. Self-host or Confluent Cloud / AWS MSK / Aiven / Redpanda Cloud. Heavyweight.
• ★ Redpanda — Kafka-API-compatible, single binary, no ZooKeeper / KRaft management; faster, easier ops. The default new "I want Kafka" pick in 2026.
• Apache Pulsar — competitor to Kafka with multi-tenancy and tiered storage built in.
• WarpStream (Confluent) — Kafka-API-compatible, S3-backed; cheap for high throughput, slightly higher latency.

Lightweight pub/sub

• ★ NATS + JetStream — Go-based; tiny single binary; pub/sub, request/reply, persistent streams, key-value, object store. The default light pick. Synadia Cloud is the hosted option.
• RabbitMQ — long-running classic; AMQP; mature; great UI.
• Mosquitto / EMQX / HiveMQ — MQTT brokers; for IoT and resource-constrained clients.
• Redis Streams — bundled with Redis; small / medium scale.
• DragonflyDB — Redis-compatible streams; faster.
• ZeroMQ / nanomsg — embeddable messaging; library not broker.

Cloud-provider native

• ★ Cloudflare Queues — fire-and-forget; pairs with Workers.
• ★ GCP Pub/Sub — global, scales effortlessly; the cloud default.
• AWS SNS + SQS — pub/sub + durable queues; the AWS default combo.
• AWS Kinesis — Kafka-equivalent on AWS.
• AWS EventBridge — event bus + schema registry; great for cross-service events.
• Azure Service Bus, Azure Event Hubs / Event Grid — Azure equivalents.

Event-driven architecture platforms

• Inngest / Trigger.dev — durable functions on top of any underlying broker; see Workflow Engines.
• EventStoreDB — event sourcing-first DB.
• Hookdeck — event gateway; good for inbound webhooks → fan-out.

Schema registry / contracts

• ★ Confluent Schema Registry — Avro / Protobuf / JSON Schema for Kafka messages.
• Apicurio Registry — open-source alternative.
• AWS Glue Schema Registry — for Kinesis / MSK.
• Buf Schema Registry (BSR) — Protobuf-focused.

Bridges / gateways

• Kafka Connect — connectors to/from Kafka and DBs / object stores / SaaS.
• Debezium — CDC (change data capture) from databases to Kafka / NATS / Pulsar.
• Conduit (Meroxa) — modern Connect alternative.

When to reach for what

• In-process events: an EventEmitter or mitt; you don't need a broker.
• Background jobs: Job Queues — pg-boss, BullMQ, Inngest.
• Cross-service events, low volume: GCP Pub/Sub or Cloudflare Queues; managed.
• Cross-service events, you operate it, modest scale: NATS + JetStream.
• Event sourcing / replayable streams: Kafka or Redpanda.
• IoT / many clients with intermittent connectivity: MQTT (EMQX, Mosquitto).
• Webhook fan-out: Hookdeck or Cloudflare Queues.

Pick this if…

• Default OSS streaming, simple ops: Redpanda.
• Default hosted streaming: Confluent Cloud or GCP Pub/Sub.
• Light pub/sub + KV in one binary: NATS JetStream.
• AWS-native: SNS + SQS; Kinesis if streaming.
• All-Cloudflare: Cloudflare Queues + Workflows.