Service Discovery & Coordination

The "how do services find each other and agree on things" layer.

On Kubernetes

• ★ DNS via CoreDNS + Service objects — default; "service.namespace.svc.cluster.local" — works for 90% of needs.
• Headless Services + StatefulSets for stateful workloads.
• Service mesh (Istio / Linkerd / Cilium) layers on top — see Service Mesh.

Outside Kubernetes

• ★ Consul (HashiCorp) — service catalog + health checks + KV + DNS interface. The default for non-k8s service discovery. License now BSL.
• etcd (CNCF) — distributed key-value store; what k8s itself uses. Strong consistency; raft-based. Use when you need a coordination store, not when you need a service registry.
• ZooKeeper — old, mature; still common in JVM / Kafka / HBase shops.
• NATS — pub/sub messaging that includes simple service discovery via subjects.

Lightweight alternatives

• mDNS / Avahi — local-network discovery; works for small homelab / LAN.
• dns-sd / Bonjour — same niche.
• Just DNS — for small teams, plain DNS works fine; over-engineering Consul too early is a common mistake.

Distributed coordination primitives

• ★ etcd — leader election, distributed locks, configuration.
• Consul — same; plus rich service catalog.
• ZooKeeper — same; older API.
• PostgreSQL advisory locks — for many app-level "only one of N processes does X" needs, this is enough.
• Redis Redlock — controversial pattern; use lightly.

When to reach for what

• K8s cluster, find services: k8s DNS. Stop here.
• Mixed VMs + containers, find services: Consul.
• Need a generic strongly-consistent KV: etcd.
• Need leader election for one process across N hosts: etcd or PostgreSQL advisory locks.
• You think you need ZooKeeper: etcd is usually a friendlier option for new code.
• Pub/sub messaging + lightweight discovery: NATS.

Common patterns

• Service registration — services register themselves on start, deregister on stop. Consul agent / etcd lease / k8s Endpoints all do this.
• Health checks — Consul or k8s probes; deregister unhealthy services.
• Sidecar pattern — local agent (Consul, dnsmasq, etc.) gives services a stable lookup interface.
• DNS-based — easiest; works everywhere; what k8s does internally.

Patterns to adopt

• ★ Use what your platform gives you. k8s DNS is fine. Don't run Consul on top of k8s "just to be safe."
• Health checks must reflect actual readiness. "Process is up" ≠ "ready to serve traffic."
• Don't put secrets in your KV — use secrets management.
• TTL / leases on registrations. Stale entries cause more outages than missing ones.
• Don't make discovery a runtime dependency for startup. Cache the last known good if discovery is briefly down.

Pick this if…

• k8s service discovery: k8s DNS (built-in).
• Non-k8s mixed env: Consul.
• Strong-consistency KV / coordination: etcd.
• Older Java / Big Data shop: ZooKeeper.
• You only need pub/sub + lightweight discovery: NATS.
• Tiny / homelab: mDNS or just DNS.