Distributed Tracing

Trace stores / backends

• ★ Tempo (Grafana Labs) — object-store-backed; cheap; pairs with Grafana stack. The default OSS pick.
• ★ Jaeger (CNCF) — older heavyweight; great UI; supports Cassandra / Elasticsearch / Tempo as backend.
• Zipkin — OG; still fine; Twitter-derived.
• SigNoz — open-source APM; bundles tracing + metrics + logs in one product.
• Apache SkyWalking — broad APM platform; popular in Asia.
• Honeycomb — paid; best-in-class trace querying with high-cardinality fields. Free tier exists.
• Datadog APM, New Relic — enterprise paid.

Instrumentation (the actual SDKs in your code)

• ★ OpenTelemetry SDKs — vendor-neutral standard for traces / metrics / logs. The default for new code in 2026. SDKs in every major language.
• OTel auto-instrumentation — no-code instrumentation for HTTP / DB / message libraries.
• Sentry SDK — application monitoring; can also export OTel.
• Vendor-specific SDKs — Datadog / New Relic / Dynatrace; less recommended unless you're paying for that vendor.

Trace collection / pipeline

• ★ OpenTelemetry Collector — receives OTLP, processes (sample, redact, batch), exports to any backend. Default.
• Jaeger Agent / Collector — Jaeger-specific; OTel Collector now preferred.

Sampling

• Head-based sampling — decide at trace start (cheap, but biased).
• Tail-based sampling — decide after the trace finishes; keeps interesting traces (errors / slow). Run in OTel Collector or Refinery (Honeycomb).
• Adaptive sampling — adjust rates based on volume / error rate.

Visualization

• ★ Grafana — Tempo / Jaeger / Zipkin / OTLP backends; correlate traces with logs and metrics. Default UI.
• Jaeger UI — Jaeger-native; great for trace deep-dives.
• SigNoz UI — bundled.

Patterns to adopt

• ★ Instrument at the edges: HTTP server, HTTP client, DB driver, message queue. The SDKs auto-do most of this.
• ★ Propagate trace context (W3C traceparent) across every service-to-service call. Auto-instrumentation does this.
• Add custom span attributes for business-relevant data (user_id, org_id, endpoint).
• Tail sampling — keep all errors + 1% of normal is a sensible default.
• Don't trace background tickers — they explode trace counts.
• Correlate traces with logs by including trace_id in log lines.
• Limit trace depth — sometimes traces nest 50+ deep through internal helpers; pull common spans up.

Costs

• Tempo on R2 / S3 — extremely cheap; ingest cost dominates over storage.
• Hosted Honeycomb / Datadog — costs scale with span volume; sample aggressively.
• Jaeger on Cassandra / ES — heavy ops; consider Tempo unless you have specific reasons.

Pick this if…

• Default OSS: Tempo + Grafana + OTel Collector.
• Want a UI built around traces: Jaeger.
• One product for everything (traces + metrics + logs): SigNoz.
• Need high-cardinality querying: Honeycomb (paid).
• Already on Datadog / New Relic: that vendor's APM.
• Default instrumentation: OpenTelemetry SDKs in your services.