Log Aggregation

Storage backends

• ★ Loki (Grafana) — log store with label-based indexing; cheap to run; pairs with Grafana / Prometheus stack. The default OSS pick.
• ★ OpenSearch — Elasticsearch fork; full-text search; richer queries than Loki; heavier to run.
• Elasticsearch — back to a permissive license; the heavyweight.
• Graylog Open — log management built on OpenSearch.
• VictoriaLogs — VictoriaMetrics's log store.
• Quickwit — Rust-based log search; modern.
• Mezmo / Better Stack / Axiom — hosted; see observability.

Shippers / collectors

• ★ Vector (Datadog) — Rust-based; observability data pipeline. Very fast, transformable, supports log / metric / trace routing. Default for new setups.
• ★ Fluent Bit (CNCF) — lightweight C agent; tiny footprint; the default in Kubernetes DaemonSets.
• Fluentd — older sibling of Fluent Bit; bigger; less recommended for new code.
• Promtail (Grafana) — Loki's specific shipper; being deprecated in favor of Grafana Alloy.
• Grafana Alloy — successor to Promtail / Grafana Agent; OpenTelemetry-native.
• Filebeat / Logstash — Elastic's shippers; common but heavier.
• rsyslog / syslog-ng — classic Linux log forwarders; still relevant.

OpenTelemetry path

• ★ OpenTelemetry Collector — vendor-neutral; collect logs / traces / metrics via OTLP; export anywhere.
• OTLP logs — emerging standard; SDKs add it slowly.

Live log tailing

• ★ stern (k8s) — multi-pod log tail.
• kail — alternative.
• lnav — interactive log navigator for files; SQL-like queries.
• journalctl -f — systemd's built-in.
• tail / multitail — universal classics.

Patterns to adopt

• ★ Structured logs first. JSON logs, one event per line. Everything else flows from this.
• ★ Keep per-request correlation IDs in every log line. Match trace_id from your tracing.
• Don't index every field. Loki labels / Elasticsearch fields explode if you let users include their email in every label.
• Sample high-volume logs. Even at $0.50/GB, 100 GB/day adds up.
• Retention tiers. Hot (queryable, expensive) vs. cold (archival, cheap). Loki + S3 lets you do this automatically.
• Don't log secrets. Tokens / passwords / cookies leak via logs constantly.

Common architectures

Modern Grafana stack on k8s:

• Fluent Bit / Alloy DaemonSet → Loki (writes) → S3 / R2 backing store → Grafana (reads).

Heavy enterprise:

• Filebeat → Logstash (parse) → OpenSearch → Kibana.

OpenTelemetry-first:

• OTel Collector daemonset → OTLP exporter → Vector / Loki / Mimir / Tempo / paid backend.

Classic Linux:

• rsyslog → /var/log/* + remote rsyslog server (or Fluent Bit → Loki).

Pick this if…

• Default new setup, OSS: Loki + Fluent Bit (or Vector) + Grafana.
• Need rich full-text search: OpenSearch + Filebeat + Kibana.
• One pipeline for logs + metrics + traces: Vector or Grafana Alloy.
• Vendor-neutral: OTel Collector + OTLP exporters.
• Tail logs interactively: stern (k8s) or lnav (files).
• Hosted, low ops: Better Stack, Axiom, or Grafana Cloud.