Chaos Engineering

The discipline: deliberately break things in controlled ways so you discover failure modes before they happen for real.

Open-source tools

★ Chaos Mesh (CNCF) — Kubernetes-native; CRD-driven faults: pod kill, network delay, IO chaos, time skew, kernel chaos. The default.
LitmusChaos (CNCF) — alternative; large hub of pre-built experiments.
Chaos Toolkit — language-agnostic framework with drivers.
Pumba — Docker chaos (network delay, pause, kill).
Toxiproxy (Shopify) — TCP-level proxy with configurable faults; great for apps.
kube-monkey — original Netflix Chaos Monkey-shape for k8s.

Toxiproxy — TCP-level faults for any app.
Service mesh fault injection — Istio / Linkerd can inject HTTP 5xx and delays.
fail-rs / failpoints (Rust), toxiproxy clients for many languages.

Recovery playbooks — simulate region failure, DB primary loss, certificate expiry, etc.
Run-once tools like stress-ng, tc qdisc add … netem, manual iptables blocks for ad-hoc tests.

★ Start in staging, not prod. Chaos Engineering ≠ "break prod for fun." Earn trust in lower envs first.
One variable at a time. A pod kill and a network partition is two experiments.
Pre-define abort criteria. "If error rate > 5%, halt the experiment."
Steady-state hypothesis. Define what "healthy" means for the system before the experiment.
Inject during business hours with team watching. Off-hours chaos = real outage with no one awake.
Chaos in CI — small failure injection in integration tests catches many regressions.

Single pod / node loss. Most basic test; many apps surprise you.
Network latency between services. Slow dependencies are common.
DNS failure / slow DNS. Apps without timeouts hang.
Disk full / IO slow. Logs / temp files / WAL growth.
Certificate expiry simulation. Time-skew tools like Chaos Mesh's TimeChaos.
DB primary loss / failover. Check that connection pooling reconnects.
Region / AZ outage. If you're multi-AZ.

Run chaos on systems with no monitoring. You won't see what broke.
Run chaos on systems without a rollback plan. "Stop it" must be one button.
Run chaos against customer data without isolation. Use shadow envs for invasive tests.