Distributed Storage

Object storage (S3-compatible self-hosted)

• ★ MinIO — single binary; production-grade S3-compatible store; erasure coding; ETL features. The default.
• SeaweedFS — broader; supports S3 + HDFS + filesystem; more features but more complexity.
• Garage — newer; lighter than MinIO; good for small clusters.
• Ceph RGW — S3-compatible gateway on top of Ceph; for serious scale.
• Wasabi / Backblaze B2 — managed S3-compatible; cheap; not really self-hosted but worth listing.

Block / cluster storage

• ★ Rook-Ceph — Ceph deployed as a k8s operator; the heavyweight production choice for k8s storage.
• Ceph (raw) — ancestor; runs on bare metal / VMs without k8s.
• OpenEBS Mayastor — modern alternative; NVMe-over-Fabrics; performant.
• Portworx — commercial; common in big enterprise.
• Longhorn (Rancher) — see k8s-storage; simpler than Ceph.

Distributed filesystems (POSIX-ish)

• CephFS — POSIX FS on top of Ceph.
• GlusterFS — older; still works; less recommended for new setups.
• JuiceFS — POSIX FS over object storage; cheap; great for ML workloads.
• MooseFS / LizardFS — older.

Hybrid / cache layers

• Alluxio — caching layer in front of object stores for analytics.
• JuiceFS — also fits this category.
• Cunoia / Quobyte — proprietary alternatives.

Multi-region / geo

• MinIO active-active replication — between sites.
• Ceph multi-site / RGW — geo-replication.
• Cloudflare R2 multi-region — managed.

Choosing between the heavyweights

• Ceph — battle-tested, complex; needs careful planning; pays off at scale.
• MinIO — simpler to operate; great for "I want S3 in my cluster"; not the same scale ceiling.
• Rook-Ceph — Ceph + k8s automation; if you're already on k8s.
• Longhorn — much simpler than Ceph; ideal for small / mid clusters.

When you don't need this layer

• Single-VPS workload — local disk is fine; back up to R2 / B2.
• Small k8s cluster — Longhorn solves it.
• Cloud-managed — use the cloud's storage; don't reinvent.

Patterns to adopt

• ★ Match storage tier to workload. Block (Longhorn / Ceph RBD) for DBs; object (MinIO / Ceph RGW) for app uploads; FS (CephFS / NFS) for shared static.
• Erasure coding > replication at scale; cheaper, similar durability.
• Plan for failure modes — pull a node, simulate disk death, time the recovery.
• Backups still matter. Distributed storage protects against node loss, not against rm -rf or ransomware.

Pick this if…

• Self-host S3-compatible: MinIO.
• Lighter MinIO alternative: Garage.
• k8s + serious scale: Rook-Ceph.
• k8s + small/medium scale: Longhorn.
• POSIX FS over S3 (cheap, big): JuiceFS.
• Already on a managed cloud: their object + block storage; this category is mostly N/A.