Lakehouse Table Formats

Open table formats turn a directory of Parquet files in S3 / R2 / GCS / Azure Blob into a transactional, queryable table — supporting ACID writes, schema evolution, time travel, and concurrent compute engines. They sit underneath Data Warehouses, Stream Processing, and Query Engines & Federated SQL. The catalog layer above them is in Data Catalog & Lineage.

The three (almost-two) formats

• ★ Apache Iceberg — the dominant open format in 2026; Apache 2.0. REST catalog spec, branching, compaction, hidden partitioning. Backed by Snowflake, Cloudflare, AWS (Glue + S3 Tables), Apple, Netflix. Pick this unless you have a specific reason not to.
• ★ Delta Lake — Apache 2.0; Databricks-origin but governed by the Linux Foundation. Strong on Spark; Delta Lake 4 / "Delta UniForm" (2024–2025) interops with Iceberg readers. Default if you already live in Databricks; otherwise consider Iceberg.
• Apache Hudi — Apache 2.0; mutation-friendly (upserts, MoR), strong on streaming-CDC ingestion; smaller momentum in 2026 but still actively developed.
• Apache Paimon (formerly Flink Table Store) — Apache 2.0; Flink-native streaming-first lakehouse format; growing inside the Flink ecosystem.

Catalogs that speak Iceberg

• ★ Apache Polaris (Snowflake) — Iceberg REST Catalog; Apache 2.0.
• ★ Unity Catalog OSS (Databricks) — Iceberg + Delta REST Catalog; Apache 2.0.
• Project Nessie — Git-like branching over Iceberg tables; Apache 2.0.
• AWS Glue Data Catalog — paid; default on AWS.
• Lakekeeper — Iceberg REST Catalog Rust implementation; Apache 2.0.
• See Data Catalog & Lineage for the full catalog list.

Compute engines that read these formats

• DuckDB Iceberg / Delta extensions — single-node SQL on a lakehouse table.
• ClickHouse Iceberg / Delta — compute over object-store tables.
• Spark, Flink, Trino, Presto, Dremio — full-stack engines.
• Snowflake, BigQuery, Redshift Spectrum — managed support for Iceberg (Delta via UniForm).
• Polars + DeltaLake-rs — Python-native reads of Delta from a script.
• PyIceberg — pure-Python Iceberg client; Apache 2.0; great for custom ingestion / inspection.

Storage layers

• Amazon S3, Google Cloud Storage, Azure Blob Storage — the cloud defaults.
• Cloudflare R2 — S3-compatible, no egress fees; AWS S3 Tables and Cloudflare both shipped Iceberg-native object-store features in 2024–2025.
• MinIO — self-host S3 API; AGPLv3.
• Garage, SeaweedFS, Ceph RGW — alternative S3-compatible self-host.
• Tigris — global S3-compatible store optimized for distributed reads.

Patterns to adopt

• Pick Iceberg by default. It has the most catalog implementations, the most engine support, and the strongest momentum in 2026.
• REST Catalog > Hive Metastore. Avoid HMS for new projects unless an engine forces it.
• Compaction is non-optional. Streaming writers create many small files; schedule compaction (Iceberg rewrite_data_files) in Data Orchestration.
• Time-travel for safety, not normal queries. It's a snapshot mechanism; expensive as a query pattern.
• Branching for migrations. Iceberg branches / Nessie give you "preview a destructive change without committing."
• Z-ordering / clustering. Cluster on common predicate columns; large speedups on column-pruned scans.
• Schema evolution is not free renaming. Iceberg supports add/drop/reorder; rename works but downstream views need attention.

License & licensing watch-outs

• Iceberg / Delta / Hudi / Paimon are all Apache 2.0 — fully permissive.
• Catalogs vary: Polaris / Unity Catalog OSS / Nessie are Apache 2.0; Glue is paid AWS; some commercial REST catalogs (Tabular, Lakehouse) have proprietary control planes.

Pick this if…

• Default in 2026: Iceberg + Polaris (or Unity Catalog OSS) on R2 / S3.
• You're already on Databricks: Delta with UniForm enabled.
• Streaming-CDC ingestion as the primary write pattern: Hudi or Paimon.
• Single-node Python / SQL: DuckDB or PyIceberg directly against object storage.
• No cloud bill: MinIO + Iceberg + Trino or DuckDB.
• Multi-engine federation: Iceberg with a REST catalog; every engine reads it.