Data Exploration & EDA

The "I have a CSV / Parquet / table — what's in it?" layer. Pairs with Notebooks (where exploration usually happens), Data Quality & Testing (formal assertions), and Data Warehouses (when the data is too big for one machine).

DataFrame engines (Python)

• ★ Polars — Rust core; Arrow-backed; lazy + streaming engine; multi-threaded; MIT. The most-recommended pandas alternative in 2026 — usually 5–50× faster on the same hardware. Pairs naturally with DuckDB.
• ★ pandas — the default; BSD-3. Still ubiquitous; pandas 2.x added PyArrow-backed types and copy-on-write. Use it where libraries assume it.
• DuckDB Python API — duckdb.sql("SELECT * FROM 'file.parquet'") returns a DataFrame; Polars / pandas / Arrow interop. MIT.
• Modin — drop-in pandas API on top of Ray / Dask; Apache 2.0; great for "I have pandas code and need it bigger."
• Dask — distributed pandas / Numpy; BSD-3.
• Vaex — out-of-core DataFrames; less active in 2026 than Polars + DuckDB.
• cuDF (RAPIDS) — pandas-compatible on GPUs; Apache 2.0; great if you have NVIDIA hardware.

DataFrame in other languages

• dplyr / data.table (R) — the original tidy and high-performance APIs; mature.
• DataFrames.jl (Julia) — fast, idiomatic.
• @dprint/dataframe-js, arquero, DuckDB-Wasm — JavaScript-side; great for browser-native analytics.

SQL-on-files

• ★ DuckDB CLI — duckdb -c "SELECT … FROM 'data.parquet'" is the fastest interactive SQL on files; MIT. The unbeatable "ad-hoc query a file" tool.
• sqlite3 / csvsql / q — SQL on CSVs; older but useful.
• chDB — embedded ClickHouse, similar UX to DuckDB; Apache 2.0.
• textql, octosql — Go-based SQL on CSV / JSON / log files.

TUI / shell tools

• ★ VisiData — terminal spreadsheet; opens CSV, Parquet, SQLite, JSON, even gzipped logs; GPL-3. Indispensable on a server.
• xsv / csvkit / miller (mlr) — fast CSV manipulation in shell; MIT-ish licenses.
• jq + yq — JSON / YAML; not tabular but constantly useful.
• fx, gron — JSON exploration TUIs.
• up — pipe-and-edit shell tool; great for iterative ETL.

Profiling / one-shot reports

• ★ ydata-profiling (formerly pandas-profiling) — one-line HTML report with summaries, missing-value heatmaps, correlations. MIT. The default "what's in this DataFrame?" tool.
• Sweetviz — comparative profiling (train vs. test); BSD-3.
• AutoViz — auto-EDA charts; Apache 2.0.
• D-Tale — interactive pandas explorer in a browser tab; LGPL-2.1.
• Lux — auto-recommends visualizations from a DataFrame; Apache 2.0.

Notebook-adjacent

• See Notebooks for Jupyter / Marimo / Quarto.
• Datasette — exposes any SQLite DB (or CSV → SQLite) as a queryable web app; Apache 2.0. Great for quick exploratory web UIs over a dataset.
• Rill Data — sub-second slice-and-dice on Parquet / S3; Apache 2.0; see BI & Dashboards.

Patterns to adopt

• Reach for Polars or DuckDB first. pandas only when a library demands it.
• Lazy is free speed. Polars LazyFrame and DuckDB pushdown both prune and parallelize.
• Profile before plotting. ydata-profiling answers "what types, what nulls, what cardinality" in one cell.
• Don't load what you can scan. DuckDB / Polars stream Parquet from S3 without ever materializing it.
• Cache intermediates as Parquet. Faster reload than pickle; portable across Polars / pandas / DuckDB / R.
• Visidata over pandas in shell. When you've SSH'd into a box and need to look at a file, VisiData is faster than spinning up Python.

Pick this if…

• Default Python DataFrame in 2026: Polars.
• Default SQL-on-files: DuckDB.
• One-shot "look at a file" in a terminal: VisiData.
• One-shot "look at a DataFrame" in a notebook: ydata-profiling.
• Your code already uses pandas: stick with it; modernize types to PyArrow.
• Browser-native EDA: DuckDB-Wasm or arquero.
• GPU-accelerated: cuDF.