What Is Parquet? Columnar File Format vs CSV, Avro & ORC

Key takeaways

• Apache Parquet stores data by column, not by row. Analytical engines read only the columns a query names.
• On an 11.2M-row NYC taxi dataset in DuckDB v1.5.5 (Apple M3 Max, OS-cached, median of 3), Parquet-Zstd was 164 MB vs 1.09 GB of CSV (6.8× smaller) and 22–60× faster. count(*) was ~160× faster because it answers from row-group metadata.
• Use CSV for small, human-readable interchange. Use JSON for nested event payloads. Use Avro for row-oriented streaming. Use ORC if you are deep in Hive. Use Parquet as the default for analytics.
• Iceberg, Delta Lake, and DuckLake are table formats. They usually store data as Parquet files and add transactions, schema history, and time travel. Parquet itself does none of that.
• Skip Parquet for tiny files, row-level updates, single-record streaming appends, or anything a person needs to open in a text editor.

What is a Parquet file?

A Parquet file is a binary table stored column-wise. CSV stores row1.col1, row1.col2, … then the next row. Parquet stores every value of col1, then every value of col2. Same table, different layout.

That is why it compresses well and why it scans fast. Values in one column share a type and often a small set of repeated values, so dictionary encoding and run-length encoding shrink the column before any general compressor runs. A query that aggregates one column never has to decode the other nineteen.

Inside the file:

• Row groups split the table horizontally. A row group is the unit an engine can skip entirely.
• Column chunks hold one column inside a row group. This is the unit column pruning actually skips.
• Pages sit inside a column chunk. Compression and encoding apply at page level.
• The footer stores the schema plus per-chunk statistics (min, max, null count). Engines use those stats for predicate pushdown: if you filter trip_distance > 10 and a chunk's max is 8, that chunk is never read.

Parquet is an Apache Software Foundation project. Spark, Hadoop, Presto/Trino, DuckDB, pandas (via PyArrow), and Arrow all read and write it. That ecosystem is the practical reason it won over ORC outside Hive-centric stacks.

Parquet format v2 added encodings aimed at numerics (delta binary packed, byte-stream split). Most engines, including DuckDB, read both v1 and v2. You do not pick Parquet vs "Parquet v2" as a product; you pick a writer setting.

How much smaller and faster is Parquet than CSV?

We exported the same 11,198,026 NYC TLC yellow taxi trips (official TLC Parquet, Jan–Mar 2025, 20 columns) to CSV and to Parquet with Snappy and Zstd, then queried both in DuckDB v1.5.5 on an Apple M3 Max with 36 GB RAM. Files were OS-cached for both formats. Timings are the median of three runs. The full method, results, and commands to reproduce it on your own machine are on our CSV-vs-Parquet benchmark page.

File size:

Query speed, CSV vs Parquet-Zstd:

Two caveats worth keeping attached to these numbers. DuckDB's parallel CSV reader is already fast (~0.5 s over 1.1 GB), so this is not "CSV is slow in DuckDB." The Parquet advantage is columnar pruning, compression, and metadata. And count(*) is the extreme case: Parquet answers it from row-group metadata without scanning data. The fair lower bound is the all-columns query at 22×. Single-column aggregates land around 60×. A usable rule: 10–100× faster for analytics, depending on how many columns the query touches.

How does Parquet compare to CSV, JSON, Avro, and ORC?

CSV is a text table. Fine for a 2 MB export you will open in a spreadsheet. It has no types, no compression, and every query reads every column.

JSON is a document, not a table. Good for events with optional nested fields. Bad as a warehouse format: you pay to parse text, and column pruning is weak unless you extract fields into a real table first.

Avro is the row-oriented binary counterpart to Parquet. It shines when you write one record at a time and the schema will change (Kafka, ingestion). It is the wrong default for SELECT avg(col) FROM 200_columns.

ORC is Parquet's closest peer: columnar, compressed, stats in the file. It started in Hive and still wins some compression bake-offs there. Parquet is the default in Spark, DuckDB, pandas, Arrow, and most cloud warehouses. Unless you already live in Hive, pick Parquet for interoperability.

Why is Parquet faster for analytics?

The speed comes from skipping work, not from a faster parser:

1. Column pruning. SELECT avg(trip_distance) reads one column chunk per row group. The other 18 columns stay on disk. CSV cannot do this.
2. Predicate pushdown. The footer stores min/max per chunk. WHERE trip_distance > 10 skips every chunk whose max is ≤ 10 — but only if such a chunk exists. Our taxi file is ordered by pickup time, so long trips appear in every row group, no chunk has a max under 10, and nothing gets skipped: the 38× on the filtered sum is column pruning and compression, same as the other rows. Pushdown pays off when the filter column correlates with file or row-group order. A date filter on time-partitioned files can skip most of the dataset before reading a byte.
3. Encoding, then compression. Dictionary encoding collapses low-cardinality columns (passenger count, vendor id, payment type) before Snappy or Zstd runs. Same-type columns compress better than mixed-type rows.

count(*) is a fourth, narrower trick: the row count lives in metadata, so DuckDB does not scan the file. Treat that 160× as a format feature, not as typical query speed.

How do you read and write Parquet in DuckDB?

DuckDB queries Parquet in place. No load step.

Read a file by path:

Copy code
SELECT vendorid, trip_distance, total_amount
FROM 'trips.parquet'
WHERE trip_distance > 10;

Read many files, including Hive-partitioned directories. Partition columns come from the folder names (year=2025/month=3/…):

Copy code
SELECT *
FROM read_parquet(
  's3://bucket/trips/*/*/*.parquet',
  hive_partitioning = true
)
WHERE year = 2025 AND month = 3;

Write with Zstd. DuckDB's default Parquet codec is still Snappy; set Zstd when you care about size (6.8× vs 5.1× on the taxi data):

Copy code
COPY (
  SELECT * FROM 'trips.csv'
) TO 'trips.parquet' (
  FORMAT PARQUET,
  COMPRESSION ZSTD
);

Inspect what the writer actually put in the file:

Copy code
SELECT
  path_in_schema,
  compression,
  stats_min_value,
  stats_max_value
FROM parquet_metadata('trips.parquet');

MotherDuck uses the same functions. Point read_parquet at a local path, HTTPS URL, or S3 prefix and CREATE TABLE AS if you want the result managed in MotherDuck.

Is Parquet the same as Iceberg, Delta Lake, or DuckLake?

No. Parquet is a file format. Iceberg, Delta Lake, and DuckLake are table formats. A table format is a catalog of many data files plus a transaction log: which files belong to the table right now, what the schema is, and how to time-travel.

Most Iceberg, Delta, and DuckLake tables store their data as Parquet. The table format adds what a single Parquet file cannot do: ACID commits, concurrent writers, schema evolution that does not rewrite history, and SELECT * FROM t AT (TIMESTAMP => …).

DuckLake keeps that catalog in a SQL database (Postgres, MySQL, DuckDB, or MotherDuck) instead of in JSON/Avro files on object storage. The data is still Parquet. If you have a pile of Parquet files and you need a table, you want a table format. If you have a table and you need a file, you want Parquet.

Which format should you use?

• If you are storing a rectangular table that will be queried with SQL, write Parquet. That is the default for DuckDB, Spark, pandas, and cloud warehouses.
• If a person has to open the file, write CSV. Keep it small.
• If the payload is an event with optional nested fields, JSON (or JSON → Parquet once the shape stabilizes).
• If you are publishing a Kafka topic or a row-at-a-time ingestion pipe, Avro.
• If the lake already speaks ORC and Hive, stay on ORC. Do not convert for sport.
• If multiple writers need transactions, time travel, or schema history on object storage, put Parquet under Iceberg, Delta Lake, or DuckLake. Do not pretend a directory of Parquet files is a table.

When should you not use Parquet?

Parquet is a bad fit when the unit of work is a row, a person, or a tiny file.

• Small files. Not because of size — Parquet's schema and footer overhead only dominates once a file shrinks to a few hundred bytes, and even a small extract usually compresses smaller as Parquet. Skip it because the speed gain on a file that small is nothing anyone will notice, and a CSV you can sanity-check with head beats a binary file you cannot.
• Row-level updates and deletes. Parquet files are immutable. Changing one row means rewriting the row group or the file. Use a database, or a table format that can mark deletes and compact later.
• Single-record streaming appends. Opening a Parquet writer for one event is the wrong shape. Buffer, then flush a row group, or use Avro/JSON on the wire and compact to Parquet in batch.
• Human inspection. You cannot cat a Parquet file. Excel in Microsoft 365 can import Parquet through Get Data; double-clicking a .parquet still does not open it like a CSV. If the audience is a person with Excel and no Power Query habit, send CSV.
• Point lookups of one row by primary key. That is OLTP. Use Postgres. Parquet is built for scans, not single-row seeks.

DuckDB and MotherDuck inherit these limits when they query files directly. They do not magically make a Parquet directory updatable. If you need updates, load the data into a table (or a DuckLake) and treat the Parquet as the import format.