What is a Parquet file? Columnar storage explained

Apache Parquet is an open-source columnar file format released under the Apache License 2.0. A CSV stores a table row by row as plain text. Parquet stores the values of each column together in binary, with the column types and statistics written into the file.

Twitter and Cloudera started the project in 2013, and it became an Apache top-level project on April 27, 2015. The specification is open, so DuckDB, pandas and pyarrow all read and write the same files.

A Parquet file starts and ends with the 4-byte magic string PAR1. In between, the rows are split into row groups, each row group holds one column chunk per column, and each column chunk is made of pages. The format docs suggest row groups of 512 MB to 1 GB and data pages of about 8 KB.

The file metadata comes after the data, followed by a 4-byte metadata length and the closing PAR1. A reader starts at the end of the file, reads the schema, the location of every row group and the min/max statistics, and then fetches only the byte ranges it needs.

A query that needs 3 of 50 columns reads only those 3 column chunks. A filter is compared with each row group's min/max statistics, so row groups that can't match are skipped. And because the values of one column sit together, they encode and compress better than rows of mixed text.

Each page is first encoded with PLAIN, dictionary (RLE_DICTIONARY), RLE, one of the DELTA encodings or BYTE_STREAM_SPLIT, and then compressed with a codec:

• SNAPPY: the default in pandas, pyarrow and DuckDB
• ZSTD: what rows.page uses when it exports Parquet
• GZIP and BROTLI
• LZ4_RAW, the replacement for the deprecated LZ4 (LZO is defined too)
• UNCOMPRESSED

DuckDB measured the difference in December 2024 on the TPC-H scale factor 20 lineitem table. The CSV was 15.95 GB, the Parquet file 3.78 GB with Snappy and 3.22 GB with zstd. TPC-H query 1 took 6.72 s on the CSV and 0.88 s on Snappy Parquet, and query 21 took 19.95 s versus 2.08 s.

Yes. The schema sits in the file metadata, so every column has a fixed type such as integer, double, string, date or timestamp, and nested lists and maps are part of the format too. A CSV stores everything as text, so every tool that reads it has to guess the types again.

When you open a Parquet file in rows.page, the Columns panel shows each column's type, distinct count, share of empty values, min and max for numbers and dates, and a histogram or the top 10 values.

• Use Parquet for large files, for repeated queries that touch a few columns, and when column types must survive the trip between tools.
• Use CSV when someone needs to read the file in a text editor or open it in a spreadsheet. Parquet is binary, so cat shows nothing useful and you can't fix a value by hand.

Size limits differ in the browser too. rows.page holds CSV, TSV, JSON and JSONL files to 1.5 GB per browser tab, but local Parquet files skip that check because DuckDB reads them in place.

rows.page

1. Drop a .parquet or .pq file on rows.page. DuckDB-WASM reads it in your browser, and nothing is uploaded unless you press Share.
2. Filter with country:NL or price:>20, or run SQL against the table data.
3. Click Export to save the view as CSV, JSON, JSONL, Markdown (first 1,000 rows) or zstd Parquet.

To open a public file without downloading it, put its URL after rows.page/, for example https://rows.page/https://example.com/data.parquet.

DuckDB

The DuckDB CLI or Python package reads Parquet with plain SQL and writes it with COPY:

-- Read a Parquet file
SELECT * FROM 'data.parquet' LIMIT 100;

-- Convert a CSV to zstd-compressed Parquet
COPY (SELECT * FROM 'data.csv') TO 'data.parquet' (FORMAT PARQUET, COMPRESSION ZSTD);

pandas

pandas needs pyarrow or fastparquet installed and loads the whole table into memory. df.to_parquet() writes Snappy-compressed files by default.

import pandas as pd

df = pd.read_parquet("data.parquet")
df.to_csv("data.csv", index=False)