Parquet Is Not Free: Layout, Encoding and Scan Cost

Q: Why is my Parquet query still slow if Parquet is columnar?

Columnar layout gives you column pruning for free — the engine reads only the columns your query references, so selecting three columns from a forty-column table already avoids most of the data. What it does not give you for free is row pruning: skipping the rows whose values cannot satisfy your predicate. Row pruning depends entirely on physical layout the writer controls. The engine reads the footer, checks each row group's min/max statistics for your filter column, and skips row groups whose range cannot contain the value. If the data was written in arrival order rather than sorted on the column you filter, that column's min/max is smeared across every row group and always overlaps your predicate, so nothing gets skipped. The result is a full scan of the columns you selected even when the predicate matches a tiny fraction of rows. The fix is not a faster engine; it is writing the file sorted on the query's filter column, sized into sensible row groups, with the page index emitted so the engine can skip below the row-group level.

Q: What is the difference between encoding and compression in Parquet?

They are different mechanisms that people constantly conflate. Compression — Snappy, Gzip, Zstd — takes the encoded bytes and squeezes them smaller, trading CPU on read to inflate them. It reduces bytes at rest and does nothing to how many values the engine has to decode. Encoding changes the actual on-disk representation of the column before compression runs. Dictionary encoding replaces repeated values with small integer codes and stores the dictionary once per column chunk, which both shrinks low-cardinality columns and lets some engines evaluate predicates against the dictionary without decoding every value. Run-length encoding collapses runs of identical values, which is why sorting a low-cardinality column makes it both smaller and faster to scan. Delta encoding stores differences between consecutive values, ideal for sorted timestamps and monotonic IDs. The practical consequence is that switching your compression codec optimises storage cost, while choosing the right encoding and sort order optimises how much data the engine decodes on every query — a scan-cost lever, not a storage-cost one.

Q: What row group size should I use in Parquet?

The Apache Parquet project recommends large row groups in the 512 MB to 1 GB range, ideally aligned so that one row group maps to one storage block, with a data page size around 8 KB. The reasoning is a balance of two failure modes. Row groups that are too small — the extreme being thousands of tiny files with one small row group each — impose a fixed per-file and per-row-group cost: a footer read and an object-store round trip for almost no data, which swamps any columnar benefit. Row groups that are too large — the extreme being a single row group spanning a multi-gigabyte file — make the row group the smallest unit the engine can skip, so predicate pushdown can never prune within it and you lose row pruning entirely. The 512 MB to 1 GB target keeps the metadata overhead low while still giving the engine many independently-skippable units per file. Page size around 8 KB then provides the finer skip granularity when the page index is present. These are recommendations rather than hard defaults, and the right number depends on your object store and query engine, but starting far from this range is the most common layout mistake.

Q: What is the Parquet page index and why does it matter?

The page index is an optional pair of structures — the column index and the offset index — stored in the file footer that record per-page minimum and maximum values and the byte offset of each page within a column chunk. It matters because without it the smallest thing an engine can skip is an entire row group, which may be hundreds of megabytes. With the page index, the engine can consult per-page statistics and seek directly to and decode only the individual pages (around 8 KB each) whose min/max range can contain the value your predicate is looking for, skipping the rest of the chunk. On selective queries this is frequently the single largest saving, because a filter that matches a small, clustered set of rows can be served by reading a handful of pages instead of the whole column chunk. The catch is that some writers do not emit the page index by default, so teams silently forfeit sub-row-group skipping without realising it. Like row-group statistics, the page index only helps when the data is clustered — the values you filter on must be physically grouped so that most pages can be excluded, which again comes back to writing the file sorted on the column your queries actually filter.

Q: Why does sort order affect Parquet performance so much?

Sort order is the hinge that makes every skipping mechanism work, because both row-group statistics and the page index prune by value range, and range pruning only helps when values are physically clustered. When a table is sorted by a column, that column's values within any given row group span a narrow min/max range, so a predicate on it can exclude most row groups immediately; the same is true page by page. When the table is written in arrival order, that column's values are scattered, every row group's min/max spans nearly the full domain, and no predicate can exclude anything — the statistics exist but are useless. Sort order also drives encoding efficiency: a sorted low-cardinality column collapses into long run-length-encoded runs, and a sorted timestamp column delta-encodes to near nothing, so sorting cuts both scan cost and storage at once. The unavoidable constraint is that a single physical copy can only be sorted one way, so it can be optimised for one or two access patterns, not all of them. Choosing the sort key is therefore a decision about which queries you are optimising for, and queries filtering on other columns will fall back to fuller scans unless you maintain a differently-sorted copy.

Q: What is the small-files problem with Parquet?

The small-files problem is what happens when a pipeline writes many tiny Parquet files instead of fewer large ones — classically a streaming or micro-batch job that emits one file per batch, producing thousands of files a few megabytes each. Every Parquet file carries a footer that must be read before any data, and every file is a separate object-store request with its own latency. When files are tiny, this fixed per-file overhead dominates: the engine spends its time on footer reads and round trips rather than on scanning data, and the columnar advantages are drowned by coordination cost. Query planners also struggle to parallelise efficiently across a swarm of tiny files. The fix is compaction — a maintenance job that rewrites the small files into targets in the recommended 512 MB to 1 GB range with healthy row groups — and this is one of the main jobs open table formats like Iceberg and Delta automate for you. It is the highest-leverage first move for a slow lakehouse, because it needs no query rewrites and no schema changes: you simply stop paying the per-file tax by having fewer, larger, well-formed files.

Q: Does switching from Snappy to Zstd make my queries faster?

Usually not in the way people hope. Changing the compression codec from Snappy to Zstd typically reduces the bytes stored at rest, because Zstd achieves higher compression ratios, and it can reduce the bytes transferred from object storage, which helps if your bottleneck is network throughput. But it does nothing to change how many rows or values the engine must decode, and it costs more CPU to decompress. If your query is slow because the layout forces a full scan — no useful sort order, missing statistics, no page index — a better codec compresses the same excessive data more tightly and still reads all of it. The levers that actually cut the scan are the layout ones: sorting on the filter column so statistics prune, sizing row groups so pruning has granularity, choosing encodings so columns decode cheaply, and emitting the page index for sub-row-group skipping. Compression is a real storage-cost lever and worth tuning, but treating a codec change as a query-performance fix is a common misdiagnosis; it addresses bytes at rest, not the amount of data your query is forced to touch.

Q: How do Iceberg and Delta relate to Parquet layout?

Parquet is the file format that stores the actual column data; Iceberg and Delta Lake are table formats that sit above the files and manage them as a coherent, versioned table. The relationship matters for layout because the failures that make Parquet slow — small files, drifting sort order, missing or stale statistics, uneven file sizes — are exactly the properties a table format can own and maintain as a background operation. Iceberg and Delta run compaction to rewrite small files into well-sized ones, can cluster or sort data on chosen columns, track file-level and column-level statistics for pruning, and expose maintenance operations so the physical layout stays healthy as new data arrives instead of degrading with every write. In other words, they turn physical layout from an accident of whoever wrote the last batch into a governed property of the table. You still need to make the right choices — which columns to sort or cluster on, how aggressively to compact — but the table format gives you the machinery to enforce them continuously. Underneath, the bytes are still Parquet, and every principle about row groups, encodings, footer statistics and the page index continues to apply.

Satyam Kumar

ブログに戻る

ai-architecture

Parquet Is Not Free: Encoding, Layout and the Scan You Didn't Need

By Satyam KumarAugust 21, 202610 min read

parquet columnar storage predicate pushdown row group data lakehouse apache iceberg delta lake query performance data engineering dictionary encoding page index column pruning object storage analytics engineering duckdb 2026

Parquet Is Not Free: Encoding, Layout and the Scan You Didn't Need

Frequently Asked Questions

この記事を共有する

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

AI＆クラウドアーキテクト。数百万人にスケールするシステム構築を支援。

LinkedIn GitHub

Parquet Is Not Free: Encoding, Layout and the Scan You Didn't Need

Frequently Asked Questions

この記事を共有する

Comments

Leave a comment