Read Fewer Parquet Columns With DuckDB, Then Inspect the Plan

Article · Measurement

A Parquet export contains an identifier, a region and a payload. The report needs ten identifiers. Reading every column into Rust and then filtering the rows works, but it asks the application to carry data it never needed.

DuckDB can push a projection and a filter into its Parquet scan. The projection tells the reader which columns the query needs. The filter may also let it avoid row groups whose metadata rules them out. These are useful mechanisms to inspect before rewriting the report or adding another data system.

There is a trap in describing them: a plan that contains a filter is not a measurement of how many bytes the operating system read. This example keeps those claims separate. It generates a small file, checks the expected count and sum, and prints an estimated EXPLAIN plan. It does not measure an executed scan or a speedup.

Start with a file whose contents we know

The fixture writes ten thousand ordered identifiers to a local Parquet file. It requests row groups of 2,048 rows, giving the reader more than one group to consider. Region alternates between two values. Payload contains a repeated string, so it is deliberately easy to compress; its character count is not a claim about physical file size.

The query selects identifiers from 8,000 through 8,009. Here is the inspection and correctness check:

fn query_file(c: &Connection, path: &str) -> Result<()> {
    let mut explain =
        c.prepare("EXPLAIN SELECT id FROM read_parquet(?) WHERE id>=8000 AND id<8010")?;
    for row in explain.query_map(params![path], |row| row.get::<_, String>(1))? {
        println!("EXPLAIN plan:\n{}", row?);
    }
    let (count, total): (i64, i64) = c.query_row(
        "SELECT COUNT(*), CAST(SUM(id) AS BIGINT) FROM read_parquet(?) WHERE id>=8000 AND id<8010",
        params![path],
        |row| Ok((row.get(0)?, row.get(1)?)),
    )?;
    assert_eq!(count, 10);
    assert_eq!(total, (8000..8010_i64).sum::<i64>());
    let groups: i64 = c.query_row(
        "SELECT COUNT(DISTINCT row_group_id) FROM parquet_metadata(?)",
        params![path],
        |row| row.get(0),
    )?;
    assert!(groups > 1, "fixture needs more than one row group");
    println!("checked selected rows={count}, sum={total}, file row groups={groups}; plan is not a byte-read measurement");
    Ok(())
}

The full source is duckdb_parquet.rs. It pins the Rust wrapper to 1.20000.0 and checks the linked DuckDB engine is v2.0.0. The generated file lives in a temporary directory owned by the fixture.

The aggregate query must return count ten and the expected integer sum. It is a separate query from the SELECT whose plan is printed. The metadata query also requires multiple row groups, so a setup error cannot leave us discussing pruning over a one-group file. These remain expected checks for the recorded engine run.

DuckDB's Parquet reading documentation explains projection and filter pushdown, including the role of file statistics in skipping work. The Parquet metadata functions let us inspect the file's layout instead of inferring it from its extension.

Read the scan part of the plan

Find the file scan in the printed plan. Check the projected columns and the interval condition near that scan. The query needs the identifier and does not ask for payload. If a later version or a changed query produces an unexpected plan, keep that output and investigate before claiming the same behavior.

The fixture prints the estimated plan for inspection, without freezing its exact text or operator names. Plain EXPLAIN does not execute the query; EXPLAIN ANALYZE does. To check pruning during execution, DuckDB 2.0 also exposes query.total_row_groups_scanned and query.total_row_groups_to_scan in its profiling metrics. Those measurements are not collected by this fixture.

It is also possible for a filter to reach the scan and still skip little. A row group's minimum and maximum can cover the requested value even when few matching rows exist inside it. Randomly distributed values often make a broad interval in every group. The ordered identifiers in this fixture make the example easy to reason about; they are not representative of every export.

For a real dataset, inspect the statistics of the column used by the filter. Then compare the number of relevant groups with the total. Group size, ordering and the actual predicate all matter. A claim such as “Parquet only reads matching rows” is too strong: the engine may need to read and decode data within a group before deciding which rows qualify.

Keep the useful SQL shape

Project the columns needed by the report as early as its query allows. Apply selective conditions where the engine can reason about them. If a query transforms a field before filtering, inspect the plan rather than assuming that the transformed condition can use the file's original statistics.

Selecting every column and discarding payload in Rust throws away the projection opportunity. Reading every row into a vector and then filtering also moves memory pressure into the application. The small CSV report follows the same principle: let the query return the result the caller needs, with a clear schema at the boundary.

This does not mean every report needs one complicated SQL statement. A readable view or a few named steps can preserve the intent. Recheck the resulting plan when those steps change. After refactoring a view or helper, check that the scan still requests only the needed columns and receives the intended filter.

Expand the experiment without changing its claim

A useful next comparison would generate the same values in shuffled order and inspect the row-group statistics again. Keep the row count, schema and query fixed. That isolates file organization more clearly than changing the data, machine and query together.

For remote object storage, also record request counts, bytes and caching behavior. Local disk observations do not automatically predict network costs. A list of input files should be stable too: a broad wildcard can include a new export part between runs and change both the answer and the work.

Schema evolution is another separate check. If one file changes a field type or omits a column, decide how the report handles that change rather than silently combining whatever it finds. Use the file metadata and explicit schema options appropriate to that contract.

If the scan lists payload after a report change, find which expression needs it. If the filter no longer reaches the scan, inspect the transformation that moved it. When both look right, use executed row-group and I/O measurements to decide whether this file layout actually saves work. Keep the file and cache conditions recorded with those measurements.