Streaming FastQC in polars-bio: compatible and scalable

FastQC is the de-facto first look at any sequencing run — but it is a single-threaded Java tool, and the fast Rust reimplementations tend to trade away correctness. polars-bio now runs the full FastQC module suite as a single streaming pass over FASTQ, computed on Apache DataFusion: FastQC 0.12.1-compatible under explicit parity checks (bit-exact for deterministic/count modules, tolerance-checked for floating metrics), and a fraction of the time and memory.

What shipped

Each QC module is a streaming accumulator (update → merge → finalize) folded over Arrow batches, so the work parallelizes across partitions and merges deterministically. All 12 core FastQC modules are implemented and validated against FastQC 0.12.1 golden output — bit-for-bit on deterministic/count modules, with floating metrics checked to tight tolerances (Kmer Content matches the FastQC top-20 set at single-partition parity; FastQC's top-20 k-mer list is inherently order-sensitive on real data). One call computes them all in a single out-of-core pass:

import polars_bio as pb

qc = pb.fastqc("reads_R1.fastq.gz")
qc.per_base_quality.collect()   # any module as a LazyFrame
qc.summary().collect()          # PASS / WARN / FAIL per module

# ...or as SQL
pb.sql("SELECT * FROM fastqc('reads_R1.fastq.gz')").collect()

The dataset

We benchmark on a real, citable clinical run rather than a synthetic file: SRR39421268, a Homo sapiens HER2 breast-cancer targeted-capture (exome) library.

prefetch SRR39421268
fasterq-dump --split-spot -Z SRR39421268 | bgzip -c > SRR39421268.fastq.gz  # -Z streams to stdout; bgzip -c writes BGZF
bgzip -r SRR39421268.fastq.gz                                               # build the .gzi index (what lets polars-bio split for parallel scan)

The bgzip -r reindex step writes the BGZF .gzi index — that is what lets polars-bio split the file and scan it in parallel. All three tools are compared on the 11 modules FastQC runs by default.¹

Environment

All three tools were run on the same laptop; wall-clock is best-of-two warm runs, memory via /usr/bin/time -l (footprint) with the FastQC JVM sampled separately.

Tools compared

How each tool was run — all three compute the same 11 default FastQC modules (Kmer Content off):

# Fetch + convert to an indexed BGZF (.gzi lets polars-bio split it for parallel scan)
prefetch SRR39421268
fasterq-dump --split-spot -Z SRR39421268 | bgzip -c > reads.fastq.gz  # -Z → stdout; bgzip -c → BGZF
bgzip -r reads.fastq.gz                                               # write the .gzi index (required for parallel scan)

# FastQC — 11 default modules, single-threaded
fastqc --nogroup reads.fastq.gz

# RastQC — disable its default Kmer Content so it runs the same 11 modules
printf 'kmer\tignore\t1\n' >> limits.txt
rastqc -t 8 --limits limits.txt reads.fastq.gz          # -t 1/2/4/8

# polars-bio — N partitions (target_partitions = 1/2/4/8)
python - <<'PY'
import polars_bio as pb
pb.set_option("datafusion.execution.target_partitions", "8")
pb.fastqc("reads.fastq.gz").tidy.collect()
PY

Performance: four real datasets, 0.7M → 64M reads

We benchmark on four real, citable libraries: our flagship 64M-read clinical HER2 exome (SRR39421268) plus the three short-read runs from the RastQC preprint itself (0.72M, 4.3M, 24.8M reads). All three tools compute the same 11 default modules; lower is better.

Our flagship: the 64M-read clinical exome

polars-bio scales 7.6× to 15.5 s at 8 cores — 13.8× faster than FastQC (213 s, single-threaded) and 3.7× faster than RastQC's best (57.1 s; RastQC barely improves past 4 threads). polars-bio is faster than both at every thread count — and does it at ~2.5× less total CPU than RastQC.

Each row lists wall (elapsed) and total CPU (user + system CPU-seconds — the actual machine work):

polars-bio's total CPU stays flat (~120 CPU-s) as threads rise — each partition adds ~1 core of real work, so wall time falls near-linearly. RastQC spends ~2.5× more total CPU (~300 CPU-s) yet its wall time floors at ~57 s: it is throwing more machine at the problem and getting less back.

It also holds on the RastQC paper's datasets

The flagship is a single run. To check the pattern holds across sizes, we repeated the matched comparison — all three tools, the same 11 default modules — on the three short-read libraries taken from the RastQC preprint: 0.72M, 4.3M, and 24.8M reads, thread-scaled 1 → 8. (All fetched with prefetch + fasterq-dump, converted to indexed BGZF; FastQC single-threaded as it has no parallel mode.)

The largest, DRR013000 (24.8M reads), tells the whole story:

polars-bio scales ~7× to 5.3 s at 8 cores — 12.8× faster than FastQC and 4.2× faster than RastQC's best (22.4 s at 4 threads, after which RastQC plateaus). Even single-threaded, polars-bio (38.1 s) is nearly 2× faster than FastQC — and its total CPU (~39 CPU-s) is less than half of RastQC's (~93 CPU-s).

All four, side by side

Across all four sizes the ranking never changes: polars-bio is fastest at every thread count, and its single-threaded time already beats FastQC on every run.

Each tool's best config, as wall / total-CPU-s:

Paired R2 mates behave identically (full 1/2/4/8 grid in the repo). Two things stand out in RastQC. On the small 0.72M file it shows no thread benefit at all (~2.4 s flat, 1→8t) — but that is by design: its ~28 MB BGZF is below RastQC's 50 MB parallelism threshold, so -t is a no-op there (polars-bio parallelizes it anyway, down to 0.14 s). On the two larger runs, which do clear the threshold, RastQC's single-threaded time is still slower than FastQC (18.6 vs 17.6 s; 70.3 vs 68.4 s; 243.6 vs 212.5 s) — whereas polars-bio scales cleanly and leads throughout, at 2–2.5× less total CPU.

Memory

We measure peak memory exactly as RastQC's own benchmark does — /usr/bin/time -l, maximum resident set size — applied identically to every tool. Because the rest of this comparison runs the same 11 default modules (Kmer off), we report RastQC both ways: its default (Kmer on, the config the paper measured) and Kmer off (strict 11-module parity). polars-bio always runs the 11-module parallel path; FastQC is single-threaded per file.

# macOS /usr/bin/time -l, "maximum resident set size" (bytes ÷ 1048576 = MB), same for all
/usr/bin/time -l python - <<'PY'                                  # polars-bio (target_partitions=N)
import polars_bio as pb
pb.set_option("datafusion.execution.target_partitions", "8")
pb.fastqc("reads.fastq.gz").tidy.collect()
PY
/usr/bin/time -l rastqc -t 8 -q --time            -o out reads.fastq.gz   # RastQC default (Kmer on)
/usr/bin/time -l rastqc -t 8 --limits kmer_off.txt -o out reads.fastq.gz  # RastQC 11-module (Kmer off)
/usr/bin/time -l fastqc -t 1 --quiet              -o out reads.fastq.gz   # FastQC (single-threaded JVM)

Peak RSS (MB) across all four datasets at 1 / 4 / 8 threads (FastQC is single-threaded, so one value):

Four honest takeaways — no tool wins memory outright:

1. RastQC's default is genuinely lean and reproduces the paper: 52 / 358 / 344 MB at 4 threads vs its reported 49 / 332 / 315 — which confirms our method matches theirs.
2. A RastQC quirk: turning Kmer off raises its memory 2–3× on large files (flagship at 4 threads: 419 → 1483 MB). Disabling a module via --limits drops RastQC's duplication-memory bounding — so the strict 11-module-parity comparison paradoxically penalizes RastQC. We flag it rather than lean on it.
3. polars-bio's memory is the most predictable — 270–680 MB across a 90× range of input sizes, growing gently with threads and barely at all with file size (its duplication tracking is bounded by FastQC's 100k-unique cutoff). It is competitive with RastQC's default and never balloons.
4. FastQC's JVM RSS (~470–640 MB at 1 thread) sits in the paper's range, but is heap/GC-driven, not data-driven: fastqc -t 4 reserves ~250 MB per worker and inflates a single-file run to ~900 MB for zero speedup, because FastQC's -t is per-file, not intra-file. That is true in the source, not just observed: AnalysisQueue sets its slot count to threads, then starts one Thread per whole file (one AnalysisRunner = one SequenceFile), and AnalysisRunner.run() reads that file's sequences serially and applies every module in a plain loop — no thread is spawned inside a file. Measured: eff cores = 1.0 at -t 1, -t 2, and -t 4 on one file.

Correct at every thread count

FastQC is single-threaded, so "correct" is unambiguous. Both fast tools are multi-threaded, which raises a question the speed charts don't answer: does the output change with the thread count? We diffed every module at 1 vs N threads on a second run — ERR5897746, 4.25M reads:

• polars-bio is value-identical at 1 and 4 partitions after parsing — the checked output values match exactly across partition counts.
• RastQC is not — its duplication and k-mer modules return materially different numbers at -t 1 vs -t 4.

Here is the full per-module drift against the FastQC 0.12.1 golden. Each cell is the largest deviation across every data point in that module; the per-base rows are re-binned into RastQC's own position bins for a like-for-like comparison (RastQC has no --nogroup, so it always groups positions):

RastQC's per-base plots are fine — the coarser look is just grouping, not error — and its adapter-content values match after re-binning, although it emits two tail bins per adapter that FastQC does not. But per-sequence quality is badly wrong: 671,121 reads land in a Q40 bin FastQC never emits (the same rounding defect that misbinned a third of the 64M run), %GC is off by one, and — uniquely — its duplication and k-mer outputs change with the thread count. At -t 1, RastQC's FastQC-comparable duplication percentages match within rounding (Total Deduplicated Percentage is +0.002 points), but at -t 4 total dedup jumps from 89.48% to 95.64% (+6.16 points) and the level-1 total bin is +9.95 points. Its most-enriched k-mer is correct at -t 1 but the top-20 list already differs; at -t 4, even the top k-mer is wrong.

polars-bio reproduces FastQC bit-for-bit on deterministic/count modules, stays within tight tolerances for floating metrics, and returns the same parsed values at the checked partition counts — its per-partition accumulators merge associatively, and Kmer Content (the one non-associative module) is computed on a single partition, so it is partition-invariant in this comparison too. You never have to ask which thread setting gave you the trustworthy number.

The takeaway

On a 64-million-read clinical exome run, polars-bio is the only tool that is both FastQC-compatible under these parity checks and genuinely fast: 13.8× faster than FastQC, 3.7× faster than RastQC, at ~2.5× less total CPU and with the steadiest memory of the three (~270–680 MB, never ballooning) — while RastQC, the other fast option, silently misbins a third of the reads in one module — and gives different duplication and k-mer numbers depending on the thread count. That lead is not an artefact of one file — it holds from 0.7M to 64M reads. And it is just another table in the engine: SELECT * FROM fastqc('reads.fastq.gz').