polars-bio 0.38.0: Sequence Operations, DataFusion 55.2, and Smaller, Faster Wheels
polars-bio 0.38.0 adds sequence operations as Polars expressions: reverse complement, GC content, translation, mean quality and more. At one thread they are 2.1–3.6× faster than seqkit on FASTQ and 4.3–4.8× faster on translation, and at 8 threads GC content runs 24× faster than seqkit. The release also moves to DataFusion 55.2, reads plain gzip FASTQ about 2× faster, and ships wheels that are 30% smaller and up to 14% faster.
Sequence operations as Polars expressions
Importing polars_bio registers a .seq namespace on every Polars expression:
import polars as pl
import polars_bio as pb # registers pl.Expr.seq
reads = (
pb.scan_fastq("reads.fastq.gz")
.with_columns(
gc=pl.col("sequence").seq.gc_content(),
qual=pl.col("quality_scores").seq.mean_quality(),
)
.filter((pl.col("gc") > 0.4) & (pl.col("qual") >= 20))
.collect(engine="streaming")
)
| Expression | What it computes |
|---|---|
.seq.revcomp() |
reverse complement |
.seq.reverse() |
reversed string, e.g. a FASTQ quality alongside revcomp |
.seq.complement() |
complement, not reversed |
.seq.gc_content() |
GC fraction, 0 to 1 |
.seq.translate() |
protein translation, frames ±1–3, every NCBI table |
.seq.mean_quality() |
mean Phred quality |
They are native Polars plugins: they run in parallel without the GIL, stream,
and work on any String column. The same functions are available in SQL as
seq_revcomp, seq_gc_content, seq_translate and so on. Results match seqkit
2.14.0 and Biopython 1.88 on golden test corpora.
Benchmarks
Three pipelines, each producing the same file with every tool: reverse complement → FASTQ, GC % + mean quality → TSV, and translation → FASTA. Inputs are public FASTQ runs of 0.72M, 4.25M, 24.8M and 64.3M reads and the Ensembl GRCh38 CDS (0.38M sequences), BGZF-compressed. The machine was an Apple M3 Max; wall time is the best of three runs.
One thread
Wall time in seconds, one thread:
| Pipeline | Reads | polars-bio | seqkit | seqtk | seqfu | Biopython |
|---|---|---|---|---|---|---|
| Reverse complement | 0.72M | 0.21 | 0.53 | 0.63 | 1.46 | 1.37 |
| 4.25M | 1.69 | 5.10 | 4.79 | 9.12 | 9.58 | |
| 24.8M | 8.06 | 21.97 | 21.29 | 47.42 | 48.28 | |
| 64.3M | 19.72 | 58.08 | 59.35 | 126.54 | 144.07 | |
| GC + mean quality | 0.72M | 0.19 | 0.65 | 1.77 | — | 3.37 |
| 4.25M | 1.60 | 5.74 | 13.34 | — | 27.96 | |
| 24.8M | 7.71 | 26.57 | 66.20 | — | 114.29 | |
| 64.3M | 18.75 | 66.03 | 187.21 | — | 380.41 | |
| Translation | 0.38M | 0.73 | 3.52 | — | — | 21.83 |
polars-bio is the fastest tool on every pipeline. The price is memory: it peaks at 140–360 MB, against 2–160 MB for the other tools, which process one record at a time.
Multiple threads
Wall time in seconds:
| Pipeline | Reads | polars-bio, 1 → 8 threads | seqkit, -j 1 → -j 8 |
At 8 threads |
|---|---|---|---|---|
| GC + mean quality | 0.72M | 0.19 → 0.05 (3.9×) | ||
| 4.25M | 1.60 → 0.24 (6.6×) | |||
| 24.8M | 7.71 → 1.10 (7.0×) | 26.71 → 27.00 | 24× faster | |
| 64.3M | 18.75 → 2.98 (6.3×) | 66.15 → 67.42 | 23× faster | |
| Reverse complement | 0.72M | 0.21 → 0.08 (2.8×) | ||
| 4.25M | 1.69 → 0.35 (4.8×) | |||
| 24.8M | 8.06 → 1.90 (4.2×) | 22.41 → 22.66 | 12× faster | |
| 64.3M | 19.72 → 5.68 (3.5×) | 58.95 → 60.26 | 11× faster |
polars-bio reads BGZF files in parallel partitions, so these pipelines scale
with threads. seqkit's -j doesn't speed up these subcommands on BGZF input.
Plain gzip can only be decompressed on one thread, so compress with bgzip for
multi-threaded work.
Plain gzip about 2× faster
Plain gzip FASTQ is now decoded on a faster path. At one thread on 24.8M reads, reverse complement went from 15.1 s to 9.9 s and GC + quality from 14.5 s to 8.8 s. Gzip input is now within 5–15% of BGZF.
DataFusion 55.2
polars-bio moves from DataFusion 53.0 to 55.2.0, with datafusion-python 55.0
and Arrow 59.3. Upgrade datafusion together with polars-bio: the requirement is
now datafusion>=55.0.0,<56. External sorts no longer fail with
Resources exhausted when several partitions sort at once, and remote reads that
fail mid-stream now return an error instead of retrying forever. Interval
operations run at the same speed as in 0.37.0.
Smaller, faster wheels
The PyPI wheels are now stripped and built with fat link-time optimisation. On macOS arm64, against a stripped build without LTO:
| Without LTO | 0.38.0 wheel (fat LTO) | |
|---|---|---|
| Wheel size | 79.5 MB | 55.5 MB (−30%) |
| BAM scan, 1 thread | baseline | 10–14% faster |
| VCF scan, 1 thread | baseline | 12% faster |
| FASTQ scan, 1 thread | baseline | 5% faster |
Also in 0.38.0
pb.set_progress_bar(False)turns off polars-bio's progress bar at any time (#504).df.config_metano longer keeps large frames alive: in a 10 × 160 MB loop peak memory fell from 1.7 GB to 0.33 GB (#503).
Try it
The Sequence operations guide and the
API reference cover every function. The benchmark
harness and raw results (the seqops-expressions-* folders under results/)
are in polars-bio-bench.

