Skip to content

Features

polars-bio is a fast, memory-efficient Python DataFrame library for genomics, built on Apache DataFusion, Apache Arrow, and Polars. The features below follow the flow of a typical analysis — read → operate → write — and then how to scale it.

Prefer lazy & streaming execution

Whenever possible, use lazy scans (scan_*) and lazy operations instead of their eager counterparts. Lazy execution lets polars-bio push filters and column projections down to the scan, so far less data is read and materialized in memory. For datasets that don't fit in memory, process them out-of-core: collect with Polars' streaming engine via collect(engine="streaming"), or bypass it entirely with output_type="datafusion.DataFrame" to stream results straight to Parquet/CSV/JSON or to count rows. See Benchmarking DataFrame paths in polars-bio for how much the input and execution path matters in practice.

Typical workflow

A common end-to-end pipeline stays lazy from input to output — read → operate → write — so nothing larger than a batch is ever held in memory:

import polars as pl
import polars_bio as pb

# 1. Read — lazily scan bioinformatic files (no data is read yet)
variants = pb.scan_vcf("cohort.vcf.gz")
exons = pb.scan_gff("gencode.annotation.gff3.gz").filter(pl.col("type") == "exon")

# 2. Operate — range operation: keep variants that overlap an exon
in_exons = pb.overlap(variants, exons)

# 3. Write — stream the result straight to Parquet, never fully materialized
in_exons.sink_parquet("variants_in_exons.parquet")

Each step is covered in detail below: reading files, genomic operations, and writing files.

Genomic operations

Overlap, nearest, count, coverage, merge, cluster, complement, subtract, and more — exposed through a DataFrame API and a native parallel engine. Also covers pileup/depth computation and flexible overlap output modes.

Reading files

Eager (read_*), lazy (scan_*), and SQL-ready (register_*) access to all supported input formats (BED, VCF, BCF, VCF Zarr, BGEN, PGEN, BAM, SAM, CRAM, FASTQ, FASTA, A2M, A3M, Stockholm, GFF3, GTF, Pairs, BigWig, BigBed, Cooler .cool/.mcool). Prefer scan_* — it enables indexed reads with predicate and projection pushdown. Also covers optional BAM tags, schema inspection, coordinate-system handling, and the metadata attached to every DataFrame.

Protein structures

Read PDB/mmCIF collections and local Foldcomp files or database subsets as atom or residue tables. Residue output includes backbone coordinates and six angles, with coordinates in Angstroms and angles in degrees. Lazy, eager and SQL access use native Rust readers without a Python codec dependency.

Writing files

Eager (write_*) and streaming (sink_*) output, coordinate-sorted writes, header preservation, BAM/SAM tag writing, and compression.

SQL processing

Register datasets as tables and query them with SQL (Apache DataFusion), access registered tables programmatically, and build reusable views.

DataFrames support

Use polars-bio with file paths, Polars, or Pandas DataFrames — and how the backend/dtype choice affects performance (Arrow-backed Pandas ≈ Polars).

Cloud storage

Stream bioinformatic files directly from S3, GCS, and Azure via Apache OpenDAL, with per-provider configuration.

Parallel processing

One global target_partitions setting controls parallelism for both input reads and range operations across CPU cores.