Skip to content

Features

polars-bio is a fast, memory-efficient Python DataFrame library for genomics, built on Apache DataFusion, Apache Arrow, and Polars. The features below follow the flow of a typical analysis — read → operate → write — and then how to scale it.

Prefer lazy & streaming execution

Whenever possible, use lazy scans (scan_*) and lazy operations instead of their eager counterparts. Lazy execution lets polars-bio push filters and column projections down to the scan, so far less data is read and materialized in memory. For datasets that don't fit in memory, process them out-of-core: collect with Polars' streaming engine via collect(engine="streaming"), or bypass it entirely with output_type="datafusion.DataFrame" to stream results straight to Parquet/CSV/JSON or to count rows. See Benchmarking DataFrame paths in polars-bio for how much the input and execution path matters in practice.

Typical workflow

A common end-to-end pipeline stays lazy from input to output — read → operate → write — so nothing larger than a batch is ever held in memory:

import polars as pl
import polars_bio as pb

# 1. Read — lazily scan bioinformatic files (no data is read yet)
variants = pb.scan_vcf("cohort.vcf.gz")
exons = pb.scan_gff("gencode.annotation.gff3.gz").filter(pl.col("type") == "exon")

# 2. Operate — range operation: keep variants that overlap an exon
in_exons = pb.overlap(variants, exons)

# 3. Write — stream the result straight to Parquet, never fully materialized
in_exons.sink_parquet("variants_in_exons.parquet")

Each step is covered in detail below: reading files, genomic operations, and writing files.

Genomic operations

Overlap, nearest, count, coverage, merge, cluster, complement, subtract, and more — exposed through a DataFrame API and a native parallel engine. Also covers pileup/depth computation and flexible overlap output modes.

Reading files

Eager (read_*), lazy (scan_*), and SQL-ready (register_*) access to all supported input formats (BED, VCF, VCF Zarr, BAM, CRAM, FASTQ, FASTA, GFF3, GTF, Pairs, BigWig, BigBed). Prefer scan_* — it enables indexed reads with predicate and projection pushdown. Also covers optional BAM tags, schema inspection, coordinate-system handling, and the metadata attached to every DataFrame.

Writing files

Eager (write_*) and streaming (sink_*) output, coordinate-sorted writes, header preservation, BAM/SAM tag writing, and compression.

SQL processing

Register datasets as tables and query them with SQL (Apache DataFusion), access registered tables programmatically, and build reusable views.

DataFrames support

Use polars-bio with file paths, Polars, or Pandas DataFrames — and how the backend/dtype choice affects performance (Arrow-backed Pandas ≈ Polars).

Cloud storage

Stream bioinformatic files directly from S3, GCS, and Azure via Apache OpenDAL, with per-provider configuration.

Parallel processing

One global target_partitions setting controls parallelism for both input reads and range operations across CPU cores.