Cloud storage ☁️

On this page: Example · Supported features · AWS S3 · Google Cloud Storage · Azure Blob Storage

polars-bio supports direct streamed reading from cloud storages (e.g. S3, GCS) enabling processing large-scale genomics data without materializing in memory. It is built upon the OpenDAL project, a unified data access layer for cloud storage, which allows to read bioinformatic file formats from various cloud storage providers. For Apache DataFusion native file formats, such as Parquet or CSV please refer to DataFusion user guide.

Example

import polars_bio as pb
## Register VCF files from Google Cloud Storage that will be streamed - no need to download them to the local disk, size ~0.8TB
pb.register_vcf("gs://gcp-public-data--gnomad/release/2.1.1/liftover_grch38/vcf/genomes/gnomad.genomes.r2.1.1.sites.liftover_grch38.vcf.bgz", "gnomad_big", allow_anonymous=True)
pb.register_vcf("gs://gcp-public-data--gnomad/release/4.1/genome_sv/gnomad.v4.1.sv.sites.vcf.gz", "gnomad_sv", allow_anonymous=True)
pb.overlap("gnomad_sv", "gnomad_big", streaming=True).sink_parquet("/tmp/overlap.parquet")

It is especially useful when combined with SQL support for preprocessing and streaming processing capabilities.

Tip

If you access cloud storage with authentication provided, please make sure the allow_anonymous parameter is set to False in the read/describe/register_table functions.

Supported features

Feature	AWS S3	Google Cloud Storage	Azure Blob Storage
Anonymous access
Authenticated access
Requester Pays
Concurrent requests¹
Streaming reads

Note

¹For more information on concurrent requests and block size tuning please refer to issue.

AWS S3 configuration

Supported environment variables:

Variable	Description
AWS_ACCESS_KEY_ID	AWS access key ID for authenticated access to S3.
AWS_SECRET_ACCESS_KEY	AWS secret access key for authenticated access to S3.
AWS_ENDPOINT_URL	Custom S3 endpoint URL for accessing S3-compatible storage.
AWS_REGION or AWS_DEFAULT_REGION	AWS region for accessing S3.

Google Cloud Storage configuration

Supported environment variables:

Variable	Description
GOOGLE_APPLICATION_CREDENTIALS	Path to the Google Cloud service account key file for authenticated access to GCS.

Azure Blob Storage configuration

Supported environment variables:

Variable	Description
AZURE_STORAGE_ACCOUNT	Azure Storage account name for authenticated access to Azure Blob Storage.
AZURE_STORAGE_KEY	Azure Storage account key for authenticated access to Azure Blob Storage.
AZURE_ENDPOINT_URL	Azure Blob Storage endpoint URL for accessing Azure Blob Storage.