Ensembl VEP caches¶

vepyr annotates against an Ensembl VEP cache converted into a point-lookup-optimized, per-chromosome Parquet layout by vepyr.build_cache. This page describes the cache types, the supported Ensembl releases, the entities a converted cache contains, and their on-disk sizes.

Cache types¶

VEP ships three flavours of the GRCh38 cache, differing only in which transcript set they carry:

Type	Transcripts	`build_cache(cache_type=…)`
ensembl	Ensembl/GENCODE transcripts	`ensembl`
refseq	RefSeq transcripts	`refseq`
merged	Ensembl and RefSeq (both transcript sets)	`merged`

The variation, regulatory, and motif data are the same across types; only the transcript/exon/translation entities differ (merged is the union, so it is the largest).

Supported releases¶

Release 115 (GRCh38) — the golden reference VCFs in this project are produced with the ensemblorg/ensembl-vep:release_115.2 docker image.
Release 116 (GRCh38).

Both are built the same way; a converted cache directory is named <release>_<assembly>_<type>, e.g. 115_GRCh38_merged, 116_GRCh38_merged.

Layout & entities¶

build_cache downloads the Ensembl cache tarball and converts each entity into a directory of per-chromosome Parquet shards:

<cache_dir>/<release>_<assembly>_<type>/
  variation/chr1.parquet … chrY.parquet   (+ non-standard contigs)
  transcript/chr1.parquet …
  exon/…  translation_core/…  translation_sift/…  regulatory/…  motif/…

The entities and what they feed in the CSQ output:

Entity	Feeds	Notes
variation	`Existing_variation`, co-located allele frequencies (1000G, gnomAD exomes/genomes), `MAX_AF*`, `CLIN_SIG`, `SOMATIC`, `PHENO`, `PUBMED`	By far the largest entity; also carries the warm/cold `tier` that plugin caches inherit.
transcript	`Gene`, `Feature`, `SYMBOL`, `BIOTYPE`, `STRAND`, `CANONICAL`, `MANE*`, `TSL`, `APPRIS`, `CCDS`, source/xref fields	The transcript models; differs by cache type.
exon	`EXON`/`INTRON` numbering, `cDNA_position`, `CDS_position`	Exon/intron structure.
translation_core	`Protein_position`, `Amino_acids`, `Codons`, `HGVSp`, `ENSP`	Protein translations.
translation_sift	`SIFT`, `PolyPhen`	Precomputed missense predictions; the second-largest entity (per-protein blobs).
regulatory	regulatory-region consequences	Regulatory features.
motif	TF-binding / motif consequences (`MOTIF_NAME`, `MOTIF_POS`, `HIGH_INF_POS`, `MOTIF_SCORE_CHANGE`, `TRANSCRIPTION_FACTORS`)	Present from release 116; empty in the release-115 caches.

(The Ensembl translation entity is split into two Parquet entities on conversion: translation_core and translation_sift.)

Cache format & lookup internals¶

Every cache — the Ensembl variation/transcript/… entities and the custom plugin caches — uses the same point-lookup-optimized Parquet layout, so a lookup reads only the handful of pages that could contain the queried positions rather than scanning the whole file.

Parquet storage¶

The writer properties are tuned for random point lookups, not scans:

Property	Value	Why
Compression	ZSTD, level 3	Good ratio; fast enough to decode per page.
Dictionary encoding	disabled	Avoids a per-take dictionary load; ZSTD recovers the ratio (the no-dict file is actually smaller).
Data page size	≤ 4 KiB	Small pages → fine-grained page index → a lookup touches minimal bytes.
Data page row count	≤ 512 rows	Bounds how many rows a single page decode yields.
Statistics	Page-level	Emits `ColumnIndex` + `OffsetIndex` in the footer — the read-side position→page directory.
Row group size	1,000,000 rows	Large groups keep footer/metadata overhead low; the page index gives intra-group resolution.
Sorting columns	`(tier, start)`	Physical clustering — see Sorting within a shard.

Sorting within a shard¶

Rows within each shard are physically sorted by (tier, start) and written in that order, and the sort is recorded in the Parquet SortingColumn metadata:

By tier first — all warm rows (tier 0) are written before all cold rows (tier 1). This clusters common variants into a contiguous run of pages, so a buffer of common-variant lookups touches a small, dense region instead of pages scattered across the file.
By start within each tier — each tier's run is ascending by genomic start. Ascending, non-overlapping start ranges per page are what make the ColumnIndex (per-page min/max of start) an effective pruning directory: resolving a query position to its candidate page(s) is a binary-search-like metadata lookup, and coalescing adjacent pages into one read is cheap.

Because the file is split into a warm block then a cold block (each independently start-sorted), a single start value can appear in both blocks; the lookup resolves candidate pages across both. Writing warm-first keeps the hot working set contiguous — the whole point of the tier.

Page index → the `PageDir`¶

Since page-level statistics are enabled, each shard's footer carries a ColumnIndex (per-page min/max of start) and an OffsetIndex (per-page byte offset + row range). At open time the reader builds a PageDir over the start leaf column from these indexes. Resolving a set of query positions to the minimal set of candidate page row-ranges is then a metadata-only operation — no column data is read until the ranges are known.

Row groups¶

Shards use 1,000,000-row row groups. Row groups bound the footer metadata size; within a group the small (≤ 512-row) pages plus the page index provide the actual point-lookup resolution. A whole chromosome is typically one or a few row groups.

Runtime lookup — async reader + monotonic cursor, in batches¶

Annotation runs in position-ordered buffers. For each buffer the runtime does one page-scoped, three-phase take per shard, reading only that buffer's candidate pages:

Resolve — the buffer's sorted, de-duplicated start positions are mapped through the PageDir to candidate page row-ranges. Metadata only; no data read.
Locate — a start-only projected read over just those pages (a RowSelection built from the ranges) streams start values back in batches through a CoalescingAsyncReader — an async Parquet reader that merges nearby page byte-ranges (within a 512 KiB gap) into single I/O calls. A monotonic row-offset cursor advances exactly one step per streamed row, staying in lockstep with the selection, and records the exact file offset of every row whose start is in the buffer's probe set. The cursor only moves forward, so there is no back-seeking.
Take — a final projected read at those exact offsets pulls just the payload columns for the matched rows into one compact RecordBatch.

Sizing¶

Measured on-disk sizes of the converted Parquet caches in /Users/mwiewior/workspace/data_vepyr (GRCh38):

Cache	Total	variation	translation_sift	transcript	translation_core	exon	regulatory	motif
`115_GRCh38_ensembl`	29 G	26 G	2.7 G	478 M	196 M	178 M	71 M	—
`115_GRCh38_refseq`	29 G	26 G	2.8 G	289 M	155 M	73 M	71 M	—
`115_GRCh38_merged`	32 G	26 G	5.5 G	664 M	348 M	240 M	71 M	—
`116_GRCh38_merged`	36 G	27 G	7.0 G	762 M	417 M	272 M	71 M	81 M

Observations:

variation dominates every cache (~26–27 G, ~80–90% of the total) — it is the full set of known variants with population frequencies.
translation_sift is the clear #2 and scales with the transcript set: merged (both transcript sets) is ~2× the single-set caches, and 116 is larger than 115.
merged ≈ ensembl + refseq for the transcript/exon/translation entities (it carries both), while sharing the same variation/regulatory data.
motif is populated only from release 116 (empty in the 115 caches).

Building a cache¶

import vepyr

vepyr.build_cache(
    release=116,
    cache_dir="/Users/mwiewior/workspace/data_vepyr",
    cache_type="merged",          # "ensembl" | "refseq" | "merged"
    assembly="GRCh38",
    partitions=8,                 # conversion parallelism
)
# → /Users/mwiewior/workspace/data_vepyr/116_GRCh38_merged/<entity>/chr*.parquet

See the API reference for the full signature. Plugin caches (e.g. AlphaMissense) are built separately and layered on top — see Plugins.