Skip to content

Auxiliary

Metadata

get_metadata(df)

Get all metadata attached to a DataFrame or LazyFrame.

Returns all metadata including: - Source file information (format, path) - Format-specific metadata (VCF INFO/FORMAT fields, FASTQ quality encoding, etc.) - Comprehensive Arrow schema metadata (if available)

Parameters:

Name Type Description Default
df

Polars DataFrame or LazyFrame (or Pandas DataFrame)

required

Returns:

Type Description
dict

Dict with keys:

dict
  • "format": File format identifier (e.g., "vcf", "fastq", "bam")
dict
  • "path": Original file path
dict
  • "coordinate_system_zero_based": Boolean indicating coordinate system (True=0-based, False=1-based, None=not set)
dict
  • "header": Format-specific header data as dict, may include:
  • For VCF: "info_fields", "format_fields", "sample_names", "version", "contigs", "filters", etc.
  • For FASTQ: quality encoding information
  • For other formats: format-specific metadata
  • "_datafusion_table_name": Internal DataFusion table name (for debugging)

Examples:

Get all metadata from a VCF file:

import polars_bio as pb
lf = pb.scan_vcf("file.vcf")
meta = pb.get_metadata(lf)

Access basic metadata:

meta["format"]                        # Returns: 'vcf'
meta["path"]                          # Returns: 'file.vcf'
meta["coordinate_system_zero_based"]  # Returns: False (1-based for VCF)

Access VCF-specific metadata:

info_fields = meta["header"]["info_fields"]
format_fields = meta["header"]["format_fields"]
sample_names = meta["header"]["sample_names"]
version = meta["header"]["version"]
contigs = meta["header"]["contigs"]

Source code in polars_bio/_metadata.py
def get_metadata(df) -> dict:
    """Get all metadata attached to a DataFrame or LazyFrame.

    Returns all metadata including:
    - Source file information (format, path)
    - Format-specific metadata (VCF INFO/FORMAT fields, FASTQ quality encoding, etc.)
    - Comprehensive Arrow schema metadata (if available)

    Args:
        df: Polars DataFrame or LazyFrame (or Pandas DataFrame)

    Returns:
        Dict with keys:
        - "format": File format identifier (e.g., "vcf", "fastq", "bam")
        - "path": Original file path
        - "coordinate_system_zero_based": Boolean indicating coordinate system (True=0-based, False=1-based, None=not set)
        - "header": Format-specific header data as dict, may include:
            - For VCF: "info_fields", "format_fields", "sample_names", "version", "contigs", "filters", etc.
            - For FASTQ: quality encoding information
            - For other formats: format-specific metadata
            - "_datafusion_table_name": Internal DataFusion table name (for debugging)

    Examples:
        Get all metadata from a VCF file:
        ```python
        import polars_bio as pb
        lf = pb.scan_vcf("file.vcf")
        meta = pb.get_metadata(lf)
        ```

        Access basic metadata:
        ```python
        meta["format"]                        # Returns: 'vcf'
        meta["path"]                          # Returns: 'file.vcf'
        meta["coordinate_system_zero_based"]  # Returns: False (1-based for VCF)
        ```

        Access VCF-specific metadata:
        ```python
        info_fields = meta["header"]["info_fields"]
        format_fields = meta["header"]["format_fields"]
        sample_names = meta["header"]["sample_names"]
        version = meta["header"]["version"]
        contigs = meta["header"]["contigs"]
        ```
    """
    result = {
        "format": None,
        "path": None,
        "coordinate_system_zero_based": None,
        "header": None,
    }

    if _has_config_meta(df):
        # Polars DataFrame/LazyFrame
        try:
            metadata = df.config_meta.get_metadata()
        except (KeyError, AttributeError, TypeError):
            return result

        result["format"] = metadata.get(SOURCE_FORMAT_KEY)
        result["path"] = metadata.get(SOURCE_PATH_KEY)
        result["coordinate_system_zero_based"] = metadata.get(COORDINATE_SYSTEM_KEY)

        header_json = metadata.get(SOURCE_HEADER_KEY)
        if header_json:
            try:
                result["header"] = json.loads(header_json)
            except (json.JSONDecodeError, TypeError):
                pass

    elif _is_pandas_dataframe(df):
        # Pandas DataFrame
        if hasattr(df, "attrs"):
            result["format"] = df.attrs.get(SOURCE_FORMAT_KEY)
            result["path"] = df.attrs.get(SOURCE_PATH_KEY)
            result["coordinate_system_zero_based"] = df.attrs.get(COORDINATE_SYSTEM_KEY)

            header_json = df.attrs.get(SOURCE_HEADER_KEY)
            if header_json:
                try:
                    result["header"] = json.loads(header_json)
                except (json.JSONDecodeError, TypeError):
                    pass

    return result

print_metadata_json(df, indent=2)

Print metadata as pretty-formatted JSON.

Parameters:

Name Type Description Default
df Union[DataFrame, LazyFrame]

Polars DataFrame or LazyFrame

required
indent int

Number of spaces for indentation (default: 2)

2
Example
import polars_bio as pb
lf = pb.scan_vcf("file.vcf")
pb.print_metadata_json(lf)
Source code in polars_bio/_metadata.py
def print_metadata_json(df: Union[pl.DataFrame, pl.LazyFrame], indent: int = 2) -> None:
    """Print metadata as pretty-formatted JSON.

    Args:
        df: Polars DataFrame or LazyFrame
        indent: Number of spaces for indentation (default: 2)

    Example:
        ```python
        import polars_bio as pb
        lf = pb.scan_vcf("file.vcf")
        pb.print_metadata_json(lf)
        ```
    """
    meta = get_metadata(df)
    print(json.dumps(meta, indent=indent, default=str))

print_metadata_summary(df)

Print a human-readable summary of all metadata.

Displays a formatted summary of all metadata attached to a DataFrame or LazyFrame, including format, path, coordinate system, and format-specific information.

Parameters:

Name Type Description Default
df Union[DataFrame, LazyFrame]

Polars DataFrame or LazyFrame

required
Example
import polars_bio as pb
lf = pb.scan_vcf("file.vcf")
pb.print_metadata_summary(lf)
Source code in polars_bio/_metadata.py
def print_metadata_summary(df: Union[pl.DataFrame, pl.LazyFrame]) -> None:
    """Print a human-readable summary of all metadata.

    Displays a formatted summary of all metadata attached to a DataFrame or LazyFrame,
    including format, path, coordinate system, and format-specific information.

    Args:
        df: Polars DataFrame or LazyFrame

    Example:
        ```python
        import polars_bio as pb
        lf = pb.scan_vcf("file.vcf")
        pb.print_metadata_summary(lf)
        ```
    """
    meta = get_metadata(df)
    if not meta or not any([meta.get("format"), meta.get("path"), meta.get("header")]):
        print("No metadata available")
        return

    print("=" * 70)
    print("Metadata Summary")
    print("=" * 70)
    print()

    # Basic metadata
    if meta.get("format"):
        print(f"Format: {meta['format']}")
    if meta.get("path"):
        print(f"Path: {meta['path']}")
    if meta.get("coordinate_system_zero_based") is not None:
        coord_sys = "0-based" if meta["coordinate_system_zero_based"] else "1-based"
        print(f"Coordinate System: {coord_sys}")

    # Format-specific metadata
    if meta.get("header"):
        header = meta["header"]
        print()
        print("Format-specific metadata:")
        print("-" * 70)

        # VCF-specific
        if meta.get("format") == "vcf":
            if "version" in header:
                print(f"  VCF Version: {header['version']}")
            if "sample_names" in header:
                samples = header["sample_names"]
                print(f"  Samples ({len(samples)}): {', '.join(samples[:5])}")
                if len(samples) > 5:
                    print(f"    ... and {len(samples) - 5} more")
            if "info_fields" in header:
                print(f"  INFO fields: {len(header['info_fields'])}")
                for field_id in list(header["info_fields"].keys())[:3]:
                    field = header["info_fields"][field_id]
                    print(
                        f"    - {field_id}: {field.get('type')} ({field.get('description', 'No description')})"
                    )
                if len(header["info_fields"]) > 3:
                    print(f"    ... and {len(header['info_fields']) - 3} more")
            if "format_fields" in header:
                print(f"  FORMAT fields: {len(header['format_fields'])}")
                for field_id in list(header["format_fields"].keys())[:3]:
                    field = header["format_fields"][field_id]
                    print(
                        f"    - {field_id}: {field.get('type')} ({field.get('description', 'No description')})"
                    )
                if len(header["format_fields"]) > 3:
                    print(f"    ... and {len(header['format_fields']) - 3} more")
            if "contigs" in header and header["contigs"]:
                print(f"  Contigs: {len(header['contigs'])}")
            if "filters" in header and header["filters"]:
                print(f"  Filters: {len(header['filters'])}")

        # Other formats can be added here as needed

    print()
    print("=" * 70)

set_source_metadata(df, format, path='', header=None)

Set standardized source file metadata.

Stores metadata about the source file format, path, and format-specific header information. This standardized approach works across all file formats (VCF, FASTQ, BAM, GFF, BED, FASTA, CRAM).

Parameters:

Name Type Description Default
df

Polars DataFrame or LazyFrame (or Pandas DataFrame)

required
format str

File format identifier (e.g., "vcf", "fastq", "bam")

required
path str

Original file path (default: "")

''
header dict

Format-specific header data as dict (default: None) For VCF: {"info_fields": {...}, "format_fields": {...}, "sample_names": [...], ...} For other formats: format-specific metadata

None
Example
import polars_bio as pb
lf = pb.scan_vcf("sample.vcf")
header = {"info_fields": {...}, "sample_names": ["sample1"]}
pb.set_source_metadata(lf, format="vcf", path="sample.vcf", header=header)
Source code in polars_bio/_metadata.py
def set_source_metadata(df, format: str, path: str = "", header: dict = None):
    """Set standardized source file metadata.

    Stores metadata about the source file format, path, and format-specific
    header information. This standardized approach works across all file
    formats (VCF, FASTQ, BAM, GFF, BED, FASTA, CRAM).

    Args:
        df: Polars DataFrame or LazyFrame (or Pandas DataFrame)
        format: File format identifier (e.g., "vcf", "fastq", "bam")
        path: Original file path (default: "")
        header: Format-specific header data as dict (default: None)
                For VCF: {"info_fields": {...}, "format_fields": {...}, "sample_names": [...], ...}
                For other formats: format-specific metadata

    Example:
        ```python
        import polars_bio as pb
        lf = pb.scan_vcf("sample.vcf")
        header = {"info_fields": {...}, "sample_names": ["sample1"]}
        pb.set_source_metadata(lf, format="vcf", path="sample.vcf", header=header)
        ```
    """
    if _has_config_meta(df):
        # Polars DataFrame/LazyFrame
        metadata_updates = {
            SOURCE_FORMAT_KEY: format,
            SOURCE_PATH_KEY: path,
            SOURCE_HEADER_KEY: json.dumps(header) if header else "",
        }
        df.config_meta.set(**metadata_updates)
    elif _is_pandas_dataframe(df):
        # Pandas DataFrame
        if not hasattr(df, "attrs"):
            df.attrs = {}
        df.attrs[SOURCE_FORMAT_KEY] = format
        df.attrs[SOURCE_PATH_KEY] = path
        df.attrs[SOURCE_HEADER_KEY] = json.dumps(header) if header else ""

Session options & logging

set_option(key, value)

Set a configuration option.

Parameters:

Name Type Description Default
key

The configuration key.

required
value

The value to set (bool values are converted to "true"/"false").

required
Example
import polars_bio as pb
pb.set_option("datafusion.bio.coordinate_system_zero_based", False)
Source code in polars_bio/context.py
def set_option(key, value):
    """Set a configuration option.

    Args:
        key: The configuration key.
        value: The value to set (bool values are converted to "true"/"false").

    Example:
        ```python
        import polars_bio as pb
        pb.set_option("datafusion.bio.coordinate_system_zero_based", False)
        ```
    """
    Context().set_option(key, value)

get_option(key)

Get the value of a configuration option.

Parameters:

Name Type Description Default
key

The configuration key.

required

Returns:

Type Description

The current value of the option as a string, or None if not set.

Example
import polars_bio as pb
pb.get_option("datafusion.bio.coordinate_system_zero_based")
'true'
Source code in polars_bio/context.py
def get_option(key):
    """Get the value of a configuration option.

    Args:
        key: The configuration key.

    Returns:
        The current value of the option as a string, or None if not set.

    Example:
        ```python
        import polars_bio as pb
        pb.get_option("datafusion.bio.coordinate_system_zero_based")
        'true'
        ```
    """
    return Context().get_option(key)

set_loglevel(level)

Set the log level for the logger and root logger.

Parameters:

Name Type Description Default
level str

The log level to set. Can be "debug", "info", "warn", or "warning".

required

Note

The log level should be set as a first step after importing the library. Once set it can be only decreased, not increased. In order to increase the log level, you need to restart the Python session.

Example:

import polars_bio as pb
pb.set_loglevel("info")

Source code in polars_bio/logging.py
def set_loglevel(level: str):
    """Set the log level for the logger and root logger.

    Args:
        level: The log level to set. Can be "debug", "info", "warn", or "warning".

    !!! note
        The log level should be set as a **first** step after importing the library.
        Once set it can be only **decreased**, not increased. In order to increase
        the log level, you need to restart the Python session.

        Example:
        ```python
        import polars_bio as pb
        pb.set_loglevel("info")
        ```
    """
    level = level.lower()
    if level == "debug":
        logger.setLevel(logging.DEBUG)
        root_logger.setLevel(logging.DEBUG)
        logging.basicConfig(level=logging.DEBUG)
    elif level == "info":
        logger.setLevel(logging.INFO)
        root_logger.setLevel(logging.INFO)
        logging.basicConfig(level=logging.INFO)
    elif level == "warn" or level == "warning":
        logger.setLevel(logging.WARN)
        root_logger.setLevel(logging.WARN)
        logging.basicConfig(level=logging.WARN)
    else:
        raise ValueError(f"{level} is not a valid log level")

Errors

MissingCoordinateSystemError

Bases: Exception

Raised when a DataFrame lacks coordinate system metadata.

Range operations require coordinate system metadata to determine the correct interval semantics. This error is raised when:

  • A Polars LazyFrame/DataFrame lacks polars-config-meta metadata
  • A Pandas DataFrame lacks df.attrs["coordinate_system_zero_based"]
  • A file path registers a table without Arrow schema metadata

For Polars DataFrames, use polars-bio I/O functions (scan_, read_) which automatically set the metadata.

For Pandas DataFrames, set the attribute before passing to range operations:

df.attrs["coordinate_system_zero_based"] = True  # 0-based coords

Example
import pandas as pd
import polars_bio as pb

pdf = pd.read_csv("intervals.bed", sep="        ", names=["chrom", "start", "end"])
pb.overlap(pdf, pdf)  # Raises MissingCoordinateSystemError

# Fix: set the coordinate system metadata
pdf.attrs["coordinate_system_zero_based"] = True
pb.overlap(pdf, pdf)  # Works correctly
Source code in polars_bio/exceptions.py
class MissingCoordinateSystemError(Exception):
    """Raised when a DataFrame lacks coordinate system metadata.

    Range operations require coordinate system metadata to determine the
    correct interval semantics. This error is raised when:

    - A Polars LazyFrame/DataFrame lacks polars-config-meta metadata
    - A Pandas DataFrame lacks df.attrs["coordinate_system_zero_based"]
    - A file path registers a table without Arrow schema metadata

    For Polars DataFrames, use polars-bio I/O functions (scan_*, read_*) which
    automatically set the metadata.

    For Pandas DataFrames, set the attribute before passing to range operations:
        ```python
        df.attrs["coordinate_system_zero_based"] = True  # 0-based coords
        ```

    Example:
        ```python
        import pandas as pd
        import polars_bio as pb

        pdf = pd.read_csv("intervals.bed", sep="\t", names=["chrom", "start", "end"])
        pb.overlap(pdf, pdf)  # Raises MissingCoordinateSystemError

        # Fix: set the coordinate system metadata
        pdf.attrs["coordinate_system_zero_based"] = True
        pb.overlap(pdf, pdf)  # Works correctly
        ```
    """

    pass

CoordinateSystemMismatchError

Bases: Exception

Raised when two DataFrames have different coordinate systems.

This error occurs when attempting range operations (overlap, nearest, etc.) on DataFrames where one uses 0-based coordinates and the other uses 1-based coordinates.

Example
df1 = pb.scan_vcf("file1.vcf", one_based=False)  # 0-based
df2 = pb.scan_vcf("file2.vcf", one_based=True)   # 1-based
pb.overlap(df1, df2)  # Raises CoordinateSystemMismatchError
Source code in polars_bio/exceptions.py
class CoordinateSystemMismatchError(Exception):
    """Raised when two DataFrames have different coordinate systems.

    This error occurs when attempting range operations (overlap, nearest, etc.)
    on DataFrames where one uses 0-based coordinates and the other uses 1-based
    coordinates.

    Example:
        ```python
        df1 = pb.scan_vcf("file1.vcf", one_based=False)  # 0-based
        df2 = pb.scan_vcf("file2.vcf", one_based=True)   # 1-based
        pb.overlap(df1, df2)  # Raises CoordinateSystemMismatchError
        ```
    """

    pass