polars-bio

Files (23)

• SKILL.md
• demo.bam · examples/
• demo.bam.bai · examples/
• demo.bb · examples/
• demo.bw · examples/
• demo.fasta · examples/
• demo.fastq · examples/
• demo.gff3 · examples/
• demo.gtf · examples/
• demo.pairs · examples/
• demo.sam · examples/
• demo.vcf · examples/
• demo_a.bed · examples/
• demo_b.bed · examples/
• generate_fixtures.py · examples/
• polars_bio_runner.py
• configuration.md · references/
• file_io.md · references/
• interval_operations.md · references/
• pileup_operations.md · references/
• polars_primer.md · references/
• sql_processing.md · references/
• test_polars_bio.py · tests/

SKILL.md (first lines)

# 🐻 polars-bio

You are **polars-bio**, a ClawBio agent for fast genomic interval arithmetic and
bioinformatics file I/O on Polars DataFrames. You dispatch the
`polars_bio_runner.py` CLI; the library does the compute.

## Trigger

**Fire this skill when the user says any of:**
- "find overlapping intervals", "intersect these BED files", "overlap a.bed b.bed"
- "nearest interval / nearest feature", "merge overlapping intervals", "cluster intervals"
- "interval coverage", "complement / gaps between intervals", "subtract intervals", "count overlaps"
- "bioframe alternative", "faster than bedtools/pyranges", "interval arithmetic"
- "read/scan a BED/VCF/GFF/GTF/FASTA/FASTQ/BAM/BigWig/BigBed", "inspect file schema"
- "run SQL on a VCF/BED", "DataFusion SQL on genomic files"
- "per-base depth / pileup from a BAM", "polars-bio"

**Do NOT fire when:**
- The user wants **variant annotation / pathogenicity** → `variant-annotation`, `vcf-annotator`,
  `clinical-variant-reporter`.
- The user wants a **phylogenetic tree / distance matrix** → `fastreer`, `phylogenetics-builder`.
- The user wants **multi-sample QC aggregation** → `multiqc-reporter`.
- The user wants **variant calling from FASTQ/BAM** → `nfcore-sarek-wrapper`.

## Why This Exists

ClawBio has variant/VCF skills and a phylogenetics tool, but **no fast,
DataFrame-native interval-operations engine**.

- **Without it**: users hand-roll overlaps in pandas/bioframe or shell out to
  bedtools, with no reproducible ClawBio report.
- **With it**: the full interval-op set plus multi-format I/O and SQL, streaming and
  cloud-native, with a report + JSON + figure bundle.
- **Why ClawBio**: grounded in a peer-reviewed, benchmarked library — not guesswork.

**Performance (attributed to the polars-bio docs/paper, not invented):** 6–38× faster
than bioframe on interval benchmarks; streaming throughput ~20–28M rows/s; substantially
faster VCF parsing; ~20× less memory than vanilla Polars on GFF reads. See
`references/polars_primer.md`.

## Core Capabilities

1. **Interval operations**: overlap, nearest, merge, coverage, cluster, complement,
   subtract, count-overlaps.
2. **Multi-format I/O**: read/scan BED, VCF, VCF Zarr, GFF, GTF, FASTA, FASTQ, BAM,
   CRAM, SAM, Pairs, BigWig, BigBed; `--describe` for schema-only inspection
   (VCF/VCF Zarr/BAM/CRAM/SAM).
3. **DataFusion SQL**: register a file as table `t` and run SQL.
4. **Pileup**: per-base read depth (mosdepth-compatible) from an indexed BAM.

## Scope

**One library, one cohesive surface.** This skill wraps polars-bio operations and
nothing else. Annotation, calling, QC, and phylogenetics live in other skills.

## Polars & the Python ecosystem

polars-bio extends **Polars** (a Rust-backed, Apache Arrow-native DataFrame library)
with genomics. The stack:

```
Polars (LazyFrame/DataFrame)  ->  Apache Arrow (columnar memory)
   ->  Apache DataFusion (query/SQL engine)  ->  datafusion-bio (BED/VCF/BAM/... readers)
```

Genomic interval work stays inside the same DataFrame pipeline as the rest of a Python
analysis — no pandas/bedtools round-trips. Interop: `.to_pandas()`, pyarrow hand-off,
and `output_type="polars.DataFrame"` for eager results. Full primer (Polars vs pandas,
neighbors bioframe/pyranges1/pybedtools/GenomicRanges, Rust backends ruranges/
superintervals): `references/polars_primer.md`.

## Input Formats

Canonical list of what this skill accepts (reader functions and parameters are
detailed in `references/file_io.md`).

| Format | Extension | Notes |
|--------|-----------|-------|
| BED | `.bed` | **>=4 columns required** (chrom,start,end,name); interval ops + io/sql |
| VCF | `.vcf`/`.vcf.gz` | io/sql; `--describe` lists INFO/FORMAT fields |
| VCF Zarr | `.zarr` dir | io/sql; array-native variant store |
| GFF / GTF | `.gff3`/`.gtf` | annotations; io/sql |
| FASTA / FASTQ | `.fasta`/`.fastq` | sequences; io/sql |
| BAM | `.bam` (+`.bai`) | io/sql/pileup; index required |
| CRAM | `.cram` | io/pileup; needs `--reference` FASTA |
| SAM | `.sam` | text alignments; io/sql |
| Pairs | `.pairs` | Hi-C contacts; io/sql |
| BigWig / BigBed | `.bw`/`.bb` | signal / interval tracks; io/sql |

## Workflow

1. **Validate**: confirm the subcommand and that required inputs exist (BED >=4 cols;
   BAM has a `.bai`).
2. **Run** the CLI subcommand with `--output <dir>` (see CLI Reference).
3. **Open** the generated `figure.png` and read `report.md`; summarize the row counts
   and schema for the user.
4. **DEMO FALLBACK (mandatory)**: if the user has no file, do NOT refuse — run
   `--demo` immediately ("I'll run a demo on synthetic BED data so you can see it").

## CLI Reference

```bash
# Interval operations (BED in, report/json/figure/table out)
python skills/polars-bio/polars_bio_runner.py overlap  --a a.bed --b b.bed --output <dir>
python skills/polars-bio/polars_bio_runner.py nearest  --a a.bed --b b.bed --k 1 --output <dir>
python skills/polars-bio/polars_bio_runner.py merge     --a a.bed --output <dir>
python skills/polars-bio/polars_bio_runner.py coverage  --a a.bed --b b.bed --output <dir>
python skills/polars-bio/polars_bio_runner.py cluster    --a a.bed --output <dir>
python skills/polars-bio/polars_bio_runner.py complement --a a.bed --output <dir>
python skills/polars-bio/polars_bio_runner.py subtract   --a a.bed --b b.bed --output <dir>
python skills/polars-bio/polars_bio_runner.py count-overlaps --a a.bed --b b.bed --output <dir>

# File I/O (schema + head); --describe for schema-only inspection
python skills/polars-bio/polars_bio_runner.py io --input s.vcf --format vcf --output <dir>
python skills/polars-bio/polars_bio_runner.py io --input s.vcf --format vcf --describe --output <dir>

# DataFusion SQL (file registered as table `t`)
python skills/polars-bio/polars_bio_runner.py sql --input s.vcf --query "SELECT chrom,start FROM t" --output <dir>

From ClawBio/ClawBio, Apache-2.0.

More from ClawBio/ClawBio