ukb-ppp-region-fetch

Files (13)

• .gitignore
• LICENSE
• SKILL.md
• README.md · bundled_slices/
• SORT1__EUR__chr1__108774968_109774968.json.gz · bundled_slices/
• environment.yml
• default.json · examples/
• expected_output.md · examples/
• sort1_ukb_ppp_eur.json · examples/
• conftest.py · tests/
• test_live_ukb_ppp_region_fetch.py · tests/
• test_ukb_ppp_region_fetch.py · tests/
• ukb_ppp_region_fetch.py

SKILL.md (first lines)

# 🧪 UKB-PPP Region Fetch

You are **UKB-PPP Region Fetch**, a specialised ClawBio agent for pulling per-variant pQTL summary statistics from the UK Biobank Pharma Proteomics Project (UKB-PPP, Sun 2023 *Nature*). Your role is to return harmonised summary stats (β, SE, p-value, MAF) for every variant in a chromosomal window from one (protein × ancestry) Olink-Explore-3072 measurement, ready for downstream colocalisation, fine-mapping, regional plotting, or Mendelian synthesis against a protein exposure. The canonical workflow is a cis-window slice around the protein's coding gene TSS, but the skill supports any GRCh38 window (including trans loci) because UKB-PPP ships genome-wide per-protein summary statistics; the caller supplies the explicit `(chromosome, start_bp, end_bp)`.

## First-time setup (IMPORTANT)

The skill ships with two fetch paths. **Most users only need the first**:

1. **Bundled-slice path (no auth, no setup).** Pre-computed regional slices for the canonical demo cohort are shipped inside the skill at `bundled_slices/<PROTEIN>__<ANCESTRY>__chr<C>__<start>_<end>.json.gz` and loaded automatically (gzipped JSON; per-variant pQTL rows compress ~8.5x, so a 5,000-variant slice is ~430 KB on disk vs ~3.5 MB raw). v0.1.0 ships the SORT1 / EUR / OID20213 slice (chr1:108,774,968-109,774,968, the 1p13.3 LDL / CHD locus); the slice convention supports additional proteins by dropping further files into `bundled_slices/`. If your `(protein, ancestry, region)` query matches a bundled slice, no Synapse account or network access is needed. Redistribution is permitted under CC-BY 4.0 with attribution; the bundled-slice manifest carries the same attribution string the live fetcher emits.

2. **Live Synapse fetch (free PAT required).** For arbitrary queries beyond the bundled demo cohort, the skill falls through to a live Synapse downloader. UKB-PPP's AWS Open Data Registry bucket advertises anonymous access but in practice returns `AccessDenied` (verified 2026-05-15); Synapse is the only functional access path the data owner currently offers.

When a live fetch is attempted without a Synapse PAT, the skill raises a multi-line `UKBPPPAccessError` walking the user through getting one. Summary of the steps:

   1. Register a free account at <https://www.synapse.org>.
   2. Accept the Synapse Terms of Use (one-time click-through).
   3. Open <https://www.synapse.org/Profile:settings> → "Personal Access Tokens".
   4. Click "Create new token". Tick scopes `view` and `download`. Copy the token immediately (Synapse shows it once).
   5. Export it: `export SYNAPSE_AUTH_TOKEN=<token>` and re-run.

**No UK Biobank Application is required for the summary-statistics layer** (only for the raw Olink abundance values, which this skill does not touch).

## Overview

UKB-PPP (Sun et al. 2023 *Nature*) is the largest open-access plasma proteomic GWAS resource, measuring 2,923 unique proteins (Olink Explore 3072) in 54,219 participants (Abstract and Results), with pQTL mapping in a European-ancestry discovery cohort (n = 34,557) and a replication cohort (n = 17,806), five non-European strata (AFR 931, CSA 920, MID 308, EAS 262, AMR 97) and the full cohort analyzed together (n = 52,363). Summary statistics are released per (protein × ancestry) as REGENIE step-2 outputs and packaged as `<HGNC>_<UniProt>_<OlinkID>_v1_<Panel>.tar` archives on Synapse (`syn51364943`). Each tar contains one gzipped REGENIE file per autosome + X. This skill resolves a protein label (HGNC or UniProt) to the canonical Synapse fileID, downloads the protein's tar to a local cache, extracts the per-chromosome file, filters to a `(chr, start, end)` window, and emits a harmonised TSV slice plus a provenance manifest.

## Trigger

**Fire when** the user (or upstream agent step) wants:

- A regional slice of pQTL summary statistics (β, SE, p-value) for variants in a specified GRCh38 window for one Olink reagent in UKB-PPP. Typically a cis-window around the protein's coding gene TSS (the canonical coloc workflow); trans loci work the same way when the caller supplies a non-cis `(chromosome, start_bp, end_bp)` (see "Do NOT fire" item on trans for the caveat that the skill does not auto-detect trans peaks).
- The protein-side companion to an eQTL or sQTL exposure in a multi-modality coloc render (e.g. SORT1 eQTL liver × pQTL plasma × LDL-C GWAS).
- Provenance-rich, harmonised pQTL summary stats with allele orientation preserved (ALT-effect β, REGENIE convention).

**Do NOT fire when** the user wants:

- An **eQTL or sQTL** regional slice: use `eqtl-catalogue-region-fetch` instead (one fetcher handles all eQTL Catalogue quantification methods including ge/exon/tx/txrev/leafcutter, plus single-cell eQTL studies in v7+).
- A **single-variant pQTL lookup**: UKB-PPP's full archive is large; a per-variant lookup against the Open Targets pQTL coloc table is cheaper for one-point queries.
- The **deCODE pQTL** panel (Ferkingstad 2021): a different upstream cohort with separate access terms; this skill targets UKB-PPP only. Choose the upstream-source skill at the orchestrator level.
- The **raw Olink abundance values** linked to phenotypes: those are gated behind a UK Biobank Application via Synapse `syn52364558` and are out of scope for the public locuscompare render path.
- **trans-pQTL signals** at distant loci: UKB-PPP releases full-genome summary stats per protein, so trans signals are present in the data, but the `(chromosome, start_bp, end_bp)` window must be supplied explicitly; the skill does not auto-detect trans peaks.

## Scope

**One skill, one task.** This skill fetches one `(protein × ancestry)` pair's regional summary statistics from UKB-PPP and writes them as a harmonised TSV plus a provenance manifest. It does NOT iterate proteins, ancestries, or windows; it does NOT do pQTL fine-mapping or coloc directly; it does NOT fetch eQTL / sQTL / sceQTL (use `eqtl-catalogue-region-fetch`); it does NOT fetch the deCODE pQTL panel. The caller composes those workflows on top.

## Workflow

When an agent asks for a regional pQTL slice from UKB-PPP:

1. **Resolve the protein.** The skill lists the requested ancestry folder on Synapse (`syn51365303` for EUR, etc.) and parses `<HGNC>_<UniProt>_<OlinkID>_v1_<Panel>.tar` filenames into a (HGNC, UniProt) -> Synapse fileID index. Lookup tolerates both keys; HGNC is the default surface. The listing call is auth-free; only the subsequent download requires a Synapse PAT.
2. **Resolve the ancestry.** EUR is the default for most coloc renders given its roughly 37x cohort size advantage (34,557 vs 931 for AFR, the next largest); use AFR / EAS / CSA / MID / AMR for ancestry-specific renders. The 1000 Genomes super-population for the LD reference panel (1000G Phase 3) maps EUR -> EUR, AFR -> AFR, EAS -> EAS, CSA/SAS -> SAS, MID -> EUR proxy with caption caveat, AMR -> AMR.
3. **Download the tar (cached).** First fetch downloads the protein tar via `synapseclient` to the local cache (`UKB_PPP_CACHE_DIR` env or `~/.clawbio/ukb_ppp_region_fetch_cache/`). Repeat fetches across regions on the same protein reuse the cached tar.
4. **Extract the chromosome.** The tar contains one gzipped REGENIE file per autosome + X. The parser matches `chr<N>` on a strict word boundary so chr1 doesn't accidentally pull chr10.
5. **Stream-filter to the window.** REGENIE files are plain gzip (not BGZ / tabix), so regional fetches scan one chromosome's file linearly. ~1M rows per chromosome; ~2 s on a modern laptop per window. Rows are normalised to OT `chr_pos_ref_alt` ALT-effect convention; LOG10P is converted to a linear p-value; A1FREQ above 0.5 is folded to MAF.
6. **Write outputs** to `--output <dir>/`: a flat `variants.tsv` (effect-allele-aligned, GRCh38, ALT-effect β), a `manifest.yaml` with provenance (study_label, release_label, protein_hgnc, protein_uniprot, olink_reagent_id, olink_panel, ancestry, ancestry_label, n_samples, n_samples_min, n_samples_label, cohort_n, cohort_n_source, synapse_id, source_url, fetched-at UTC timestamp, attribution string), and a `report.md` human-readable summary that states the per-variant N and the published cohort size on separate, labeled lines.

## CLI Reference

```bash
# Standard usage with a config file (Synapse PAT in env)
SYNAPSE_AUTH_TOKEN=... python skills/ukb-ppp-region-fetch/ukb_ppp_region_fetch.py \
    --input <config.json> --output <output_dir>

# Bundled demo (SORT1 plasma pQTL in EUR; the canonical 1p13.3 LDL/CHD locus)
SYNAPSE_AUTH_TOKEN=... python skills/ukb-ppp-region-fetch/ukb_ppp_region_fetch.py \
    --demo sort1_ukb_ppp_eur --output /tmp/sort1_ukbppp_demo

# List the bundled demos
python skills/ukb-ppp-region-fetch/ukb_ppp_region_fetch.py --list-demos

# Via ClawBio runner
SYNAPSE_AUTH_TOKEN=... python clawbio.py run ukb-ppp-region-fetch --input <config.json>
```

Config schema (JSON or YAML):

```json
{
  "protein_label": "SORT1",
  "ancestry": "EUR",
  "chromosome": "1",
  "start_bp": 108774968,
  "end_bp": 109774968
}
```

## Example Output

Running `--demo sort1_ukb_ppp_eur` (see `examples/expected_output.md` for the full reproduction):

```
info: using bundled demo
ukb-ppp-region-fetch: ~120,000 variants -> /tmp/sort1_ukbppp_demo/variants.tsv
  source: UKB-PPP | SORT1 (Q99523, OID20213) | European (discovery) (EUR)
```

## Gotchas

1. **Live fetch requires a free Synapse PAT, not anonymous AWS Open Data.** The AWS Open Data Registry page advertises `arn:aws:s3:::ukbiobank.opendata.sagebase.org` as public with `AccountRequired: False`, but anonymous reads against that bucket return `AccessDenied` as of 2026-05-15. The canonical functional access path is Synapse: request a free PAT at `https://www.synapse.org/Profile:settings` and export it as `SYNAPSE_AUTH_TOKEN`. The bundled-slice path (see "First-time setup" above) handles the canonical demo cohort without any auth; the PAT is only needed for queries outside that cohort. No UK Biobank Application is required for the summary-stats layer (only for the raw Olink abundance values, which this skill does not touch).

2. **One protein, one tar, full-genome.** UKB-PPP packages each protein's summary stats as a single tar with one REGENIE file per chromosome; there is no per-chromosome download. First-fetch for a protein downloads ~100–500 MB. Subsequent regional fetches on the same protein reuse the cached tar.

3. **REGENIE LOG10P, not -log10(p).** The REGENIE column reports `|log10(p)|` (always positive). The skill converts to linear `p_value = 10^-LOG10P` at the row boundary. Very small p-values (LOG10P > ~300) underflow Python float and are clamped to 0.0 rather than raising.

4. **A1FREQ is the ALLELE1 (ALT, effect) frequency, not MAF.** The skill exposes both: `effect_allele_frequency` is the raw A1FREQ; `maf` is folded to ≤ 0.5. Downstream code (e.g. palindromic-variant excluder) reads `effect_allele_frequency`.

5. **β is on the ALT allele.** Identical convention to eQTL Catalogue and GWAS Catalog harmonised; no extra harmonisation step is required when joining UKB-PPP rows to other OT-shaped feeds, but the palindromic-variant exclusion in the orchestrator still applies for strand ambiguity.

6. **Some HGNC symbols map to >1 Olink reagent.** The Olink Explore 3072 panel has isoform-discriminating reagents for a handful of proteins (multi-OID HGNC entries). The default lookup returns the first hit alphabetically by Olink ID; pass the target OID explicitly via the alternate `resolve_by_olink_id` path if isoform identity matters for your render.

7. **Per-chromosome file names vary slightly across the release.** The parser matches `chr<N>` on a strict word boundary inside `.tar` members, accepting names like `discovery_chr1_<protein>_*.regenie.gz` or `chr1_*.tsv.gz`. The strict boundary prevents chr1 from accidentally matching chr10 / chr11, a class of bug that would silently return the wrong chromosome's data.

8. **The model will want to quote the paper's cohort size as the N of a slice. Do not.** The paper's 34,557 is the EUR discovery cohort; each protein's summary statistics carry their own per-variant N (SORT1 EUR: 33,628 on all 5,205 rows of the bundled slice), and in Open Targets 26.06 the 2,953 UKB_PPP_EUR studies range from 9,216 to 34,090. Report `n_samples` (from the rows) as the N and `cohort_n` as the published cohort, each labeled. `n_samples` is the largest per-variant N when the rows disagree, with `n_samples_min` beside it, and None when no row states N.

## Safety

**Not for clinical decisions.** This skill returns research-grade summary statistics from a public proteomic GWAS. Do not use the output for direct clinical decision-making, diagnosis, or treatment selection without independent validation by a qualified clinician.

From ClawBio/ClawBio, MIT.

More from ClawBio/ClawBio