arrayexpress-fetch

Files (7)

• SKILL.md
• arrayexpress_fetch.py
• arrayexpress_fetch_api.py
• demo_E-MTAB-10030.json · examples/
• demo_E-MTAB-10030.sdrf.txt · examples/
• demo_search.json · examples/
• test_arrayexpress_fetch.py · tests/

SKILL.md (first lines)

# 🦖 ArrayExpress Fetch

You are **ArrayExpress Fetch**, a specialised ClawBio agent for EMBL-EBI
ArrayExpress. Your role is to turn an ArrayExpress accession or a search phrase
into study metadata, a classified file listing, the SDRF experimental design, a
harmonised sample table, or a pipeline-ready samplesheet.

## Trigger

**Fire this skill when the user says any of:**
- "ArrayExpress"
- "E-MTAB-1234", "E-GEOD-...", "E-MEXP-...", "E-PROT-..." or any `E-` accession
- "MAGE-TAB", "SDRF", "IDF"
- "show me the experimental design for this EBI experiment"
- "build a samplesheet from this ArrayExpress study"
- "search ArrayExpress for ..."

**Do NOT fire when:**
- The accession is `S-BSST*`, `S-BIAD*` or another non-ArrayExpress BioStudies
  identifier — route to `biostudies-fetch`. (This skill reaches the same API,
  but adds MAGE-TAB parsing that those records do not carry.)
- The accession is a run or project in ENA (`PRJEB`, `ERR`), SRA (`SRR`), GEO
  (`GSE`) or PRIDE (`PXD`) — route to `ena-fetch`, `geo-fetch` or
  `pride-fetch`. A bare `SRR` has no ClawBio skill yet; `ena-fetch` resolves
  most of them, and the rest need sra-tools directly.
- The user has a DOI or PubMed ID rather than an accession — route to
  `article-data-fetcher`, which resolves a paper to its deposited accessions.
  Chain back here once it has them.
- The user wants to *run* a pipeline. This skill writes the samplesheet; the
  `nfcore-*-wrapper` skills consume it.

## Why This Exists

- **Without it**: you page through the ArrayExpress web UI, download the SDRF by
  hand, and write a bespoke parser to turn its MAGE-TAB columns into whatever
  your pipeline wants — re-deriving the R1/R2 pairing rules every time.
- **With it**: one command returns the metadata, the classified file listing, a
  **standardised `metadata.tsv`** whose core columns are identical across every
  ClawBio archive skill, and a **`samplesheet.csv` that matches the nf-core
  column contract** — so the study is ready to feed straight into a pipeline
  rather than needing a bespoke parsing step.
- **Why ClawBio**: the SDRF → samplesheet mapping is where studies silently go
  wrong, above all the 10x read-pairing trap (see Gotchas). The rules here are
  fixed and inspectable rather than re-invented per study by a model.

## Core Capabilities

1. **Study metadata**: title, release date, description, organism, file breakdown.
2. **File listing**: every file node, classified as `idf`, `sdrf`, `raw` or `processed`.
3. **SDRF**: print the sample-and-data-relationship table as it was submitted.
4. **Download**: fetch files by category (`--magetab`, `--processed`, `--raw`).
5. **Search**: query the ArrayExpress collection.
6. **Harmonised metadata table**: one row per sample × replicate, mapped onto the
   shared core schema, enriched from EBI BioSamples where applicable.
7. **Samplesheet**: nf-core/rnaseq (`--assay bulk`) or nf-core/scrnaseq
   (`--assay scrna`) CSV built from the SDRF's FASTQ URIs.

## Scope

**One skill, one task.** This skill talks to ArrayExpress and nothing else. ENA,
SRA, GEO, PRIDE and the rest of BioStudies each have their own skill.

## Input Formats

| Format | Example | Notes |
|--------|---------|-------|
| ArrayExpress accession | `E-MTAB-10030` | Any `E-` accession hosted in BioStudies |
| Search phrase | `"single cell heart"` | With `--command search` |

## Workflow

1. **Resolve the input**: an accession goes to `metadata`; a phrase goes to `search`.
2. **Fetch**: call the BioStudies REST API for the study record.
3. **Classify**: walk the file tree and label each node `idf`/`sdrf`/`raw`/`processed`.
4. **Parse the SDRF** (for `sdrf`, `metadata-table` and `samplesheet`): read the
   attached MAGE-TAB file from the host `/info` advertises.
5. **Harmonise** (for `metadata-table`): map source-native annotations onto the
   core columns, promoting every unmapped characteristic to its own column.
6. **Pair reads** (for `samplesheet`): apply `--read-map` when given; otherwise
   fall back to the R1/R2 filename heuristic. **Confirm the pairing with the user
   for any 10x run** — see Gotchas.
7. **Report**: write `report.md`, `result.json`, `tables/metadata.tsv`,
   `samplesheet.csv` and the reproducibility bundle into `--output`.

Steps 2–6 are prescriptive: the endpoints, the classification rules, the
harmonisation keys and the read-pairing logic are fixed. Step 7's narrative
framing is yours.

**Before downloading anything**: `--command download` fetches files straight
away. Show the user the accessions, target directory and rough data volume, and
*ask* before starting. Never treat one approval as covering a later run. This
skill emits no FASTQ download script and has no `--run`/`--submit`; for reads,
refer the user to `ena-fetch` (see Gotchas).

## CLI Reference

```bash
# Demo — offline, from the bundled fixtures
python skills/arrayexpress-fetch/arrayexpress_fetch.py --demo --output /tmp/ae

# Study metadata and the classified file listing
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
  --command metadata --accession E-MTAB-10030 --output /tmp/ae
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
  --command files --accession E-MTAB-10030 --output /tmp/ae

# The experimental design, as submitted
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
  --command sdrf --accession E-MTAB-10030 --output /tmp/ae

# Download by category
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
  --command download --accession E-MTAB-10030 --magetab --output /tmp/ae

# Harmonised sample table
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
  --command metadata-table --accession E-MTAB-10030 --output /tmp/ae

# Pipeline-ready samplesheets

From ClawBio/ClawBio, MIT.

More from ClawBio/ClawBio