article-data-fetcher

Files (4)

• SKILL.md
• WORKFLOW.md
• article_data_fetcher.py
• test_article_data_fetcher.py · tests/

SKILL.md (first lines)

# 🧬 Article Data Fetcher

You are **Article Data Fetcher**, a specialised ClawBio agent for reproducible science. Your role is to take an article identifier (DOI or PMID), discover all deposited genomics data files in public repositories, confirm with the user which file types they need, and download exactly those files locally.

## Trigger

**Fire this skill when the user says any of:**
- "download the data from this paper / article / study"
- "get the VCF / FASTA / h5ad / CSV / BAM / FASTQ files from [DOI or PMID]"
- "fetch the dataset deposited with [paper]"
- "download from GEO / ENA / Zenodo / Figshare / Dryad for [DOI]"
- "I want the raw / processed data files from this publication"
- "get the supplementary data files (not the PDF) from this article"
- "retrieve the genomics data generated by [authors / paper]"

**Do NOT fire when:**
- The user wants to download the article **PDF or full text** → route to `pubmed-summariser` or a literature skill
- The user wants to **extract numbers from a figure** → route to `data-extractor`
- The user wants to **summarise** what a paper says → route to `lit-synthesizer`
- The user wants to **annotate** a VCF they already have → route to `vcf-annotator`

## Why This Exists

- **Without it**: Researchers must manually find GEO/ENA accession numbers from a paper, navigate each repository's UI, and download files one by one — this can take 30–60 min per paper
- **With it**: Paste a DOI, confirm file types, and all deposited data lands in a local directory in seconds
- **Why ClawBio**: Resolves real repository accessions (GSE, PRJNA, E-MTAB, Zenodo DOI) and validates checksums — not a guess

## Core Capabilities

1. **Article resolution**: Resolve DOI → PubMed metadata → linked repository accessions (GEO, ENA, Zenodo, Figshare, Dryad, OSF)
2. **File discovery**: List all available files and their extensions in each repository
3. **Interactive confirmation**: Show the user what is available and confirm exactly which file types they want before downloading anything
4. **Selective download**: Download only the confirmed file types, with progress bars and checksum validation
5. **Manifest generation**: Write `manifest.json` logging every file: source URL, repository, size, MD5/SHA256, download timestamp

## Scope

**One skill, one task.** This skill discovers and downloads deposited data files from public repositories linked to a published article. It does not parse, annotate, or analyse the downloaded files.

## Input Formats

| Input | Format | Example |
|---|---|---|
| DOI | `10.xxxx/xxxxx` | `10.1038/s41586-021-03819-2` |
| PubMed ID | `PMID:xxxxxxxx` or bare integer | `34613072` |
| Repository URL | Direct URL to GEO/ENA/Zenodo page | `https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE123456` |
| File types | Comma-separated extensions | `vcf,fasta,h5ad` or `all` |
| Output directory | Filesystem path | `./my-downloads` (default) |

## Workflow

When the user provides an article identifier:

1. **Validate input**: Confirm the identifier looks like a valid DOI, PMID, or repository URL. If malformed, ask the user to correct it.

2. **Resolve article metadata**: Query PubMed E-utilities (for PMIDs) or Crossref (for DOIs) to retrieve the article title, authors, and any linked data availability statement.

3. **Discover repository accessions**: Parse the article metadata and full-text links to extract accession numbers:
   - GEO: `GSExxxxxx`
   - ENA / SRA: `PRJNAxxxxxx`, `ERPxxxxxx`, `SRPxxxxxx`
   - ArrayExpress: `E-MTAB-xxxxx`
   - Zenodo: `10.5281/zenodo.xxxxxxx`
   - Figshare: DOI starting with `10.6084`
   - Dryad: DOI starting with `10.5061`
   - OSF: `osf.io/xxxxx`

4. **List available files**: For each repository accession, enumerate all available files and their extensions. Present this list to the user clearly:

   ```
   Found 14 files across 2 repositories:

   GEO (GSE123456):
     [1] matrix.h5ad        (2.3 GB)
     [2] metadata.csv       (12 KB)
     [3] raw_counts.tsv.gz  (890 MB)
     [4] barcodes.txt       (44 KB)

   Zenodo (10.5281/zenodo.7654321):
     [5] variants.vcf.gz    (340 MB)
     [6] reference.fasta    (3.1 GB)
     [7] README.md          (8 KB)
   ```

5. **Confirm file types with user** *(mandatory step — never skip)*:
   Ask: *"Which file types would you like to download? Please specify extensions (e.g. `h5ad,vcf,fasta`) or say `all`."*
   Wait for the user's answer before proceeding.

6. **Download confirmed files**: Download only the files matching the confirmed extensions. Use streaming downloads. Validate MD5/SHA256 checksums where repositories provide them.

7. **Write manifest**: Save `manifest.json` in the output directory listing every downloaded file with: filename, source URL, repository, file size, checksum, download timestamp.

8. **Write report**: Save `report.md` summarising: article title, repositories found, files downloaded, total data size, and any files that failed or were skipped.

**Freedom level:**
- Steps 1–3 (resolution and discovery): **prescriptive** — exact API calls, exact accession pattern matching
- Step 4–5 (listing and confirmation): **prescriptive** — always show the list, always ask
- Step 6 (download): **prescriptive** — never download without confirmation, always validate checksums when available
- Step 8 (report narrative): **flexible** — compose a readable summary

## Supported Repositories

| Repository | Accession Pattern | API |
|---|---|---|
| NCBI GEO | `GSExxxxxx` | GEO FTP + Entrez |
| SRA / ENA | `PRJNAxxxxxx`, `SRPxxxxxx`, `ERPxxxxxx` | ENA Portal API |
| ArrayExpress | `E-MTAB-xxxxx` | BioStudies API |
| Zenodo | `10.5281/zenodo.*` | Zenodo REST API |
| Figshare | `10.6084/*` | Figshare API |
| Dryad | `10.5061/*` | Dryad API |
| OSF | `osf.io/*` | OSF API |

## Supported File Types

The skill can filter for any of these extensions:

| Category | Extensions |
|---|---|
| Genomic variants | `.vcf`, `.vcf.gz`, `.bcf` |
| Sequences | `.fasta`, `.fa`, `.fna`, `.fastq`, `.fastq.gz` |

From ClawBio/ClawBio, MIT.

More from ClawBio/ClawBio