scrapling-fetch

Files (34)

• SKILL.md
• conftest.py
• pyproject.toml
• pytest.ini
• __init__.py · scripts/
• __init__.py · scripts/adapters/
• arxiv.py · scripts/adapters/
• doi.py · scripts/adapters/
• openalex.py · scripts/adapters/
• semantic_scholar.py · scripts/adapters/
• download.py · scripts/
• exceptions.py · scripts/
• extract.py · scripts/
• fetch.py · scripts/
• session.py · scripts/
• skill_api.py
• __init__.py · tests/
• arxiv_2310_04673.html · tests/fixtures/
• .gitkeep · tests/fixtures/cassettes/
• openalex_arxiv.json · tests/fixtures/
• __init__.py · tests/live/
• test_real_endpoints.py · tests/live/
• __init__.py · tests/unit/
• test_adapter_arxiv.py · tests/unit/
• test_adapter_doi.py · tests/unit/
• test_adapter_isolation.py · tests/unit/
• test_adapter_openalex.py · tests/unit/
• test_adapter_semantic_scholar.py · tests/unit/
• test_download.py · tests/unit/
• test_exceptions.py · tests/unit/
• test_extract.py · tests/unit/
• test_fetch.py · tests/unit/
• test_session.py · tests/unit/
• test_skill_api.py · tests/unit/

SKILL.md (first lines)

# scrapling-fetch

You are the suite's network boundary. Every outbound HTTP call from any other skill must go through you.

## Operating doctrine

1. Politeness is non-negotiable: rate limit, honor robots.txt, identify with a clear User-Agent.
2. Cache aggressively: same URL within TTL must come from disk, not network.
3. Fail typed: every terminal error maps to one of `FetchFailed | RateLimitExceeded | BlockedRequest | NotAPdf | OfflineMiss | ArxivIdNotFound`.
4. Offline mode is a first-class state (`SCRAPLING_OFFLINE=1`), not a workaround.
5. Adapters are isolated — a layout change on one site does not break the others.

## Sub-workflow routing

| Trigger | Reference | Action |
|---|---|---|
| "scrape this URL" | `references/scrapling-tuning.md` | `fetch.py` |
| "get this paper's metadata" | `references/adapter-authoring.md` | `adapters/arxiv.py` |
| "what does this paper cite" / "what cites this paper" | `references/adapter-authoring.md` | `adapters/openalex.py` (preferred), `adapters/semantic_scholar.py` (fallback) |
| "download this PDF" | `references/scrapling-tuning.md` | `download.py` |
| "extract clean text / readable paragraphs from a page" | `references/scrapling-tuning.md` | `extract.py` |

## Public surface

See `skill_api.py`.

From CharlesHoskinson/russellian-book-suite, MIT.

More from CharlesHoskinson/russellian-book-suite