general-publisher-access-guard

Files (12)

• SKILL.md
• 10.1021_acs.jpclett.7b00189_data.json · examples/acs-blocked-paper/
• 10.1021_acs.jpclett.7b00189_summary.md · examples/acs-blocked-paper/
• README.md · examples/acs-blocked-paper/
• 10.1103_PhysRevLett.120.145301_data.json · examples/oa-repository-bypass/
• 10.1103_PhysRevLett.120.145301_summary.md · examples/oa-repository-bypass/
• 10.1103_PhysRevLett.120.145301_text.txt · examples/oa-repository-bypass/
• README.md · examples/oa-repository-bypass/
• publisher_policy_matrix.json · resources/
• audit_publishers.py · scripts/
• check_publisher.py · scripts/
• safe_paper_retriever.py · scripts/

SKILL.md (first lines)

# General Publisher Access Guard

## Goal
To prevent autonomous research agents from being blocked by scientific journal publisher web application firewalls (Cloudflare Turnstile, Radware CAPTCHA, Akamai, Imperva) and HTTP 403/429 error loops, by auditing publisher anti-bot policies and automatically routing paper retrieval through legal open-access APIs, repository mirrors, and reconstructed abstract services.

## Instructions

When the user asks you to fetch, read, summarize, or analyze scientific literature from specific URLs or DOIs, **NEVER** use direct web scraping (`read_url_content`, `curl`, or raw `requests.get`) on known bot-blocking publisher domains. Follow the three steps below:

### 1. Pre-Flight Publisher Assessment
Before attempting to download or scrape any paper URL or DOI, inspect the domain against the curated publisher policy matrix.

```bash
${CLAUDE_SKILL_DIR}/../../venv/run cpu python ${CLAUDE_SKILL_DIR}/scripts/check_publisher.py --url "https://pubs.acs.org/doi/10.1021/acs.chemmater.9b04758"
```
Or with DOI:
```bash
${CLAUDE_SKILL_DIR}/../../venv/run cpu python ${CLAUDE_SKILL_DIR}/scripts/check_publisher.py --doi "10.1103/PhysRevLett.120.145301" --json
```

**Decision Rule:**
- If `direct_scraping_safe: false` (status `BLOCKED` or `API ONLY`), you **MUST NOT** scrape or curl the publisher web page. Proceed immediately to Step 2.
- Known blocked domains include: `pubs.acs.org`, `journals.aps.org`, `link.aps.org`, `sciencedirect.com`, `link.springer.com`, `nature.com`, `onlinelibrary.wiley.com`, `pubs.rsc.org`, `iopscience.iop.org`, `pubs.aip.org`, `ieeexplore.ieee.org`, `science.org`, `pnas.org`, `tandfonline.com`, `academic.oup.com`, and `mdpi.com`.

### 2. Execute the Safe Retrieval Cascade
Run the safe paper retriever to query legitimate APIs in priority order:

1. **Official Publisher APIs**: If credentials (`ELSEVIER_API_KEY`, `SPRINGER_API_KEY`) are present in environment variables.
2. **OpenAlex Works API**: Retrieves complete bibliographic metadata (authors, journal, citations, publication year) and reconstructs the full abstract from the inverted index.
3. **Filtered Unpaywall API**: Searches for legal Green/Gold Open Access copies hosted in institutional and disciplinary repositories (e.g. arXiv, PubMed Central, Zenodo, university `.edu` repositories), **automatically discarding any link pointing back to a blocked publisher domain**.
4. **Europe PMC API**: Direct open access XML/PDF retrieval for biomedical and chemistry literature.
5. **Preprint APIs**: Queries arXiv API or ChemRxiv API with required rate throttling (minimum 3 seconds delay).

```bash
${CLAUDE_SKILL_DIR}/../../venv/run cpu python ${CLAUDE_SKILL_DIR}/scripts/safe_paper_retriever.py \
    --doi "10.1103/PhysRevLett.120.145301" \
    --output_dir ./papers
```

The script outputs:
- `<safe_doi>_summary.md`: Structured markdown report with title, authors, citations, reconstructed abstract, and access notes.
- `<safe_doi>.pdf`: Downloaded open-access PDF (if available in a safe repository).
- `<safe_doi>_text.txt`: Plaintext extracted from the PDF via PyMuPDF.

### 3. Graceful Handling of Paywalled Literature
If no open-access copy exists across repository mirrors:
- **DO NOT** retry scraping the publisher website.
- Read and synthesize the abstract provided in `<safe_doi>_summary.md`.
- Inform the user that the full text is restricted by publisher paywalls/WAFs, and request them to upload the institutional PDF to the workspace if full-text extraction is required.

### 4. Auditing Publisher Defenses
To test live publisher WAF defenses or verify if a publisher's bot policy has updated:

```bash
${CLAUDE_SKILL_DIR}/../../venv/run cpu python ${CLAUDE_SKILL_DIR}/scripts/audit_publishers.py --publisher acs
```
Or audit all major publishers:
```bash
${CLAUDE_SKILL_DIR}/../../venv/run cpu python ${CLAUDE_SKILL_DIR}/scripts/audit_publishers.py --all
```

## Examples

### Example 1: Bypassing ACS Bot Block for Perovskite Battery Paper
Handling an ACS paper (`10.1021/acs.jpclett.7b00189`) where direct web scraping returns Cloudflare Turnstile 403:
```bash
${CLAUDE_SKILL_DIR}/../../venv/run cpu python ${CLAUDE_SKILL_DIR}/scripts/safe_paper_retriever.py \
    --doi "10.1021/acs.jpclett.7b00189" \
    --output_dir examples/acs-blocked-paper
```
See [examples/acs-blocked-paper/README.md](examples/acs-blocked-paper/README.md) for full execution traces and validation against published literature.

### Example 2: Resolving APS Paper via Legal Open-Access Repository Mirror
Handling an APS Physical Review Letters paper (`10.1103/PhysRevLett.120.145301`) by bypassing blocked `link.aps.org/pdf` and fetching the author manuscript from OSTI/MIT DSpace:
```bash
${CLAUDE_SKILL_DIR}/../../venv/run cpu python ${CLAUDE_SKILL_DIR}/scripts/safe_paper_retriever.py \
    --doi "10.1103/PhysRevLett.120.145301" \
    --output_dir examples/oa-repository-bypass
```
See [examples/oa-repository-bypass/README.md](examples/oa-repository-bypass/README.md) for step-by-step instructions.

## Constraints
- **Environment**: Requires the `cpu` environment (contains `requests`, `pymupdf` / `fitz`).
- **Zero Direct Scrape**: Never execute `curl`, `read_url_content`, or browser automation on blocked publisher domains.
- **Unpaywall Polite Pool**: Provide an email address in `UNPAYWALL_EMAIL` or `OPENALEX_EMAIL` environment variable to access high-throughput API pools.
- **Preprint Rate Limits**: arXiv requests must maintain a minimum 3-second crawl delay.
- **Institutional Access**: Paywalled papers without green open-access copies cannot be legally decrypted without subscriber credentials or manual PDF upload.

## References
- Priem, J., Piwowar, H., & Orr, R., "OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts", *arXiv:2205.01833*, 2022. [DOI: 10.48550/arXiv.2205.01833](https://doi.org/10.48550/arXiv.2205.01833)
- Piwowar, H. et al., "The state of OA: a large-scale analysis of the prevalence and impact of Open Access articles", *PeerJ*, 6:e4375, 2018. [DOI: 10.7717/peerj.4375](https://doi.org/10.7717/peerj.4375)
- Ferguson, C. et al., "Europe PMC: a stocktake by the Europe PMC Consortium", *Nucleic Acids Research*, 49(D1):D1507-D1514, 2021. [DOI: 10.1093/nar/gkaa994](https://doi.org/10.1093/nar/gkaa994)
- Cloudflare AI Crawl Control & Bot Management Whitepaper, 2024.

From learningmatter-mit/AtomisticSkills, MIT.

More from learningmatter-mit/AtomisticSkills