nature-downloader

Files (64)

• .gitattributes
• .gitignore
• LICENSE
• README.md
• README_EN.md
• SKILL.md
• openai.yaml · agents/
• banner.jpg · assets/
• publishers.json · data/
• school.schema.json · data/
• schools.yaml · data/
• nature-downloader-reference-research-2026-07-15.md · docs/
• 2026-06-21-nature-downloader-merge.md · docs/superpowers/plans/
• carsi-retry-template.tsv · examples/
• manifest-template.tsv · examples/
• publisher-verification-template.tsv · examples/
• manifest.yaml
• requirements.txt
• batch_download.mjs · scripts/
• browser_pdf_downloader.mjs · scripts/
• cdp_open_url.mjs · scripts/
• configure_credentials.py · scripts/
• configure_school.py · scripts/
• extract_pdf_text.py · scripts/
• anti-bot.mjs · scripts/lib/
• cdp-utils.mjs · scripts/lib/
• cnki.mjs · scripts/lib/
• credentials.mjs · scripts/lib/
• direct-download.mjs · scripts/lib/
• manifest.mjs · scripts/lib/
• metadata.mjs · scripts/lib/
• open-access-provider.mjs · scripts/lib/
• open-access.mjs · scripts/lib/
• pdf-utils.mjs · scripts/lib/
• provider-registry.mjs · scripts/lib/
• provider-utils.mjs · scripts/lib/
• publisher-providers.mjs · scripts/lib/
• routing.mjs · scripts/lib/
• school-config.mjs · scripts/lib/
• settings.mjs · scripts/lib/
• status-codes.mjs · scripts/lib/
• wos-supporting-information.mjs · scripts/lib/
• retry_elsevier_api.mjs · scripts/
• config.py · src/
• health_check.py · src/
• schools_loader.py · src/
• validators.py · src/
• wizard.py · src/
• test_config_credentials.py · tests/python/
• test_config_wizard.py · tests/python/
• batch-cli.test.mjs · tests/unit/
• cnki.test.mjs · tests/unit/
• credentials.test.mjs · tests/unit/
• generic-package.test.mjs · tests/unit/
• manifest.test.mjs · tests/unit/
• metadata.test.mjs · tests/unit/
• open-access-provider.test.mjs · tests/unit/
• open-access.test.mjs · tests/unit/
• provider-utils.test.mjs · tests/unit/
• publisher-providers.test.mjs · tests/unit/
• …and 4 more

SKILL.md (first lines)

# Nature Literature Downloader

This skill routes literature through lawful open-access, publisher-API, CNKI institutional, and browser-based institutional providers. `scripts/batch_download.mjs` is the orchestration entry point; school configuration, publisher credentials, metadata/OA resolution, provider downloads, content validation, and manifests are separate modules.

Verified routes are examples, not defaults. Every institution should start from the user's actual library resource URL, because resource portals, CAS callbacks, EZproxy, WebVPN, IP-authenticated database pages, and database detail pages reveal the live authorization path more reliably than a school name.

> **SI confirmation gate — do this first.** Before downloading any PDF, CAJ, HTML, XML, archive, or attachment, ask whether the user wants Supporting Information. An explicit request for SI counts as yes; an explicit request for正文 only counts as no. Otherwise ask once for the whole batch. Run the downloader with exactly one of `--si` or `--no-si`. Without either flag the script returns `si_confirmation_required` and does not create the output directory.

> **Main workflow.** Normalize the DOI/title and identify language and publisher before routing. Chinese literature always uses CNKI. For English Elsevier, Springer Nature, and IEEE articles with usable provider credentials, try the publisher API first and do not require an OA determination after a successful API download. If that API attempt fails, automatically check legitimate OA sources. Other English publishers check OA first, then use the institutional Web Access route when OA is unavailable.

```text
规范化 DOI/题名并识别语言、出版商
├─ 中文文献：直接走 CNKI
└─ 英文文献
   ├─ Elsevier / Springer Nature / IEEE，且已配置有效 Key
   │  ├─ 优先通过出版商 API 下载
   │  ├─ API 下载成功：结束，不强制判断 OA
   │  └─ API 下载失败：检查文章级 OA，再走 PMC / Unpaywall / 合法仓储
   └─ 其他出版商
      ├─ 检查文章级 OA
      └─ OA 不可用：走 Web Access 机构授权
```

> **Chinese literature is CNKI-only.** A Chinese title, `zh` metadata language, explicit CNKI source URL, or `--route cnki` must use CNKI even if another OA copy appears to exist. Reuse the user's current Chrome library/CNKI login state and prefer configured `discovery.cnki_url`. Never export cookies or collect the institutional password.

> **Publisher API fallback.** A valid API key does not guarantee full-text entitlement. When an Elsevier, Springer Nature, or IEEE API attempt returns no entitlement or no usable full text, automatically try legitimate OA sources first. Return `api_fallback_confirmation_required` and ask once whether to use Web Access only after both the publisher API and OA routes fail. Do not switch to institutional Web Access automatically.

> **Browser-state principle.** Authorized downloads depend on the exact browser profile where the user is logged in. If a proxy, CDP session, or browser automation tool opens a fresh profile or a different browser with no login state, do not treat the failure as missing library permission. Switch to a control path that reuses the user's active browser session, or ask the user to authenticate in the controlled browser instance.

> **Format principle.** PDF, HTML full text, and database-native formats such as CAJ are different deliverables. If the user asks for PDF only, require a real PDF link or `%PDF` response and report `no_authorized_pdf_found` / `pdf_fetch_failed` when none exists. Do not save CAJ, HTML, or a login page as if it were a PDF.

## Download Intake and First-Run Configuration

For every download request, first establish the paper list and ask:

```text
是否同时下载这些文献的 Supporting Information（SI，补充材料）？
```

Do metadata lookup before this question only when needed to identify the requested papers. Do not download files until the answer is known. Configure a library only when the selected route is CNKI or Web Access. Configure a publisher API only when the selected English article belongs to Elsevier, Springer Nature, or IEEE; an OA determination is not required before trying a configured provider API.

### Paid Library Resource Configuration

Ask for the library resource URL the user actually uses:

```text
请发你平时进入图书馆电子资源/数据库的平台链接。
可以是资源门户、数据库列表、Web of Science 入口、某个数据库详情页，
或跳转到统一身份认证的登录链接。
```

Then infer the authorization route from the URL before saving config:

```bash
python3 scripts/configure_school.py infer "https://example.edu/library/resources"
python3 scripts/configure_school.py url "https://example.edu/library/resources"
python3 scripts/configure_school.py show
python3 scripts/configure_school.py health --force
```

The distributed skill contains no school presets. If the user cannot provide a resource URL, ask them to locate their institution's library/database entry instead of guessing a school-specific domain.

The default config path is:

```text
~/.config/lit-dl/school.json
```

For tests or isolated profiles, set:

```bash
LIT_DL_CONFIG_DIR=/path/to/configdir
```

The downloader reads this config automatically. If `discovery.web_of_science_url` is present, `scripts/batch_download.mjs` uses it as the Web of Science entry; otherwise it falls back to `https://www.webofscience.com/wos/woscc/basic-search`.

For Chinese literature, the downloader also reads `discovery.cnki_url` when present. If absent, `scripts/batch_download.mjs --title "<中文题名>"` falls back to `https://kns.cnki.net/kns8s/defaultresult/index`.

### API-First and Open-Access Fallback

For an English article, identify its publisher before deciding when to resolve article-level OA:

1. Collect a DOI, PMID, exact title, article URL, or a definite paper list, then normalize its metadata and publisher.
2. If it belongs to Elsevier, Springer Nature, or IEEE and usable provider credentials are configured, try that publisher API first. On success, record `accessMode: publisher_api` and `oa_status: not_checked_api_first`; do not run OA resolution only to label the article.
3. If the publisher API fails, automatically search legitimate OA sources such as PMC, Unpaywall, publisher OA pages, arXiv, and other lawful repositories or clearly open PDF URLs. Preserve the failed API attempt in the manifest.
4. For all other English publishers, search those legitimate OA sources before Web Access.
5. For an exact title or an explicit OA-only request, prefer:

   ```bash
   node scripts/batch_download.mjs --title "<exact title>" --open-access --no-si --out "<project>"
   ```

   Use `--pdf-url` when the user supplies a known legitimate OA PDF URL.
6. Verify the downloaded file and record the source. Mark a successful PDF as `open_access_downloaded`.
7. If no lawful OA full text is found, mark `oa_not_found`. For a supported publisher whose API already failed, request confirmation before Web Access. For another publisher, continue to Web Access. If `--route open_access` was explicitly requested, stop after the OA result.

### Publisher API Credentials

Configure credentials lazily, only when the route first needs them:

```bash
python3 scripts/configure_credentials.py set elsevier
python3 scripts/configure_credentials.py set springer_nature
python3 scripts/configure_credentials.py set ieee --fulltext-endpoint 'https://issued-endpoint.example/articles/{doi}'
python3 scripts/configure_credentials.py set elsevier --stdin
python3 scripts/configure_credentials.py show
python3 scripts/configure_credentials.py validate <provider>
python3 scripts/configure_credentials.py delete <provider>
python3 scripts/configure_credentials.py contact-email researcher@example.org
```

Give the user the official registration link: Elsevier `https://dev.elsevier.com/`, Springer Nature `https://dev.springernature.com/docs/quick-start/api-access/`, or IEEE `https://developer.ieee.org/member/register`.

Do not proactively ask the user to paste an API key into chat. If the user voluntarily sends a publisher API key, treat that as authorization to save that exact key: do not reject it, ask them to regenerate it, or repeat it back. Pass it to `configure_credentials.py set <provider> --stdin`, keep it out of command-line arguments, logs, replies, and manifests, then report only the masked confirmation and validation status. The local hidden prompt remains the preferred path when the key has not already been provided. IEEE Metadata API access is not paid full-text access; require the issued Full-Text Access endpoint/template before treating IEEE as downloadable through the API. Secrets are stored in `~/.config/lit-dl/credentials.json` with mode `0600`.

## Resource URL Triage

Classify the user-provided URL before choosing an access path:

From ypan50918-ops/Yuan1z0825-nature-skills, Apache-2.0.

More from ypan50918-ops/Yuan1z0825-nature-skills