domain-corpus-bootstrap

Files (3)

• SKILL.md
• sources-by-field.md
• templates.md

SKILL.md (first lines)

# Domain corpus bootstrap — "become a super expert in X"

Standing procedure. When a new project or domain X appears, the corpus comes
before the analysis. An agent reasoning about X without a corpus is reasoning
from its weights, and neither it nor the user can tell where a claim came from.

**Offer this at the start of any new project or named domain.** Do not wait to
be asked. State the cost (typically a few hours wall-clock, mostly unattended)
and the deliverables below, then get a go/no-go.

## Reference implementation

The reference implementation is `FLARE/literature/` if that repository is
present (stdlib scripts under `literature/scripts/`). Read
`literature/README.md` there before writing any new tooling and generalize
those scripts. If FLARE is not on disk, follow the generic steps below —
do not invent a second pipeline.

| step | script (FLARE reference) | generic to any X? |
|---|---|---|
| seed from cited work | `build_catalog.py` | yes — walks `.tex`/`.md`, extracts bibitems, DOIs, arXiv IDs |
| seed from search history | `extract_scite_lookups.py`, `merge_scite_into_catalog.py` | yes — replays scite MCP calls out of Cursor transcripts |
| OA download | `download_literature.py` | yes |
| second-chance acquisition | `retry_failed_downloads.py` | yes |
| library request list | `build_mit_request_list.py` | yes, retarget the institution |
| register hand-fetched PDFs | `ingest_manual_pdfs.py` | yes |
| canonical filenames | `normalize_pdf_names.py` | yes |
| RLM export | `export_for_rlm.py` | yes |

## Deliverables

A bootstrap is done when all seven exist, and not before.

1. `TOPIC_MAP.md` — what X actually is, decomposed (step 1)
2. `queries.json` — the checked-in query manifest (step 2)
3. `catalog.json` — one entry per unique paper, with priority tiers
4. `pdfs/` — every legally obtainable PDF, canonically named
5. `<INSTITUTION>_REQUESTS.md` — the short, defensible list a librarian can act on
6. `ANALYSIS.md` — themes, canon, lineage, groups, settled vs contested, gaps
7. `rlm_export/` — the AGIneer handoff surface

---

## Step 1 — Scope X into a topic map

A project name is not a search. Turn it into five buckets, written to
`TOPIC_MAP.md`, before touching a database.

| bucket | question it answers | starshade example |
|---|---|---|
| core topic | the thing itself | starshade / external occulter formation-flying starlight suppression |
| sub-topics | its decomposable parts | petal edge scatter, Fresnel/apodization design, deployment mechanics, formation sensing and control, solar glint, micrometeoroid tolerance |
| adjacent physics/engineering | what a practitioner must also know | scalar diffraction and Babinet, boom/membrane structural dynamics, station-keeping ΔV budgets, stray light, contamination |
| competing approaches | what X is argued against | internal coronagraphs, interferometric nulling, apodized pupils, direct imaging with extreme AO |
| foundational theory | what the field rests on | Fresnel diffraction, Cash's offset-hyperbolic apodization, Vanderbei optimization, Marchal |

Two rules.

**Mine in-repo prior art for seeds.** If the project folder already has a
concept note, prior-art assessment, feasibility memo, or a bibliography, the
names, DOIs and programme labels in it are the highest-precision seeds
available — they were chosen by a human who understood the problem. In the
starshade case that is `starshade_concept/PRIOR_ART.md`. Harvest every author,
mission acronym, funding programme (NIAC, APRA, SAT), and DOI out of it first.

**Name the people and programmes, not just the concepts.** Fields this size
are 20–40 people. An author sweep on the six names that recur in the prior art
is usually higher-yield than any keyword query.

---

## Step 2 — Write a checked-in query manifest

The search must be re-runnable by someone else, and auditable when a reviewer
asks "did you look for Y?". A chat transcript is neither.

`queries.json` — one record per query, committed alongside the catalog:

```json
{
  "id": "q014",
  "bucket": "sub-topic:edge-scatter",
  "query": "(starshade OR occulter) AND (\"edge scatter\" OR \"solar glint\") AND (petal OR occulting)",
  "sources": ["ads", "arxiv", "spie", "ntrs"],
  "date_first_run": "2026-08-11",
  "hits": 37,
  "kept": 11,
  "note": "SPIE hits dominate; ADS misses proceedings without DOIs"
}
```

`kept` far below `hits` is fine. `kept` at zero is a finding — record it, because
a negative result ("nobody has published on modular reassembly tolerancing") is
evidence, and it is the evidence most often lost.

See `templates.md` for the full manifest schema and the request-list template.

---

## Step 3 — Choose sources by field

Database choice dominates recall, and the right set varies enormously by field.
Do not run one generic list. **Read `sources-by-field.md`** for the routing
table per domain family (astronomy/space, materials/plasma, biomedical,
engineering/EE, chemistry, earth science).

The three field-specific traps worth knowing up front:

- **Astronomy and space**: the load-bearing literature is substantially grey —
  NASA NTRS, NIAC final reports, decadal white papers, KISS/workshop short
  courses, JPL technical reports — plus SPIE proceedings, which are often the
  only publication of an instrument result. Crossref and Semantic Scholar
  under-index all of this. NASA ADS is the primary index, not a supplement.
- **Materials and plasma**: APS (PRB/PRApplied/PRMaterials), arXiv cond-mat,
  ceramics/metallurgy society journals, plus materials databases (Materials
  Project, COD, ICSD) as a distinct non-paper source class.
- **Biomedical**: PubMed/PMC first, and preprints (bioRxiv/medRxiv) are a
  separate sweep, not a fallback.

From ricfulop/cba-agentic-engineering-bootstrap, MIT.

More from ricfulop/cba-agentic-engineering-bootstrap