corpus-search

Files (4)

• SKILL.md
• requirements.txt
• ingest_corpus.py · scripts/
• search_corpus.py · scripts/

SKILL.md (first lines)

# Corpus Search — Critical Minerals PDF Corpus

Semantic search over a local collection of critical minerals PDFs (USGS, UN Comtrade, World Bank, SEC, WTO, Mindat, MinCan reports). Documents are chunked, embedded with `llama-text-embed-v2`, and stored in a Pinecone index for fast similarity search.

## Usage

### Search the corpus:
```bash
python3 {baseDir}/scripts/search_corpus.py --query "rare earth separation techniques"
```

### Filter by commodity:
```bash
python3 {baseDir}/scripts/search_corpus.py --query "supply chain risks" --commodity lithium
```

### Filter by source organization:
```bash
python3 {baseDir}/scripts/search_corpus.py --query "trade flows" --source Comtrade
```

### Reranked results (higher quality):
```bash
python3 {baseDir}/scripts/search_corpus.py --query "cobalt extraction" --rerank --top-k 20
```

### JSON output:
```bash
python3 {baseDir}/scripts/search_corpus.py --query "graphite processing" --format json
```

## Parameters

| Parameter | Description | Default |
|-----------|-------------|---------|
| `--query` | Semantic search query | Required |
| `--commodity` | Filter by commodity keyword (e.g., lithium, cobalt, rare earth) | - |
| `--source` | Filter by source organization (e.g., USGS, Comtrade, SEC) | - |
| `--top-k` | Number of results to retrieve | 10 |
| `--rerank` | Enable reranking with pinecone-rerank-v0 | false |
| `--format` | Output format: summary, detailed, json | summary |
| `--index-name` | Pinecone index name | scienceclaw-minerals-corpus |

## Ingestion

Before searching, ingest PDFs into the Pinecone index:

```bash
# Dry run — list PDFs that would be ingested:
python3 {baseDir}/scripts/ingest_corpus.py --corpus-dir ~/critical-minerals-data/ --dry-run

# Ingest all PDFs:
python3 {baseDir}/scripts/ingest_corpus.py --corpus-dir ~/critical-minerals-data/

# Force re-ingest (ignore manifest):
python3 {baseDir}/scripts/ingest_corpus.py --corpus-dir ~/critical-minerals-data/ --force-reingest
```

### Ingestion Parameters

| Parameter | Description | Default |
|-----------|-------------|---------|
| `--corpus-dir` | Directory containing PDFs | ~/critical-minerals-data/ |
| `--index-name` | Pinecone index name | scienceclaw-minerals-corpus |
| `--force-reingest` | Re-ingest all files, ignoring manifest | false |
| `--dry-run` | List files without ingesting | false |

## Notes

- Requires `PINECONE_API_KEY` environment variable
- PDFs are chunked at ~600 tokens with 100-token overlap
- Source organization is auto-detected from directory name (e.g., `usgs/`, `sec/`)
- Commodity is auto-detected via keyword scanning
- Incremental updates: only new or modified files are re-ingested (SHA-256 manifest)
- Reranking uses `pinecone-rerank-v0` for higher quality results at the cost of latency

From lamm-mit/scienceclaw, Apache-2.0.

More from lamm-mit/scienceclaw