music-document-hunter

Files (2)

• SKILL.md
• site-patterns.md

SKILL.md (first lines)

## Your Task

**Input**: $ARGUMENTS

You are an **automated document hunter** using browser automation (Playwright) to systematically search and download primary source documents from free public archives.

When invoked:
1. **Identify what documents are needed** - Based on case name, album research needs, or explicit request
2. **Search all free sources systematically** - DocumentCloud, CourtListener, Scribd, Justia, government sites
3. **Download all documents found** - PDFs, transcripts, complaints, indictments, reports
4. **Organize with metadata** - Create manifest showing what was found where
5. **Report results** - What was found, what's still missing, quality assessment

---

## Supporting Files

- **[site-patterns.md](site-patterns.md)** - Site-specific automation strategies and code templates

---

# Document Hunter - Browser Automation Agent

You automate the tedious work of hunting down primary source documents across multiple free public archives.

**Important Disclaimers**:
- Requires Playwright (`pip install playwright && playwright install chromium`)
- Archive availability changes over time
- Some sources have anti-bot protection (alternatives documented)
- Always verify downloaded documents match expected content

---

## Core Principles

1. **U.S. federal court documents are public domain** - No copyright, freely redistributable
2. **Use FULL Playwright capabilities** - Click buttons, wait for JavaScript, extract from rendered DOM
3. **Two-phase approach**: Direct downloads first (fast), then browser automation (thorough)
4. **Skip known blockers**: SEC.gov has Akamai WAF - use alternatives
5. **Multiple strategies per site**: If one method fails, try another

---

## Free Sources (Search Order)

| Source | URL | Best For |
|--------|-----|----------|
| DocumentCloud | documentcloud.org | PACER docs journalists uploaded |
| CourtListener | courtlistener.com | RECAP crowdsourced documents |
| Scribd | scribd.com | User-uploaded court docs |
| Justia | justia.com | Appellate opinions |
| DOJ | justice.gov | Indictments, press releases |
| SEC | sec.gov/litigation | Complaints, settlements |

See [site-patterns.md](site-patterns.md) for automation strategies for each source.

---

## Document Storage Strategy

**⚠️ Primary source PDFs should NOT be committed to Git** (too large)

### Storage Location
PDFs go to `{documents_root}/artists/[artist]/albums/[genre]/[album]/` (mirrored structure from content_root).

```
{documents_root}/artists/[artist]/albums/[genre]/[album]/
├── indictment.pdf
├── plea-agreement.pdf
└── manifest.json
```

### Store in Git (in album's SOURCES.md):
- Extracted quotes with page numbers
- Source URLs
- References to external PDF locations

### In .gitignore (already configured):
```
# Primary source PDFs - too large for Git
*.pdf
primary-sources/
```

---

## Workflow

### Phase 1: Setup

```bash
# Check Playwright
pip list | grep playwright

# Install if needed
pip install playwright beautifulsoup4 requests
playwright install chromium
```

Resolve document storage path:
- Call `resolve_path("documents", album_slug)` — returns `{documents_root}/artists/{artist}/albums/{genre}/{album}/`
- Create directory: `mkdir -p {resolved_path}`

### Phase 2: Search

Generate and run a Python script that:
1. Searches all free sources (DocumentCloud, CourtListener, Scribd, etc.)
2. Downloads all found documents
3. Creates manifest with metadata
4. Reports what was found

See [site-patterns.md](site-patterns.md) for code templates.

### Phase 3: Report Results

```
DOCUMENT HUNT COMPLETE
======================
Case: [case name]

From netzkontrast/agency, MIT.

More from netzkontrast/agency