tvb-wiki-ingestion

Files (1)

• SKILL.md

SKILL.md (first lines)

# TVB Wiki Ingestion

Skill for fetching new arXiv papers relevant to connectome‑based whole‑brain modeling, extracting software/concept/author mentions, and updating the wiki’s raw paper library and entity counts.

## When This Skill Activates

Use this skill when:
- You need to fetch the latest arXiv papers for the TVB wiki
- You want to update the raw paper library without running the full pipeline
- You are setting up a new ingestion source (bioRxiv, GitHub, etc.)
- The user asks “fetch new papers for the wiki” or “update the paper database”

## Prerequisites

- TVB wiki at `~/tvb‑wiki` (or `WIKI_PATH` environment variable)
- Python 3.11+ with `requests` and `xml.etree.ElementTree` (standard library)
- Network access to arXiv API (https://export.arxiv.org)

## Architecture

```
~/tvb‑wiki/
├── raw/papers/               # Raw paper markdown files (arxiv-*.md)
├── meta/entity_counts.json   # Counts of software/concept/author mentions
└── scripts/hourly_update.py  # Main ingestion script
```

The ingestion script (`hourly_update.py`) performs:

1. **Query arXiv** with domain‑specific search queries
2. **Filter duplicates** by arXiv ID
3. **Save new papers** as markdown in `raw/papers/`
4. **Extract entities** via keyword matching
5. **Update entity counts** in `meta/entity_counts.json`

## Step‑by‑Step Ingestion

### 1. Run the Ingestion Script

```bash
cd ~/tvb‑wiki
python3 scripts/hourly_update.py
```

**Output:** Prints number of new papers added, e.g., `Hourly update complete: 3 new papers`.

### 2. Inspect New Papers

List recently added raw papers:

```bash
ls -lt ~/tvb‑wiki/raw/papers/arxiv-*.md | head -5
```

View a raw paper:

```bash
cat ~/tvb‑wiki/raw/papers/arxiv-<ID>.md
```

### 3. Check Entity Counts

The script updates `meta/entity_counts.json` with mention counts. View current counts:

```bash
cat ~/tvb‑wiki/meta/entity_counts.json | jq .
```

Example structure:

```json
{
  "software": {
    "TVB": 42,
    "NEST": 18,
    "NEURON": 12
  },
  "concepts": {
    "neural mass model": 56,
    "fMRI": 89,
    "EEG": 67
  },
  "authors": {
    "John Doe": 5,
    "Jane Smith": 3
  }
}
```

### 4. Manual Ingestion with Custom Queries

If you need to ingest papers from other arXiv categories or with different keywords, you can modify the search queries inside `hourly_update.py` (function `fetch_arxiv_since`). Default queries:

```python
queries = [
    'cat:q-bio.NC+AND+all:connectome',
    'all:neural+mass+model',
    'all:dynamic+causal+modeling',
    'all:The+Virtual+Brain',
    'all:TVB',
    'all:whole+brain+model',
]
```

Add new queries, adjust `max_results`, or change the `since_date` threshold.

### 5. Add a New Ingestion Source (e.g., bioRxiv)

To extend beyond arXiv:

1. Create a new function `fetch_biorxiv_since()` that uses the bioRxiv RSS feed or API.
2. Merge results with arXiv papers.
3. Assign a `source: bioRxiv` field and save with filename prefix `biorxiv-`.
4. Update entity extraction accordingly.

See the `automated‑research‑wiki` skill for a multi‑source pattern.

## Entity Extraction Details

From maedoc/tvb-wiki, MIT.

More from maedoc/tvb-wiki