kb-ingest-bulk

Files (1)

• SKILL.md

SKILL.md (first lines)

# Bulk Ingestion (map-reduce)

Load a large source set into the knowledge base in parallel. Four phases:
**map** (parallel extract) → **route** (Python) → **reduce** (parallel synthesis,
one writer per file) → **finalize** (one rebuild + one log entry).

## Arguments

| Argument | Description |
|---|---|
| `<glob\|dir\|file-list>` | Sources to ingest |
| `--library <path>` | Target library, bypassing CLAUDE.md resolution (default: CLAUDE.md-resolved). For isolated testing / multi-library — see #209 |
| `--parallel <N>` | Concurrency for map + reduce rounds (default 16, max 64) |
| `--extractor-model <id>` | Map model (default `claude-haiku-4-5`) |
| `--size-threshold <tokens>` | Per-file reduce size guard (default 200000) |
| `--retry-failed` | Re-queue failed sources/targets from a prior run |
| `--clean` | Remove `library/.extracts/` after a successful run |

## Preflight

1. Resolve `library_path`: if `--library` given, use it; else read CLAUDE.md
   `## Knowledge Base` section. Derive `shelf_index_path = <library>/_shelf-index.md`,
   `log_path = <library>/log.md`, `extracts_dir = <library>/.extracts`.
2. Verify `knowledge-extractor` and `agent-knowledge-updater` agents are available.
3. Clamp `--parallel` to [1, 64].

## Module bootstrap

Use this `CLAUDE_PLUGIN_ROOT` importlib loader block to import the
`sdlc_knowledge_base_scripts.kb_ingest_bulk` helpers. It is self-contained — no
other file needs to be opened.

```bash
python3 -c "
import sys, os, importlib.util, json
PLUGIN_ROOT = os.environ.get('CLAUDE_PLUGIN_ROOT', '')
SCRIPTS = os.path.join(PLUGIN_ROOT, 'scripts')
INIT = os.path.join(SCRIPTS, '__init__.py')
if os.path.isfile(INIT) and 'sdlc_knowledge_base_scripts' not in sys.modules:
    spec = importlib.util.spec_from_file_location(
        'sdlc_knowledge_base_scripts', INIT, submodule_search_locations=[SCRIPTS])
    if spec and spec.loader:
        mod = importlib.util.module_from_spec(spec)
        sys.modules['sdlc_knowledge_base_scripts'] = mod
        spec.loader.exec_module(mod)
from sdlc_knowledge_base_scripts.kb_ingest_bulk import (
    discover_sources, build_bulk_manifest, load_manifest, save_manifest,
    retry_failed, persist_extract, slug_for_source, mark_source_extracted,
    mark_source_failed, route_extracts, format_extract_prompt,
    format_reduce_prompt, ReduceDispatchRequest, mark_target_reduced,
    mark_target_failed, summarize_run, write_log_entry, ExtractDispatchRequest
)
"
```

## Phase 1 — Map (parallel ≤N)

1. `discover_sources(<spec>)` → source list; `build_bulk_manifest(...)` (merge prior
   manifest via `load_manifest` if present; `retry_failed` if `--retry-failed`);
   `save_manifest`.
2. For each pending source whose extract file does NOT already exist (resume skip):
   dispatch `knowledge-extractor` with `format_extract_prompt(...)`, up to N
   concurrently (parallel Agent-tool calls). Capture each agent's JSON reply.
3. After each agent: `persist_extract(extracts_dir, slug, json)` then
   `mark_source_extracted`; on error/invalid-JSON `mark_source_failed`.
   `save_manifest` after each round.

## Phase 2 — Route (Python, no agents)

4. Load all extract JSON files. Compute `existing_files` = set of `*.md` in the
   library (excluding `_shelf-index.md`, `log.md`, `_index.md`, and anything under `raw/`).
5. `route = route_extracts(extracts, existing_files, size_threshold)`. Record each
   `route.targets` entry into `manifest["targets"]` as `pending` with `source_count`
   and `is_new`. Report `route.oversized` (skipped). `save_manifest`.

## Phase 3 — Reduce (parallel ≤N, one writer per file)

6. For each pending target: dispatch ONE `agent-knowledge-updater` with
   `format_reduce_prompt(ReduceDispatchRequest(...))`, up to N concurrently. Exactly
   one agent per file — never two agents on the same file.
7. After each: `mark_target_reduced` (success) or `mark_target_failed` (error).
   `save_manifest` after each round.

## Phase 4 — Finalize (once)

8. Rebuild the shelf-index once via `build_shelf_index.main([library_path])`.
   Directories listed in `<library>/.kb-index-ignore` (plus `raw/`) are excluded.
   If it exits with code 2, the bulk-add safety rail refused the rebuild (>= 50
   entries added and index >= 2x): neither `_shelf-index.md` nor `log.md` was
   written. Show the user the per-directory breakdown (e.g. `iso20022/: 412 added`)
   and **re-run with `--force` only after the user confirms** — never
   automatically. Skip step 9 until the rebuild has actually succeeded.
9. `write_log_entry(log_path, "## [<date>] ingest-bulk\n" + summarize_run(manifest, route.oversized))`.
10. Print the summary table. If `--clean`, remove `extracts_dir`.

## Exit / partial failure

A failed source drops out of routing; a failed target leaves its extracts on disk
for `--retry-failed`. Report all failures + oversized topics. Non-zero exit only if
nothing succeeded.

From SteveGJones/ai-first-sdlc-practices, MIT.

More from SteveGJones/ai-first-sdlc-practices