xberg-batch-extraction

Files (10)

• SKILL.md
• index.md
• advanced-features.md · references/
• cli-reference.md · references/
• configuration.md · references/
• nodejs-api.md · references/
• other-bindings.md · references/
• python-api.md · references/
• rust-api.md · references/
• supported-formats.md · references/

SKILL.md (first lines)

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:324433f2b10c7c9cdc1c06b98c1e427c8cb0ca9ccf034d336e28d49706add520
Source-Hash: blake3:934f7e3903a85762c6a94f533fc47d534e94d3274b6a2cd147fa8bda13a6b03a
Schema-Version: v1
-->

# Batch extraction

Use this when processing a directory or glob of documents in one pass.
`xberg batch` shares one extraction config across every file, runs
extractions concurrently, and returns a JSON envelope — failures on
individual files do not abort the run.

## Basic usage

```bash
# Glob expands to many paths; results come back in a JSON envelope (default)
xberg batch *.pdf

# Mixed formats, markdown content for LLM ingestion
xberg batch docs/*.docx --content-format markdown

# Recurse with the shell, then extract
xberg batch $(find ./corpus -name '*.pdf')
```

`batch` defaults to `--format json` (vs `--format text` for single
`extract`). The envelope contains `results`, `total_ms`, `per_file_ms` and, when failures
occur, `errors`. Each `results` entry is a full extracted document. One input may produce
multiple documents, so do not pair results with inputs by position. `per_file_ms` is
aligned with inputs; inspect `errors` for per-input failures.

```bash
xberg batch reports/*.pdf \
  | jq '.results[] | {chars: (.content | length), mime: .mime_type}'
```

## Parallelism

`--max-concurrent` caps how many files extract at once. When omitted, the scheduler derives document concurrency from the total thread budget. Lower it on memory-constrained hosts or when OCR/ML
models are active, since each in-flight extraction holds its own buffers.
Layout-heavy batches are further limited (1 concurrent extraction for
all-PDF-layout batches, 2 for mixed layout):

```bash
# Cap at 4 concurrent extractions
xberg batch scans/*.pdf --ocr true --max-concurrent 4
```

`--max-threads` additionally caps *total* internal threads (Rayon, ONNX
intra-op, the batch semaphore) for tightly constrained environments:

```bash
xberg batch *.pdf --max-concurrent 2 --max-threads 4
```

## Per-file config overrides

A single shared config does not always fit. `--file-configs` points at a
JSON file mapping each path to its own override object, merged on top of
the shared config for that file only:

```json
{
  "scan.pdf": { "force_ocr": true },
  "report.pdf": { "output_format": "markdown" },
  "data.xlsx": { "output_format": "json" }
}
```

```bash
xberg batch scan.pdf report.pdf data.xlsx --file-configs overrides.json
```

Keys are file paths (matching the paths passed on the command line);
values are per-file extraction config objects in snake_case, the same
shape as a config file.

## Output layout

For text/toon output with image extraction, `--output-dir` controls where
referenced image files (e.g. `image_0.png`) are written; the directory
must already exist. JSON output embeds image bytes inline and ignores
`--output-dir`.

```bash
mkdir -p out/images
xberg batch slides/*.pptx --extract-images true --output-dir out/images --format text
```

## Error recovery

Batch extraction is fault-tolerant per file: one unreadable or corrupt
document does not stop the rest. Inspect results for partial content and
surfaced errors rather than relying on the process exit code alone. Pair
with `--max-concurrent` to avoid exhausting memory when a few large files
sit in a big batch.

## Shared config

Every `extract` flag also applies to `batch` (OCR, chunking, layout,
content format, etc.) and is shared across all files unless a
`--file-configs` entry overrides it:

```bash
xberg batch invoices/*.pdf \
  --layout --layout-table-model slanet_wireless \
  --content-format markdown --max-concurrent 8
```

A config file works too and auto-discovers from the cwd upward:

```toml
output_format = "markdown"

[ocr]
backend = "tesseract"
language = "eng"

From xberg-io/xberg, MIT.

More from xberg-io/xberg