roboflow-batch-processing

Files (1)

• SKILL.md

SKILL.md (first lines)

> **For agents:** This skill is authored in [`roboflow/computer-vision-skills`](https://github.com/roboflow/computer-vision-skills). If your client already has it loaded as a skill (for example from a Roboflow plugin), use that copy. The `skill://roboflow/<name>/SKILL.md` MCP resources serve the same skills for clients that don't have them loaded.

# Batch Processing

Run a Roboflow Workflow over a very large set of images or videos on
Roboflow's autoscaling compute. Every job processes a temporary Data Staging
batch; the two input paths differ only in **who fills that batch**:

- **Asset Library job** — the files are already in Roboflow. You hand the
  platform a selection and it selects and stages them for you.
- **Staged job** — the files are outside Roboflow (local disk, cloud bucket,
  references file), so you stage them yourself with the inference-cli: only
  your machine and credentials can reach them. Nothing is imported into the
  workspace, and staged data expires after ~7 days.

## Which path? Three questions

1. **Are the files already in Roboflow (Asset Library)?** Call
   `batch_processing_asset_library_job_create` with a stable idempotency key
   and exactly one selection: `image_ids`, `query`, or `all_images=true`. The
   platform performs access checks, selects the files, stages them, verifies
   Workflow compatibility, bills, and registers the durable job. Poll the
   returned `taskId` with `batch_processing_asset_library_task_get` until the
   task is terminal; then monitor its `jobId` with `batch_processing_job_get`.
2. **Files outside Roboflow, and you only want the outputs?** Stage them
   yourself and drive the run with `batch_processing_guide` +
   `batch_processing_run` as described below (the classic ETL shape: nothing
   lands in the workspace). Staging and result export must run on the machine
   that can access the files.
3. **Files outside Roboflow that you want INSIDE it** (labeling, curation,
   training)? That is an import, not a batch processing job: mirror the
   bucket with a datasource (`connect_cloud_storage`, see the `cloud-storage`
   skill). Once mirrored, the files are Asset Library images, so if you also
   want bulk predictions, run an Asset Library job over them (the classic ELT
   shape: load first, then transform).

"Datasource" and "bucket mirror" are one thing: a datasource is the
user-facing name for a bucket-mirror config, the importer that fills the
Asset Library. It never runs Workflows itself.

In pipeline terms: a staged job is the T of an ETL flow (no load ever
happens), and an Asset Library job is the T of an ELT flow — the load
already happened, via uploads or a datasource mirror.

## Prerequisites

- The workspace needs the batch-processing feature. Gated calls fail with a
  402 "Batch processing is not enabled for this workspace. Upgrade your plan
  or contact sales at https://roboflow.com/sales."
- Jobs consume credits; the workspace must have a positive balance.
- The Workflow must already exist in the workspace (`workflows_list`,
  `workflows_create`). Jobs reference it by `workflow_id`; there is no
  inline-spec option.
- API key: the inference-cli reads `ROBOFLOW_API_KEY` from the environment,
  and every command also accepts `--api-key=<key>`. A key stored by
  `roboflow login` (`~/.config/roboflow/config.json`) is NOT picked up by
  the inference-cli: export it or pass `--api-key` explicitly. Mint one with
  the `api_keys_create` MCP tool. Never have the user paste a private key
  into chat.

## Where each step runs, and why

Two steps are inference-cli only, because the MCP server can neither read nor
write the user's disk:

- **Staging** runs on the machine that can reach the files (local disk) or
  with the user's cloud credentials (bucket sources).
- **Exporting results** (`export-batch`) downloads into a local directory.

Everything in between needs only an API key, so it works either way: the MCP
tools (`batch_processing_run`, `batch_processing_staged_job_start`,
`batch_processing_job_get`, `batch_processing_jobs_list`,
`batch_processing_job_logs`, `batch_processing_job_abort`,
`batch_processing_job_restart`, plus the
`batch_processing_asset_library_*` and `batch_processing_staging_*`
families) or the equivalent CLI commands. Prefer the MCP tools when the host has no shell or the user has
no local `inference-cli`; prefer the CLI when the user is already in a
terminal. The full command set for both content types is in sections 1-5.

The `batch_processing_guide` MCP tool routes a request: it settles the batch
id, picks the staging source, and returns the exact ordered commands. It
never touches files or the network, and it does not repeat this document.

## The master tool for staged batches: `batch_processing_run`

Prefer `batch_processing_run(batch_id, workflow_id, content_type, ...)` to
drive the flow over a staging batch you name (for Asset Library selections,
use `batch_processing_asset_library_job_create` instead). Each call reads
current state, advances what it can, and returns immediately with a status:

- `staging_required` — no batch yet; the response contains the CLI commands
  for the whole run plus cloud-credential guidance. Run them, then call again.
- `ingest_in_progress` — files still registering.
- `ingest_failed` — ingest failed or returned an unknown shard state; no paid
  job was started.
- `running` — the job was started or is still working; includes stage progress.
- `completed` — includes the export batch id and the first result files with
  signed download URLs.
- `failed` — includes logs and a restart hint.

**The server never waits on your behalf.** Non-terminal responses carry
`retryAfterSeconds`; sleep that long on your side, then call again with the
returned `job_id` and the same `batch_id` to resume. Omit optional creation
settings on a read-only resume: the paid job's stored definition is
authoritative. The server holds no state between calls.

The job is started under an id derived from (workspace, batch, workflow), so
retrying after a lost response re-registers the same job instead of paying for
a second run. (The platform checks credits before that idempotency comparison,
so a retry can still see a 429 first.) A 409 means that id already exists with
different batch/Workflow identity or conflicts with an optional setting you
explicitly supplied. Inspect the job, omit optional settings to monitor it as
stored, or pass a new explicit `job_id` for a separate run.

For the advanced knobs (`max_runtime_seconds`, `max_parallel_tasks`,
`max_image_failure_rate`, `image_outputs_to_save`) use
`batch_processing_staged_job_start` directly.

## Webhooks

From roboflow/computer-vision-skills, Apache-2.0.

More from roboflow/computer-vision-skills