batch-jobs

Files (1)

• SKILL.md

SKILL.md (first lines)

# Batch Jobs

Process large workloads reliably and asynchronously. Batch is where Zenrows'
high-scale anti-bot advantage becomes obvious: thousands to 100,000 URLs per
job, at Fetch's prices.

> Zenrows **Batch** runs on a separate host (`async.api.zenrows.com/v1`) and
> costs the same credits as Fetch (1 basic, 5 js_render, 10 premium_proxy, 25
> both; Adaptive Stealth Mode 1 to 25). Failed tasks are not billed, except
> target 404/410 answers, as on Fetch. The cloud subcommands work with any
> active API key; a 403 maps to `BATCH_ACCESS_DENIED`. The toolkit never fakes
> a cloud run.

## Spec and estimate (local, no key)

```
zenrows batch estimate jobs.jsonl [--premium-proxy] [--js-render] [--manual]
```

`jobs.jsonl` is one JSON object per line, each with a `url` (http or https,
up to 2048 bytes). Optional per-task keys:

- `external_id`: up to 128 letters, digits, `.`, `_` or `-`; echoed on each
  result row (it does not have to be unique).
- `metadata`: up to 20 string-to-string entries; keys 1 to 64 of the same
  characters, values up to 1024 bytes (UTF-8), 4 KiB in total.
- `method` and `body`: `"method": "POST"` sends `body` (up to 16 KiB): a JSON
  string goes as a form body, any other JSON value as JSON (`"body": null` means
  no body). A POST task cannot
  use `js_render`, `js_instructions` or `json_response`.
- scrape params, flat (`js_render`, `premium_proxy`, `proxy_country`, `mode`,
  `autoparse`, `extract`, ...) or nested under `zenrows_params`. Booleans are
  `true`/`false` (or `"true"`/`"false"`).

`estimate` checks every line against those limits and prices the run the way
`create` would submit it, job-level flags included, as a range: Adaptive
Stealth Mode tasks count 1 at the low end and 25 at the upper bound. It exits 1
on any invalid line.

## Extract in Batch

`"extract": "auto"` on a line runs that URL through Extract, returning
structured data instead of raw HTML.

```
{"url": "https://example.com/products", "extract": "auto", "external_id": "p1"}
```

An Extract task costs the same as a regular one, so `batch estimate` prices it
correctly. The result carries two keys: `html` (the
raw page) and `parsed` (the structured data). Validate on a single URL with
`zenrows extract <url>` before running the full batch.

## Create, wait, read

```
zenrows batch create jobs.jsonl --follow --json
zenrows batch create jobs.jsonl --manual --premium-proxy --proxy-country us --output markdown
zenrows batch create jobs.jsonl --name nightly --external-id run-42 --metadata team=growth
zenrows batch results <id> --out results.jsonl   # every row, as JSONL
zenrows batch results <id> --download out/       # each result body + _manifest.jsonl
zenrows batch content <id> <task_id>             # one task's body
```

- Every GET task runs in Adaptive Stealth Mode (`mode=auto`) by default, like
  `fetch` and `extract`: Zenrows enables JS rendering and premium proxies only
  on the pages that need them, and bills only the configuration that
  succeeds. A task that turns on `js_render` or `premium_proxy`, or sets
  `mode`, itself keeps its own settings; `--js-render` / `--premium-proxy` force that
  configuration on every task; `--manual` turns Adaptive Stealth Mode off.
  Outside Adaptive Stealth Mode, `--proxy-country` needs `--premium-proxy`.
- Job-level flags apply to every task; per-task keys override them.
- `--follow` (alias `--wait`) waits until the run ends, the run is paused, or
  `--timeout` runs out (default 10 minutes; `30s`, `5m`, `1h`; a bare number
  is milliseconds). A wait that runs out exits 0 and prints the run with a
  `note` and `next: zenrows batch wait <id>`; the job keeps running. Ctrl-C
  stops the wait, not the job, and exits 130.
- A submit of 10,000 tasks or more answers 202 with `ingest_status: pending`
  while the API still receives the tasks; results stay partial until `done`.
- `create` and `rerun` send an `Idempotency-Key` (generated, or
  `--idempotency-key`) and retry network errors and 502/503/504 under it.
  When the outcome stays unknown, the error carries `idempotency_key` and the
  exact command that repeats the request safely (the API keeps a key 24 hours
  and answers a repeat with the original job). Never repeat with a new key.
- Results are kept 14 days by default; after that a row reads `expired`, has no
  `result_url`, and `content` answers `BATCH_RESULT_EXPIRED`. `--download`
  skips expired rows and counts them. A row's `result_url` is valid 2 hours:
  list the results again for a fresh one.
- `results --status successful|failed|processing|all`, `--run <run_id>` for an
  earlier run. On a run still in progress the output has `partial: true`.

With `--json`, `create`, `status`, `wait`, `cancel`, `pause`, `resume` and
`rerun` print `{ok, job_id, run_id, status, stats, job}` (plus `jobId`, kept
for older scripts), with `paused`, `ingest_status`, `accepted_tasks`,
`rerun_of`, `retried_tasks`, `inherited_tasks`, `failure_reason` and
`failure_detail` when the API sends them, and `note` when a wait ends early.

## While it runs, and after

```
zenrows batch list [--status open|closed] [--type regular|scheduled]
zenrows batch status <id>
zenrows batch wait <id> --timeout 30m
zenrows batch pause <id>              # no new task is picked up
zenrows batch resume <id>
zenrows batch cancel <id>             # stop the run; an ended run answers as it ended
zenrows batch retry-failed <id>       # rerun the failed tasks
zenrows batch retry-failed <id> --include-pending   # also the unfinished ones
zenrows batch rerun <id>              # failed + pending tasks; successful ones carry over
zenrows batch rerun <id> --status failed,pending,processing
zenrows batch rerun <id> --all        # every task (not with --status)
zenrows batch delete <id> --yes       # job, runs and stored results; frees its slot
```

A run ends `completed`, `stopped`, `failed` or `deleted`. A paused run is not
ended: `status` shows `paused: true`, and `wait` returns at once with a note
to `zenrows batch resume <id>`.

`status`, `wait`, `create --follow` and `rerun --follow` exit 1 (`ok: false`)

From ZenRows/cli, MIT.

More from ZenRows/cli