crawl

Files (1)

• SKILL.md

SKILL.md (first lines)

# Crawl (new)

Give Crawl one start URL; it follows the links on that page (and on the
pages behind it, up to the depth you set) and returns the URLs it keeps. Use it
when you know where the items live (a category page, a blog index, a docs
section) but not the item URLs themselves. Once you have the URLs, read pages
with `zenrows fetch` / `zenrows extract`, or ask Crawl for each page itself
with `--output-format html` or `--output-format markdown`.

Crawl stays on the start URL's registrable domain; subdomains count. An account without Crawl access gets
403 `REQS008` → `CRAWL_NOT_ENABLED` ("Crawl is not enabled for this account");
do not retry it.

## Create, wait, read

```
zenrows crawl create https://example.com/products/ --depth 1 \
  --max-items 20 --max-pages 25 --include-patterns /product/ --follow --json
zenrows crawl results <id> --json            # every kept URL
zenrows crawl results <id> --out urls.jsonl  # same, as JSONL
```

- `--depth N` (1 to 100000) is optional: link hops from the start URL. `1` =
  the start page's links; `2` = also the links on each of those pages. Leave
  it out for no depth limit: the crawl follows links until `--max-items` or
  `--max-pages` stops it, or nothing is left to open. A crawl created without
  a depth has no `depth` field in its output.
- `--max-items` (kept URLs) and `--max-pages` (fetched pages) cap the crawl;
  both default to 100. `--max-pages` bounds the cost: each page is one fetch.
  The local policy `max_pages_per_run` caps `--max-pages` (default 10000);
  above it the CLI refuses with `POLICY_LIMIT_EXCEEDED` before any request.
- `--include-patterns` / `--exclude-patterns` are substring matches on the
  normalized URL and repeat:
  `--include-patterns /product/ --exclude-patterns ?add-to-cart`.
- A crawl whose pages held nothing new to follow (no link, only links to other
  sites, or only links the patterns reject) ends `completed` with no results
  and exits 0. When patterns were set, it prints a note (under `--json`, in
  `note`) to check them against the URLs the start page links to.
- `create` returns at once. `--follow` (or `zenrows crawl wait <id>`) polls
  until the crawl ends or `--timeout` seconds run out (default 600). When the
  time runs out, the command exits 0 and prints the crawl with status
  `running` and the hint `zenrows crawl wait <id>` (under `--json`, in
  `note`). The crawl keeps running;
  it is not stopped. Ctrl-C also stops the wait, not the crawl, and exits 130;
  resume with `zenrows crawl wait <id>`.
- `results` on a running crawl returns the URLs kept so far, with
  `partial: true`. Read again once the crawl ends. `--limit` sets the page size
  of each request.

## Page content

```
zenrows crawl create <url> --depth 1 --output-format html --max-pages 20 --follow
zenrows crawl create <url> --depth 1 --output-format markdown --max-pages 20 --follow
zenrows crawl results <id> --json        # fetched results carry content_url
zenrows crawl content <id> <content_id>  # one page (or pass the content_url)
zenrows crawl download <id> --out pages.jsonl  # every URL + page, NDJSON
```

`--output-format html` stores each page as fetched; `--output-format markdown`
stores it converted to Markdown. Either uses up the page budget: each kept page is one more
fetch, and `--max-pages` counts it. Raise `--max-pages` to keep the same number
of URLs. A result's `content_status` is `pending`, `fetched` or `failed`; only
`fetched` results have content (else `CRAWL_CONTENT_NOT_FOUND`). `download`
reports the crawl's `status`, and `partial: true` while it runs.

## While it runs, and after

```
zenrows crawl get <id>                       # status, coverage, stop_reason / error, one page of results
zenrows crawl get <id> --cursor <c> --json   # the next page; keep polling with next_cursor
zenrows crawl list                           # your crawls, newest first (--cursor, --limit)
zenrows crawl stop <id>                      # stop it; kept URLs stay readable
```

A crawl ends `completed` (nothing left, or a cap hit: `stop_reason`
`max_items` / `max_pages`), `stopped` or `failed` (`error.code` and
`error.detail` say why). `stopped` with `stop_reason` `user` means you stopped
it. `stopped` with no `stop_reason` and `error.code` `insufficient_credits` or
`api_key_cap_reached` means the account ran out of credits or the API key hit a
credit cap; the pages fetched before that stay readable. `failed`, and a
`stopped` crawl with an `error`, exit non-zero: `CRAWL_INSUFFICIENT_CREDITS` or
`CRAWL_KEY_CAP_REACHED` when credits ended it (the same codes a refused
`create` gets, also when an older deployment reports it `failed`), else
`CRAWL_FAILED`. `server_code` carries the API's `error.code`:
`seed_unreachable` when the start URL could not be fetched or returned an empty
page, `no_items_found` when the crawl read the start page but found no data it
could use.
`stop` on a crawl that already ended prints the status it ended with.

With `--json`, `create`, `get`, `wait` and `stop` print
`{ok, crawl_id, status, crawl}`; read `coverage`, `stop_reason` and `error`
under `crawl`. `get` adds `results` and `next_cursor`. `download` prints
`status: null` when the API sends no status.

## Limits

When the account has reached its limit of active jobs (3 by default), shared
with its Batch jobs, a create gets 429 `too_many_crawls` →
`CRAWL_TOO_MANY_CRAWLS`; nothing was created. Retry after `retry_after`
seconds (about 30), or stop one of your crawls or Batch jobs first. Any other
429 is `CRAWL_RATE_LIMITED`: requests are being throttled; retry after
`retry_after` seconds.

## Errors

Every Crawl error carries the API's code as `server_code` (when the API sent
one), and `crawl_id` on any call about one crawl.

The CLI retries transient failures, as the Crawl SDKs do: up to 3 retries,
waiting the `Retry-After` seconds (at most 30) or 250 ms x 2^n (+/-20% jitter,
at most 10 s). Reads retry on 429, 502, 503, 504 and network errors (a timeout
counts). `create` retries only with `--idempotency-key`, and only on 502, 503,
504 and network errors, never on 429. `stop` retries on 429, 502, 503 (including
`crawl_busy`) and 504, waiting the `Retry-After` seconds (at most 30) or 1 s.
The "Retry?" column is for errors that remain after these retries.

| Code | Meaning | Retry? |
| --- | --- | --- |

From ZenRows/cli, MIT.

More from ZenRows/cli