Spider

Rust crawling engine for scraping and crawling at scale, runnable locally or through the Spider Cloud API.

Infrastructure by Spider

IndexedMIT★ 2,794

Install

Pick your client. Read the repository before you give any tool access to your data.

cargo add spider

Get the weekly diff

The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.

About

Spider is a concurrency-first crawling engine written in Rust. It streams pages as they arrive and renders JavaScript only on pages that need it, scaling from a single script to a distributed worker fleet. The same engine runs Spider Cloud, which handles proxies, headless browsers and stealth fetching as a managed service.

Use cases: web-scraping

Reviews

Write a review

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Fact sheet

Category: Scraping · llm ready crawl
Self-host: Yesraw.githubusercontent.com
Open source: Yesraw.githubusercontent.com
Works with: playwright, puppeteerspider.cloud
Protocols: rest, websocket, mcpspider.cloud

Facts read from the vendor's own pages by claude-haiku-5-5 on Oct 24, 2026; a fact is shown only with the page it came from.

Signals

Stars: 2,794
Forks: 318
Open issues: 3
Last commit: Oct 23, 2026 (5 days ago)
Last release: Mar 31, 2026 (7 months ago)

From GitHub (fetched Oct 24, 2026).

Details

Packages: cargo spiderpypi spider-client
License: MIT
Repository: github.com/spider-rs/spider
Website: spider.cloud
Docs: spider.cloud/docs/overview
First seen: Oct 24, 2026
Last checked: Oct 24, 2026
Source: README Website Docs github-graphql

Cite this page

Spider. Agent Mart. https://agentmart.dev/infrastructure/spider (last updated 2026-10-24)