Skip to content
Agentmart
Submit a tool

CrawlEyes MCP Server

Web scraping & search for AI agents: extract, SearXNG search, rerank, robots compliance.

Connector by waiky-github · GitHub account verified by the MCP Registry

IndexedMITLocal (stdio)★ 2
Write a review

Install

Works with Claude Code, Codex, VS Code, Cursor, Windsurf, Claude Desktop. Read the repository before you give any tool access to your data.

npx agentmarket install connector/waiky-github-crawleyes

Asks which of your tools to add it to (Claude Code, Codex, Cursor, VS Code, Windsurf, Gemini CLI, Claude Desktop) and shows the change first. See https://agentmart.dev/cli.

Get the weekly diff

The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.

We use your email only for this newsletter. Privacy

About

Web scraping & search for AI agents: extract, SearXNG search, rerank, robots compliance. No API key.

Reviews

Write a review

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Permissions

What this server can do and what it asks for, from its tool list, install config and code. Checked Oct 26, 2026.

Tools
4 read · 0 write · 0 delete · 2 not yet classified
Secrets
1 (API keys, tokens or passwords)
Install scripts
None found
Runs as
Python package

Tools by what they do 6

Read from the source code; the running server may list different tools.

Read 4 · Look things up without changing anything.

  • deep_research
  • extract
  • extract_structured
  • search

Not yet classified 2 · We could not tell from the name and description.

  • crawl
  • sitemap

Secrets and settings it asks for

Secrets: values you paste in that let it act as you.

  • TAVILY_API_KEY

Other settings (environment variables or headers).

  • SEARXNG_URL

Network hosts it talks to 7

  • api.github.com · found in code
  • api.openai.com · found in code
  • api.tavily.com · found in code
  • github.com · found in code
  • hf-mirror.com · found in code
  • raw.githubusercontent.com · found in code
  • sitemaps.org · found in code

Files it touches

  • /home/agentuser/.git-credentials
  • /home/agentuser/crawl-public
  • Your home directory (~)

Tools (6)

Read from the source code; the running server may differ. Descriptions are the server’s own words; hidden characters are shown as [U+…].

  • crawl

    整站递归抓取：从起点 URL 沿站内链接 BFS 抓取多页（对标 Firecrawl crawl）。 每页返回标题/正文长度/正文（markdown）。max_pages/max_depth 控制规模防失控。 需要单页正文可再调 extract。Returns JSON with count/pages.

  • deep_research

    深度调研：把主题拆成子问题→搜索→抓取→合成带引用的报告。 Uses SearXNG → Tavily keyless (no API key). Optional LLM synthesis via CRAWLEYES_LLM_* env vars; without them returns an evidence-aggregate report. Returns JSON with report (Markdown) + sources.

  • extract

    抓取网页正文为 Markdown。自动去噪（过滤导航/广告），失败自动重试。 返回前会剥离隐藏文字 + 标记可疑注入，防止 prompt-injection 劫持 Agent。 respect_robots=True 时先检查目标站 robots.txt (RFC 9309)，被 Disallow 则拒绝。 format: markdown(默认) | fit | raw | markdown_with_citations。 Returns JSON with title/markdown/length.

  • extract_structured

    结构化提取：按 CSS 选择器从页面抽取重复元素为 JSON 数组（对标 Firecrawl/Crawl4AI）。 参数: url: 目标页面 base_selector: CSS 选择器定位重复元素（如 '.product-item'、'.article'） fields_json: 字段定义 JSON 数组，如 [{"name":"标题","selector":"h2","type":"text"},{"name":"链接","selector":"a","type":"attribute","attribute":"href"}] type 支持: text / attribute（需加 attribute 字段）/ html Returns JSON with count/items.

  • search

    搜索网页。SearXNG 优先，失败自动 fallback Tavily keyless（无需 API key）。 带统一限流 + 指数退避重试（最多 3 次）。 limit 会被 clamp 到 [1, 20]，防止传超大值打爆下游。 Returns JSON with title/url/description for each result.

  • sitemap

    发现站点 URL 地图。解析 {origin}/sitemap.xml（支持 index/gzip）， 缺失时回退 robots.txt 的 Sitemap 声明。返回去重后的 URL 列表 JSON。 用于整站抓取 / deep_research 扩 URL 源。带统一限流。

Signals

Stars
2
Forks
0
Open issues
0
Weekly downloads
24
Last commit
Oct 22, 2026 (7 days ago)
Last release
Sep 1, 2026 (1 month ago)
Last package publish
Sep 1, 2026 (1 month ago)

From GitHub (fetched Oct 24, 2026) and the package registry.

Details

Registry name
io.github.waiky-github/CrawlEyes
Version
0.2.0
Auth
API key
Packages
pypi crawleyes
License
MIT
Repository
github.com/waiky-github/CrawlEyes
Website
github.com/waiky-github/CrawlEyes
First seen
Oct 24, 2026
Last checked
Oct 24, 2026
Source
Official MCP Registrygithub-graphql
Cite this page

CrawlEyes MCP Server. Agent Mart. https://agentmart.dev/connectors/waiky-github-crawleyes (last updated 2026-10-24)