Skip to content
Agentmart
Submit a tool

Robots.txt & AI Crawler Checker

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

Connector by robots-check.quietworklabs.workers.dev · Domain verified by the MCP Registry

IndexedLicense unknownRemote (HTTP)
Write a review

Install

Works with Claude Code, VS Code, Cursor, Windsurf. Read the repository before you give any tool access to your data.

npx agentmarket install connector/robots-check-quietworklabs-workers-dev-robots-check

Asks which of your tools to add it to (Claude Code, Codex, Cursor, VS Code, Windsurf, Gemini CLI, Claude Desktop) and shows the change first. See https://agentmart.dev/cli.

Remote endpoint

  • https://robots-check.quietworklabs.workers.dev/mcp Remote (HTTP)

Get the weekly diff

The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.

We use your email only for this newsletter. Privacy

About

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

Reviews

Write a review

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Permissions

What this server can do and what it asks for, from its tool list and install config. Checked Oct 25, 2026.

Tools
3 read · 0 write · 0 deleteRead-only
Secrets
None asked for
Install scripts
Nothing to install (hosted)
Runs as
Hosted (remote endpoint)

Tools by what they do 3

From the running server's own tool list (tools/list) on Oct 25, 2026.

Read 3 · Look things up without changing anything.

  • ai_crawler_access
  • robots_check
  • sitemap_check

Network hosts it talks to 1

  • robots-check.quietworklabs.workers.dev · found in remote endpoint, install command

Tools (3)

As the server listed them on Oct 25, 2026. Descriptions are the server’s own words; hidden characters are shown as [U+…].

  • ai_crawler_accessWhich AI crawlers a site allows or blocksread-only

    AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.

  • robots_checkMay this user-agent crawl this URL? (RFC 9309)read-only

    Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI. Price: $0.001 USDC per call via x402 (Base); the first 5 calls per day per client are free.

  • sitemap_checkFind and validate an XML sitemapread-only

    Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.

Signals

Last release
Oct 18, 2026 (11 days ago)

Details

Registry name
dev.workers.quietworklabs.robots-check/robots-check
Version
1.0.0
Auth
Auth not stated
License
License unknown
Website
robots-check.quietworklabs.workers.dev/
First seen
Oct 24, 2026
Last checked
Oct 24, 2026
Source
Official MCP Registry
Cite this page

Robots.txt & AI Crawler Checker MCP server. Agent Mart. https://agentmart.dev/connectors/robots-check-quietworklabs-workers-dev-robots-check (last updated 2026-10-24)