Skip to content
Agentmart
Submit a tool

Robotstxt AI

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

Connector by sharozdawa · GitHub account verified by the MCP Registry

IndexedMITLocal (stdio)★ 1
Write a review

Install

Works with Claude Code, Codex, VS Code, Cursor, Windsurf, Claude Desktop. Read the repository before you give any tool access to your data.

npx agentmarket install connector/sharozdawa-robotstxt-ai

Asks which of your tools to add it to (Claude Code, Codex, Cursor, VS Code, Windsurf, Gemini CLI, Claude Desktop) and shows the change first. See https://agentmart.dev/cli.

Get the weekly diff

The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.

We use your email only for this newsletter. Privacy

About

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

Reviews

Write a review

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Permissions

What this server can do and what it asks for, from its tool list and install config. Checked Oct 25, 2026.

Tools
4 read · 0 write · 0 delete · 1 not yet classified
Secrets
None asked for
Install scripts
Not scanned yet
Runs as
Node.js package

Tools by what they do 5

Read from the source code; the running server may list different tools.

Read 4 · Look things up without changing anything.

  • analyze_robots
  • check_bot_status
  • fetch_robots
  • list_ai_bots

Not yet classified 1 · We could not tell from the name and description.

  • generate_robots

Tools (5)

Read from the source code; the running server may differ. Descriptions are the server’s own words; hidden characters are shown as [U+…].

  • analyze_robots

    Analyze pasted robots.txt content. Returns which AI bots are blocked or allowed based on the rules.

  • check_bot_status

    Check if a specific bot is blocked or allowed on a given website by fetching and analyzing its robots.txt.

  • fetch_robots

    Fetch and analyze a robots.txt file from a URL. Returns which AI bots are blocked or allowed.

  • generate_robots

    Generate a robots.txt file with specified blocked bots, sitemap URLs, and custom rules.

  • list_ai_bots

    List all known AI bots with their user-agents, companies, and descriptions. Useful for deciding which bots to block.

Signals

Stars
1
Forks
0
Open issues
0
Last commit
Mar 22, 2026 (7 months ago)
Last release
Mar 21, 2026 (7 months ago)

From GitHub (fetched Oct 24, 2026).

Details

Registry name
io.github.sharozdawa/robotstxt-ai
Version
1.0.1
Auth
No auth
Packages
npm robotstxt-ai-mcp
License
MIT
Repository
github.com/sharozdawa/robotstxt-ai
First seen
Oct 24, 2026
Last checked
Oct 24, 2026
Source
Official MCP Registrygithub-graphql

More from sharozdawa

Cite this page

Robotstxt AI MCP server. Agent Mart. https://agentmart.dev/connectors/sharozdawa-robotstxt-ai (last updated 2026-10-24)