Scrape-LE

Analyse robots.txt content and report whether a path may be crawled.

Connector by nolindnaidoo · GitHub account verified by the MCP Registry

IndexedMITLocal (stdio)★ 1

Install

Works with Claude Code, Codex, VS Code, Cursor, Windsurf, Claude Desktop. Read the repository before you give any tool access to your data.

npx agentmarket install connector/nolindnaidoo-scrape-le

Asks which of your tools to add it to (Claude Code, Codex, Cursor, VS Code, Windsurf, Gemini CLI, Claude Desktop) and shows the change first. See https://agentmart.dev/cli.

Get the weekly diff

The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.

About

Analyse robots.txt content and report whether a path may be crawled.

Reviews

Write a review

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Permissions

What this server can do and what it asks for, from its tool list, install config and code. Checked Oct 25, 2026.

Tools: 2 read · 0 write · 0 delete · 1 not yet classified
Secrets: None asked for
Install scripts: None found
Runs as: Node.js package

Tools by what they do 3

Read from the source code; the running server may list different tools.

Read 2 · Look things up without changing anything.

analyze_robots_txt
scrape_le_check

Not yet classified 1 · We could not tell from the name and description.

scrape_le_doctor

Files it touches

/private/file
/private/open/y
/private/x
Your home directory (~)

Tools (3)

Read from the source code; the running server may differ. Descriptions are the server’s own words; hidden characters are shown as [U+…].

analyze_robots_txt
Given the contents of a robots.txt file and a path, report whether the rules permit crawling it — the group naming `agent` when one does, otherwise the generic (User-agent: *) rules, and `agent` in the answer says which — along with the crawl delay, the disallowed patterns and any sitemaps. Takes the file contents directly and makes no network request of its own — fetch robots.txt with your own HTTP tool and pass what it returned.
scrape_le_check
Check whether a page can be scraped: robots.txt, anti-bot vendors, rate limits and authentication walls. Accepts one URL or several. A restricted or inconclusive answer is a normal result, not an error.
scrape_le_doctor
Report whether a browser is available, which one, and which checks can run completely.