Skip to content
Agentmart
Submit a tool

Open SEO Crawler

Self-hosted SEO crawler: run site audits and AI crawler checks, then read the issues and pages.

Connector by puneetindersingh · GitHub account verified by the MCP Registry

IndexedMITLocal (stdio)★ 71
Write a review

Install

Works with Claude Desktop. Read the repository before you give any tool access to your data.

Download the bundle

MCP bundle (.mcpb): open it with Claude Desktop.

Get the weekly diff

The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.

We use your email only for this newsletter. Privacy

About

Self-hosted SEO crawler: run site audits and AI crawler checks, then read the issues and pages.

Use cases: seo, web-scraping

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Permissions

What this server can do and what it asks for, from its tool list, install config and code. Checked Oct 25, 2026.

Tools
10 read · 3 write · 0 delete
Secrets
None asked for
Install scripts
None found
Runs as
Downloaded binary

Tools by what they do 13

Read from the source code; the running server may list different tools.

Read 10 · Look things up without changing anything.

  • bot_check
  • check_robots
  • compare_crawls
  • crawl_status
  • crawl_summary
  • detect_cms
  • get_brief
  • get_page
  • get_pages
  • list_crawls

Write 3 · Create, change, send or run things.

  • export_xlsx
  • start_crawl
  • stop_crawl

Network hosts it talks to 25

  • ahrefs.com · found in code
  • apple.com · found in code
  • bing.com · found in code
  • books.toscrape.com · found in code
  • chromewebstore.google.com · found in code
  • commoncrawl.org · found in code
  • dev.to · found in code
  • developer.amazon.com · found in code
  • developer.chrome.com · found in code
  • developers.cloudflare.com · found in code
  • developers.google.com · found in code
  • example.comundefined · found in code
  • git-scm.com · found in code
  • github.com · found in install command, code
  • google.com · found in code
  • moz.com · found in code
  • openai.com · found in code
  • other.com · found in code
  • perplexity.ai · found in code
  • platform.openai.com · found in code
  • quotes.toscrape.com · found in code
  • raw.githubusercontent.com · found in code
  • sitemaps.org · found in code
  • support.apple.com · found in code
  • web.dev · found in code

Files it touches

  • Your home directory (~)
  • ~/.site-crawler-crawls
  • ~/.site-crawler-users.json

Tools (13)

Read from the source code; the running server may differ. Descriptions are the server’s own words; hidden characters are shown as [U+…].

  • bot_check

    Fetch pages as Googlebot, Bingbot, AI search, AI assistant and training crawlers and compare each with what a browser gets: blocked, CDN challenge, noindex, redirected, thinner or different content. Requests come from this computer with each crawler's user agent.

  • check_robots

    Fetch and parse a site's robots.txt: rules, sitemaps, and which search and AI crawlers it blocks.

  • compare_crawls

    Diff two saved crawls of a site: pages added, removed and changed, and how the issue counts moved.

  • crawl_status

    Progress of a crawl started in this session: state (crawling, sitemap, near_dup, bot_check, saving, done, stopped, error), pages so far, queue, and the saved crawl name once done.

  • crawl_summary

    What the Summary tab shows, as data: pages, status codes, indexable pages, every issue grouped into errors, warnings and info with page counts and sample URLs, duplicates, redirect chains, sitemap gaps, near duplicates and the bot readability results.

  • detect_cms

    Identify the CMS behind a URL, with the crawl settings and exclude patterns the app suggests for it.

  • export_xlsx

    Write a saved crawl to an .xlsx file on this computer (All Pages and Issues Summary sheets).

  • get_brief

    The app's AI brief for a crawl: a Markdown report grouped by severity with capped URL lists and a page table, under an audit prompt. Good as one read of a whole crawl.

  • get_page

    Everything the crawl holds for one URL (meta, headings, canonical, hreflang, schema types, links, images, issues) plus the internal links pointing at it.

  • get_pages

    Rows of a saved crawl, filtered and paged. Filter by issue (a Summary heading such as "Missing H1", or any text from the issue), severity, status (404, 3xx), URL text or indexability. Default fields: url, status_code, title, title_len, meta_description, meta_len, h1, word_count, depth, indexable, canonical, canonical_kind, response_time, issues. fields="all" returns every field (large).

  • list_crawls

    Saved crawls from the last 30 days, newest first (crawls from the app and from here).

  • start_crawl

    Crawl a website with the same engine and defaults as the app (1500 pages, 5 workers, 0.4 s delay per host, depth 10, robots.txt respected; then sitemap analysis, near-duplicate content and the AI crawler readability check). Returns a job id at once; poll crawl_status. Set max_pages low (20 to 100) for a quick look. The finished crawl is saved and also shows under Load in the app.

  • stop_crawl

    Stop a running crawl like the Stop button. Pages fetched so far are kept, checked and saved.

Compare and alternatives

    Featured in

    Signals

    Stars
    71
    Forks
    23
    Open issues
    4
    Contributors
    1
    Last commit
    Oct 23, 2026 (6 days ago)
    Last release
    Oct 16, 2026 (12 days ago)

    From GitHub (fetched Oct 24, 2026).

    Details

    Registry name
    io.github.puneetindersingh/open-seo-crawler
    Version
    1.0.0
    Auth
    No auth
    License
    MIT
    First seen
    Oct 24, 2026
    Last checked
    Oct 24, 2026

    More from puneetindersingh

    Cite this page

    Open SEO Crawler MCP server. Agent Mart. https://agentmart.dev/connectors/puneetindersingh-open-seo-crawler (last updated 2026-10-24)