crawlability-audit
Audits a website for crawlability issues including robots.txt directives, sitemap presence, HTTP errors, canonical URLs, and page discoverability.
Skill by mahinigam
Install
Works with Claude Code, Claude apps, Codex, Cursor, Gemini CLI, GitHub Copilot, VS Code, OpenCode, Goose, Windsurf. Read the repository before you give any tool access to your data.
npx agentmarket install skill/mahinigam-crawlability-audit
Asks which of your tools to add it to (Claude Code, Codex, Cursor, VS Code, Windsurf, Gemini CLI, Claude Desktop) and shows the change first. See https://agentmart.dev/cli.
Get the weekly diff
The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.
About
Use cases: security
Reviews
Write a reviewNo reviews yet. Installed it? Reviews come from verified installers only. How reviews work
Permissions
What this skill can do and what it asks for, from its SKILL.md. Checked Oct 25, 2026.
- Tools
- No tools found
- Secrets
- None asked for
- Install scripts
- None found
- Runs as
- Python package
Network hosts it talks to 1
codeload.github.com· found in install command
Files it touches
~/.claude/skills/crawlability-audit
What's inside
- Skill name
crawlability-audit- Description
- Audits a website for crawlability issues including robots.txt directives, sitemap presence, HTTP errors, canonical URLs, and page discoverability.
- License
- MIT
- Instructions
- 19 lines in SKILL.md · view on GitHub
Before installing
- Ships executable scripts
Agent Skills spec issues
- allowed-tools: must be a space-separated string
Files (2)
- SKILL.md
- audit.py · scripts/
SKILL.md (first lines)
# Crawlability Audit ## When to use Use this skill when diagnosing if a website is discoverable by search crawlers and AI assistants. If a page cannot be crawled, it effectively doesn't exist for AI systems. ## Inputs - `AuditContext` containing the target URL and initial discovered pages. ## Procedure 1. Check if `robots.txt` exists and parses properly. 2. Check if a sitemap exists and is valid (extracting URL sets). 3. Assess the discoverability of URLs using the `SiteCrawler` and `PageImportance` model, which prioritizes commercial relevance, navigation score, shallow crawl depth, and inbound link counts. 4. Check for consistent canonicalization and HTTP errors (4xx, 5xx) on sampled pages. 5. Generate findings based on evidence of poor discoverability or crawlability blocks. ## Output Appends Findings and Opportunities to the `AuditContext`.
From mahinigam/skill-marketplace, MIT.
More from mahinigam/skill-marketplace
- 0.522score
freshness-and-corroboration-audit
mahinigam
Audits the freshness of information and internal consistency of facts (e.g.
SkillBFreshness: BClaude Code +9 - 0.522score
opportunity-engine
mahinigam
Generates proactive recommendations to improve AI discoverability and engagement, even when explicit defects are absent.
SkillBFreshness: BClaude Code +9 - 0.522score
semantic-integrity-audit
mahinigam
Audits semantic integrity and attribute ownership to detect category confusion and cross-product attribute contamination.
SkillBFreshness: BClaude Code +9 - 0.522score
audit-orchestrator
mahinigam
The main entrypoint for the AI Readiness Intelligence Engine.
SkillBFreshness: BClaude Code +9 - 0.517score
render-and-content-audit
mahinigam
Compares initial HTML with JS-rendered expectations to identify content only available after client-side hydration, which may be missed by AI assistants.
SkillBFreshness: BClaude Code +9 - 0.517score
semantic-structure-audit
mahinigam
Audits JSON-LD structured data and schema.org usage to ensure important entities and facts are machine-readable and consistent with visible text.
SkillBFreshness: BClaude Code +9 - 0.517score
ai-landing-context-audit
mahinigam
Assesses on-site engagement quality when a user lands on the page from an AI citation, ensuring context continuity and intent match.
SkillBFreshness: BClaude Code +9 - 0.517score
answerability-audit
mahinigam
Evaluates if a machine could construct a correct answer to an important user question from the available page evidence without guessing.
SkillBFreshness: BClaude Code +9 - 0.517score
entity-resolution-audit
mahinigam
Analyzes brand, organization, and product names across the website to detect entity ambiguity or inconsistent identity signals.
SkillBFreshness: BClaude Code +9
Signals
- Stars
- 0
- Forks
- 0
- Open issues
- 0
- Last commit (this folder)
- Sep 9, 2026 (1 month ago)
From GitHub (fetched Oct 24, 2026).
Details
- License
- MIT
- Repository
- github.com/mahinigam/skill-marketplace
- First seen
- Oct 24, 2026
- Last checked
- Oct 24, 2026
- Source
- GitHubgithub-graphql
More from mahinigam
- 0.522score
freshness-and-corroboration-audit
mahinigam
Audits the freshness of information and internal consistency of facts (e.g.
SkillBFreshness: BClaude Code +9 - 0.522score
opportunity-engine
mahinigam
Generates proactive recommendations to improve AI discoverability and engagement, even when explicit defects are absent.
SkillBFreshness: BClaude Code +9 - 0.522score
semantic-integrity-audit
mahinigam
Audits semantic integrity and attribute ownership to detect category confusion and cross-product attribute contamination.
SkillBFreshness: BClaude Code +9 - 0.522score
audit-orchestrator
mahinigam
The main entrypoint for the AI Readiness Intelligence Engine.
SkillBFreshness: BClaude Code +9 - 0.517score
render-and-content-audit
mahinigam
Compares initial HTML with JS-rendered expectations to identify content only available after client-side hydration, which may be missed by AI assistants.
SkillBFreshness: BClaude Code +9 - 0.517score
semantic-structure-audit
mahinigam
Audits JSON-LD structured data and schema.org usage to ensure important entities and facts are machine-readable and consistent with visible text.
SkillBFreshness: BClaude Code +9
Cite this page
crawlability-audit skill. Agent Mart. https://agentmart.dev/skills/mahinigam-crawlability-audit (last updated 2026-10-24)