Skip to content
Agentmart
Submit a tool

web-scraping

Builds compliant scrapers and datasets with HTTP parsing or Playwright.

Skill by 0xdewy

IndexedMIT
Write a review

Install

Works with Claude Code, Claude apps, Codex, Cursor, Gemini CLI, GitHub Copilot, VS Code, OpenCode, Goose, Windsurf. Read the repository before you give any tool access to your data.

npx agentmarket install skill/0xdewy-web-scraping

Asks which of your tools to add it to (Claude Code, Codex, Cursor, VS Code, Windsurf, Gemini CLI, Claude Desktop) and shows the change first. See https://agentmart.dev/cli.

Get the weekly diff

The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.

We use your email only for this newsletter. Privacy

About

Builds compliant scrapers and datasets with HTTP parsing or Playwright. Only use when explicitly requested.

Use cases: testing

Reviews

Write a review

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Permissions

What this skill can do and what it asks for, from its SKILL.md. Checked Oct 25, 2026.

Tools
No tools found
Secrets
None asked for
Install scripts
None found

Network hosts it talks to 1

  • codeload.github.com · found in install command

Files it touches

  • ~/.claude/skills/web-scraping

What's inside

Skill name
web-scraping
Description
Builds compliant scrapers and datasets with HTTP parsing or Playwright. Only use when explicitly requested.
License
MIT
Instructions
36 lines in SKILL.md · view on GitHub

Files (2)

  • SKILL.md
  • evals.json · evals/

SKILL.md (first lines)

# Web Scraping

Build compliant scrapers and data extractors. Use official APIs when available.

Load `../common/patterns/execution-contract.md` when writing durable scripts
or datasets.

## Safety Gate

Before scraping, check robots/TOS where relevant, rate limits, authentication
permission, data sensitivity, and whether an API is the better route. Refuse
bypass of CAPTCHA, bot defenses, paywalls, geoblocks, access controls, or
robots/TOS restrictions.

Before any code, write a short compliance note: target paths, robots/TOS
status, rate limit, user agent, and the decision to proceed or stop.

## Workflow

1. Define target pages, fields, output format, freshness needs, and allowed
   crawl scope.
2. Choose `httpx`/BeautifulSoup for static HTML; Playwright for JS-rendered or
   interaction-heavy pages.
3. Implement pagination/infinite scroll, retries, rate limits, dedupe, and
   structured output (`csv`, `json`, or `jsonl`).
4. Add screenshots/logging for brittle browser flows.
5. Test on a small sample before full collection.
6. Document run command, limits, and assumptions.

Final line:

```text
DONE: web-scraping — scraper/output ready, sample=<N> records
```

From 0xdewy/skills, MIT.

More from 0xdewy/skills

Signals

Stars
0
Forks
0
Open issues
2
Last commit (this folder)
Oct 11, 2026 (17 days ago)

From GitHub (fetched Oct 24, 2026).

Details

License
MIT
Repository
github.com/0xdewy/skills
First seen
Oct 24, 2026
Last checked
Oct 24, 2026
Source
GitHubgithub-graphql

More from 0xdewy

Cite this page

web-scraping skill. Agent Mart. https://agentmart.dev/skills/0xdewy-web-scraping (last updated 2026-10-24)