web-scraping

Files (4)

• LICENSE
• README.md
• REFERENCE.md
• SKILL.md

SKILL.md (first lines)

# Web Scraping Skill for AI Agents

This skill provides best practices and operational patterns for building and running web scrapers. It covers session management, parallel execution, error recovery, and integration with AI-powered extraction tools.

## When to Use This Skill

- **Building a web scraper** — structuring code for reliability and resumability
- **Running long scrapes** — protecting processes with tmux
- **Lead generation pipelines** — orchestrating multi-stage data extraction
- **Data enrichment** — email finding, validation, and waterfall patterns
- **AI-powered extraction** — using Crawl4AI or LLM-based parsing
- **Remote server scraping** — SSH + tmux for location-independent execution

## Core Principles

### 1. Always Protect Long-Running Processes

Any script that runs longer than 5 minutes MUST run inside a tmux session.

```bash
# Standard pattern — always follow this
tmux new -s descriptive-name
python my_scraper.py --resume
# Ctrl+B, D to detach
```

For AI agents executing scripts autonomously, use detached mode:

```bash
# Launch detached (agent can continue other work immediately)
tmux new -d -s job-name 'python my_scraper.py --resume'

# Check status without attaching
tmux capture-pane -t job-name -p | tail -10

# Check if session is still alive
tmux has-session -t job-name 2>/dev/null && echo "running" || echo "done"
```

### 2. Always Build Resume Support

Every scraper should survive interruption. Implement checkpointing:

```python
import json
import os

PROGRESS_FILE = "output/progress.json"

def load_progress():
    if os.path.exists(PROGRESS_FILE):
        with open(PROGRESS_FILE) as f:
            return json.load(f)
    return {"completed": [], "last_index": 0}

def save_progress(progress):
    # Atomic write — write to temp file, then rename
    tmp = PROGRESS_FILE + ".tmp"
    with open(tmp, "w") as f:
        json.dump(progress, f)
    os.rename(tmp, PROGRESS_FILE)
```

### 3. Always Log to Files

Never rely solely on terminal output:

```bash
# Combine terminal output AND file logging
python scraper.py --resume 2>&1 | tee output/scrape.log
```

In Python, use dual logging:

```python
import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s",
    handlers=[
        logging.FileHandler("output/scrape.log"),
        logging.StreamHandler(),  # Also print to terminal
    ],
)
```

### 4. Respect Rate Limits

Always add delays between requests. When scraping directories or registries:

```python
import asyncio
import random

async def scrape_with_rate_limit(urls, delay_range=(1, 3)):
    for i, url in enumerate(urls):
        result = await fetch(url)
        process(result)

        # Random delay to avoid detection
        delay = random.uniform(*delay_range)
        await asyncio.sleep(delay)

        if i % 100 == 0:
            logging.info(f"Progress: {i}/{len(urls)}")
            save_progress({"last_index": i})
```

### 5. Name Sessions Descriptively

```bash
# Bad — you'll forget what's running
tmux new

# Good — instantly identifiable
tmux new -s email-enrichment-batch-3
tmux new -s directory-scrape-q1-2026
tmux new -s validation-run-feb06

From demonstrikkk/datagoblin, MIT.

More from demonstrikkk/datagoblin