crawl4ai

Files (7)

• SKILL.md
• obsidian-publish.yaml · configs/
• slow-site.yaml · configs/
• spa.yaml · configs/
• batch_crawl.sh · scripts/
• README.md · tests/
• test_cli.sh · tests/

SKILL.md (first lines)

<objective>
Extract web content as clean markdown using the `crwl` CLI. This skill handles static sites, JavaScript-rendered SPAs, and structured data extraction without requiring Python code.

**Primary use cases:**

- Convert documentation sites to markdown
- Scrape JavaScript-heavy pages (React, Vue, Obsidian Publish)
- Extract structured data from e-commerce, news, or catalog sites
- Batch process multiple URLs
</objective>

<quick_start>
**Basic crawl:**

```bash
crwl crawl -o md https://example.com
```

**Save to file:**

```bash
crwl crawl -o md -O output.md https://example.com
```

**JavaScript-heavy site (use pre-made config):**

```bash
crwl crawl -C configs/spa.yaml -bc -o md "https://spa-site.com"
```

</quick_start>

<installation>
```bash
# Using uv (recommended) - include [all] for ML features
uv tool install 'crawl4ai[all]'

# Or with pip

pip install 'crawl4ai[all]'

# Run setup after installation

crawl4ai-setup

# Verify installation

crwl --help
crawl4ai-doctor

```

> **Note:** The `[all]` extra is required for ML features. Without it, `crawl4ai-download-models` will fail.
</installation>

<core_commands>
**Markdown extraction:**
```bash
crwl crawl -o md https://docs.example.com           # Basic
crwl crawl -o md-fit https://docs.example.com       # Filtered (removes boilerplate)
crwl crawl -o md -O docs.md https://docs.example.com  # Save to file
```

**Structured data extraction:**

```bash
crwl crawl -j "extract product names and prices" https://shop.com  # LLM-based
crwl crawl -s schema.json https://shop.com                          # Schema-based (faster)
```

**Deep crawling:**

```bash
crwl crawl --deep-crawl bfs --max-pages 10 https://docs.example.com  # Breadth-first
crwl crawl --deep-crawl dfs --max-pages 5 https://blog.example.com   # Depth-first
```

**Question answering:**

```bash
crwl crawl -q "What are the main features?" https://product.example.com
```

</core_commands>

<spa_handling>
**Critical pattern for JavaScript-heavy sites (React, Vue, Obsidian Publish, etc.):**

CSS selectors alone don't work—the element exists but content loads asynchronously. Use a **JS wait condition** that checks for actual rendered text.

**Use the pre-made SPA config:**

```bash
crwl crawl -C configs/spa.yaml -bc -o md "https://spa-site.com"
```

**Or create a site-specific config:**

```yaml
# my_site_config.yaml
page_timeout: 60000
wait_for: "js:() => document.body.innerText.includes('expected content')"
```

```bash
crwl crawl -C my_site_config.yaml -bc -o md "https://my-spa-site.com"
```

**Why `-bc` (bypass cache)?** SPA content can be cached before JS renders. Always use `-bc` for SPAs.
</spa_handling>

<configs>
**Pre-made configs in `configs/` folder:**

| Config | Use Case |
|--------|----------|
| `spa.yaml` | Generic SPA (waits for content length > 1000) |
| `obsidian-publish.yaml` | Obsidian Publish sites |
| `slow-site.yaml` | Sites with slow loading (60s timeout) |

From durandom/skills, GPL-3.0.

More from durandom/skills