instagram-lead-scraper

Files (9)

• SKILL.md
• hashtag-guide.md · references/
• quality-checklist.md · references/
• apify-scraper.js · scripts/
• reddit-scraper.ts · scripts/
• save-to-drive.py · scripts/
• scrape-agent-browser.ts · scripts/
• scrape-leads-scaled.ts · scripts/
• scrape-leads.ts · scripts/

SKILL.md (first lines)

# Instagram Lead Scraper (Agent Browser Edition)

**Pure agent-browser scraping** for high-quality insurance agent leads with full contact details.

## What's Different (Agent Browser vs Brave Search)

| Feature | Brave Search | **Agent Browser** |
|---------|--------------|-------------------|
| **Speed** | 50 leads in 30s | 50 leads in 10-15 min |
| **Data Quality** | Basic (username, bio, followers) | **Rich (phone, email, location)** |
| **Source** | Google search results | **Instagram directly** |
| **Discovery** | Limited | **Hashtag pages = more leads** |
| **Contact Info** | ❌ | **✅ Phone, Email extracted from bios** |

**Trade-off:** Slower but way better data.

---

## Quick Start

### Run Agent Browser Scraper (50 leads, ~10-15 min)
```bash
cd skills/instagram-lead-scraper/scripts
npx tsx scrape-agent-browser.ts 50
```

### Options
```bash
# No Google Drive upload
npx tsx scrape-agent-browser.ts 50 --no-drive

# Small test (3 leads, ~2 min)
npx tsx scrape-agent-browser.ts 3 --no-drive
```

---

## How It Works

```
1. OPEN hashtag page (e.g., #insuranceagent)
2. EXTRACT usernames from profile pictures
3. VISIT each profile page
4. SCRAPE: bio, followers, posts, email, phone, location
5. FILTER: only insurance-related profiles
6. SAVE to database + CSV
```

### Data Extracted

| Field | Example |
|-------|---------|
| **username** | @garcia_agency |
| **full_name** | The Garcia Agency |
| **bio** | American Family Insurance Auto \| Home \| Life... |
| **followers** | 4,375 |
| **following** | 493 |
| **posts** | 196 |
| **📧 email** | teamgarcia@amfam.com |
| **📞 phone** | 470.854.2010 |
| **📍 location** | Atlanta, GA |
| **niche** | auto_home / life_insurance / medicare |
| **tags** | has-email, has-phone, has-location |

---

## Hashtags Scraped

The scraper rotates through these hashtags:

1. `#insuranceagent`
2. `#lifeinsuranceagent`
3. `#finalexpense`
4. `#finalexpenseagent`
5. `#lifeinsurance`
6. `#insurancebroker`
7. `#medicareagent`
8. `#healthinsuranceagent`
9. `#insuranceagency`
10. `#insurancesales`

Each hashtag yields 10-15 profiles before moving to the next.

---

## Database Schema

All fields are stored in the `leads` table:

```typescript
{
  source: 'instagram',
  sourceType: 'profile',
  externalId: 'garcia_agency',  // username
  externalUrl: 'https://instagram.com/garcia_agency',
  fullName: 'The Garcia Agency',
  email: 'teamgarcia@amfam.com',
  phone: '470.854.2010',
  location: 'Atlanta, GA',
  bio: 'American Family Insurance...',
  followers: 4375,
  following: 493,
  posts: 196,
  leadType: 'agent',
  niche: 'auto_home',
  tags: 'has-email,has-phone,has-location',
  status: 'new',
  scrapedDate: '2026-02-02'
}
```

---

## Usage

### As a Skill

When you say:
- "scrape instagram leads" → Runs agent-browser scraper

From consuelohq/openengineer, MIT.

More from consuelohq/openengineer