apify-reddit-scraper

Files (3)

• SKILL.md
• gotchas.md · references/
• input-mapping.md · references/

SKILL.md (first lines)

# Reddit Scraper

Turn a plain-language Reddit request into one run of `harshmaur/reddit-scraper` and hand back structured posts, comments, users or communities. One Actor covers search, subreddit listings, permalinks, user profiles and community metadata, so routing is about picking the right **input field**, not the right Actor.

Disclosure: the Actor this skill routes to is paid (pay per result) and built by the skill author. Links carry no affiliate parameters.

## When to Use

- The user wants Reddit posts, comments, subreddits or user profiles as structured data.
- The user wants to monitor a brand, product or competitor on Reddit, or track mentions since a date.
- The user wants every comment from a thread, or the newest posts from one or more subreddits.
- The user lost access to the Reddit API or Reddit RSS feeds and needs the same data without an API key.

## Example prompts

Prompts this skill handles:

- "Find every Reddit post from the last 30 days where someone asks for a Notion alternative, and pull the comments."
- "Scrape the newest 200 posts in r/SaaS and r/startups."
- "Give me all comments on this thread: https://www.reddit.com/r/webdev/comments/abc123/..."
- "Track mentions of `Linear` and `Jira` on Reddit since 2026-08-01 and send each hit to my Slack channel."
- "What has u/spez posted recently?"

Out of scope (the boundary):

- "Post a reply in r/startups" — this Actor only reads Reddit. Use the Reddit API with the user's own account.
- "Find Reddit threads *about* the pain of managing invoices" — Reddit search is literal keyword matching, not semantic. Expand the topic into 3–8 short literal keywords first (see Step 1), then run one search per keyword.

## Run boundaries

Verify current Actor schema, pricing and build before dispatch; the author's historical measurements below are not guarantees. Use an authorized spending cap and supported platform charge limit, including enrichment/retries. Collect only necessary public data; do not bypass access restrictions or infer sensitive personal traits. Connector delivery and schedules require explicit authorization for the destination/cadence. Never infer per-term provenance from row position or counts.

## Prerequisites

- Apify account ([sign up](https://apify.com)); the free plan covers small runs.
- Authentication via one of:
  - `apify login` (OAuth, if using the Apify CLI)
  - `APIFY_TOKEN` environment variable
  - Token from [Apify Console → Settings → Integrations](https://console.apify.com/settings/integrations)
- Apify CLI (`npm install -g apify-cli`), or the Apify MCP server (see "Calling the Actor").

## Workflow

### Step 1 — Classify the request and pick input fields

| User gives you | Field to use | Notes |
|---|---|---|
| Keywords, a brand, a product, a phrase | `searchTerms` (array) | One search per term, each with its own `maxPostsCount`. Keep terms short and literal (`"notion alternative"`, not a sentence). Free plans search the first 40 terms per run. |
| Keywords + "only in r/X" | `searchTerms` + `withinCommunity` | Accepts `developers`, `r/developers` or the full subreddit URL. |
| A subreddit to scrape in bulk | `subredditUrls` (array of names or community URLs) | Fetches far more posts than a plain listing. **Never put post permalinks here** — they are dropped silently and the run finishes with 0 items. |
| Post, user-profile, subreddit or search-page URLs | `startUrls` (array of `{ "url": ... }`) | Search options (sort/time/community) do NOT apply to these. `/s/` share links work too; the Actor resolves them. |
| "since <date>" / "between <dates>" / "last month" / "past week" | `postedAfter`, `postedBefore` (`YYYY-MM-DD`) | Relative periods too: "last month" → `postedAfter` = today − 1 month. Also `commentedAfter` / `commentedBefore` for comments. Results are fetched newest-first and stop early once older than the window; `searchSort` and `searchTime` are ignored while a date is set. |
| "with comments" / "full thread" | `crawlCommentsPerPost: true` + `maxCommentsPerPost` | Multiplies result count and cost — see gotchas. |
| "comments that mention X" | `searchComments: true`, `searchPosts: false` | Comment search across Reddit. |
| "which subreddits are about X" | `searchCommunities: true` | Returns `dataType: "community"` rows. |
| "send to Slack / Notion / Sheets" | `mcpConnector` + `mcpTarget` (+ `mcpMessage`) | Requires an authorized MCP connector in the user's Apify account; see `references/input-mapping.md`. |

When the user's phrase is descriptive rather than literal ("people frustrated with their CRM"), draft 3–8 literal keywords (`"CRM sucks"`, `"switching from HubSpot"`, `"CRM recommendation"`), put them in the run section of your answer (Step 3), and run them as `searchTerms`.

### Step 2 — Build the input

Start from these defaults and change only what the request needs:

```json
{
  "searchTerms": ["notion alternative"],
  "searchPosts": true,
  "searchComments": false,
  "searchCommunities": false,
  "searchSort": "new",
  "searchTime": "month",
  "maxPostsCount": 50,
  "crawlCommentsPerPost": false,
  "maxCommentsPerPost": 50,
  "includeNSFW": false
}
```

- `searchSort`: `new` for monitoring, `relevance` for precise matching, `top` / `hot` for what performed, `comments` for the most discussed.
- `searchTime`: `hour` | `day` | `week` | `month` | `year` | `all`. When the user names a period — a date or a relative one like "last month", "past week" — you have two options, and you must pick one deliberately:
  - **Exact window:** `postedAfter` = today − N (and `postedBefore` if there is an end). Requests the window; verify timestamps and inputWarnings before claiming it, but forces newest-first (`sort=new`) and ignores `searchSort` and `searchTime`, so a second run of the same window with `relevance` returns the same posts again — change the terms or the window instead.
  - **Relevance ranking:** `searchSort: "relevance"` + `searchTime` and no dates. Better ranked, but the window is not guaranteed: `searchTime: "month"` with relevance returned 12 of 30 posts from 2022–2026-01 in one measured run (quoted phrases) and 0 of 24 out of window in two others (plain terms). If you take this option, check `createdAt` in the data and drop rows outside the period yourself, and say so in the report.
  Default to the exact window unless the user asks for the best-ranked matches.
- `maxPostsCount` is applied **per search term** (and per subreddit or profile URL), not as a run total: 4 terms × `maxPostsCount: 8` returned 32 posts. The input schema's title says "total across all inputs"; the README and measured runs say per term — trust per term. Max 50,000. Reddit itself caps any single listing or search at roughly 1,000 posts; for more history split by date windows or by subreddit.
- Keep the default memory (512 MB). Memory needs vary with the Actor build and input; inspect failures before increasing resources. Do not send `fastMode`: it has no effect, and the Actor ignores it.
- Do not pass `proxy`; the Actor manages its own proxying.

Fetch the live schema if a field is in doubt:

```bash
apify actors info "harshmaur/reddit-scraper" --input --json \
  --user-agent apify-awesome-skills/apify-reddit-scraper 2>/dev/null
```

### Step 3 — Show the terms and the estimate, then run

Pricing is pay per result: **$0.02 per run start + $0.002 per stored item** ($2.00 per 1,000; $1.50 per 1,000 on plans with Store discounts). Estimate before running, with `maxPostsCount` counted **per term**:

```
posts ≈ number of searchTerms × maxPostsCount   (plus maxPostsCount per subreddit or profile URL)
items ≈ posts × (1 + maxCommentsPerPost if crawlCommentsPerPost is on)
cost  ≈ $0.02 + items × $0.002
```

Worked example from a real run: 5 terms × `maxPostsCount: 24` × (1 + 5 comments) = 720 items ≈ $1.46 worst case. Read as a total it looked like 24 × 6 = 144 items ≈ $0.31, and a $0.50 charge limit stopped the run at 241 items with two threads' comments cut off. Run only within the user's authorized budget; a small estimate alone does not authorize spending. `crawlCommentsPerPost` is the usual surprise: 200 posts × 50 comments = 10,000 comments ≈ $20.

Your answer must contain a **run section** for every run, written with the numbers filled in — not left in your reasoning — and it has to hold both halves:

```
Run 1 — terms: ["switching CRM", "hate my CRM", "CRM sucks", "looking for CRM"]
Estimate: 4 terms × 6 posts = 24 posts; 24 × (1 + 3 comments) = 96 items; $0.02 + 96 × $0.002 = $0.21
Settled:  91 items, $0.202 — estimate ≥ settled: yes
```

The first two lines are decided before the run starts; the third is filled in from the finished run — `apify runs info "RUN_ID" --json --user-agent apify-awesome-skills/apify-reddit-scraper 2>/dev/null` prints the run object with `usageTotalUsd`, `statusMessage` and `defaultDatasetId`. `usageTotalUsd` keeps growing for a short while after the run finishes — re-read it until two reads agree before you call it settled. If settled is higher than the estimate, say what you mis-counted (usually `maxPostsCount` read as a total). Then run:

```bash
apify actors call "harshmaur/reddit-scraper" \
  -i '{"searchTerms":["notion alternative"],"searchTime":"month","maxPostsCount":50}' \

From sickn33/agentic-awesome-skills, MIT.

More from sickn33/agentic-awesome-skills