x-tweet-search-by-query

Files (3)

• SKILL.md
• build-search-url.py · scripts/
• parse-tweets.py · scripts/

SKILL.md (first lines)

# X — Tweet Search by Query

> Advanced search query → normalized list of tweets matching the query (with author, engagement, media, cursor).

## Language

All process output to user (progress updates, process notifications) follows the user's language.

## Objective

Collect tweets that match an X advanced-search query expression, returning structured per-tweet data and pagination cursors for unbounded bulk collection.

## Prerequisites

- Active X session in the browser (left sidebar shows logged-in avatar / @handle).
- Network capture is enabled in the browser-act session.

## Pre-execution Checks

### 1. Tool Readiness

If browser-act has been confirmed available in the current session → skip this step.

Invoke `browser-act` via Skill tool to load usage. If installation or configuration issues arise, follow its guidance to resolve then retry.

### 2. Login Verification

If login status for X has been confirmed in the current session → skip this step.

Otherwise: open `https://x.com` and observe the left sidebar:
- User avatar or @handle visible at the bottom → logged in, continue
- "Sign in" / "Log in" prompt visible → not logged in, inform the user that an X login is required and assist the login flow

User refuses or cannot log in → terminate execution.

## Capability Components

> This Skill's operational boundary = what the user can manually do in their browser. It only reads tweet data rendered by the user's own X session, never bypassing authentication. Each search request is signed by X's own page JS; the Skill triggers it via URL navigation and reads structured GraphQL responses from network traffic. Python scripts under `scripts/` only build URLs and parse responses — they do not call X directly. Run them through the bash tool.

### Network Capture: search tweets via advanced query

All search parameters are encoded into X's native advanced search syntax via the URL builder, then injected through the page URL. The browser's own JS signs the GraphQL request; the response is read from network traffic and parsed locally.

Step 1 — build the search URL with all desired filters:

`URL=$(python scripts/build-search-url.py '{raw_query}' --sort {sort} [--language {iso}] [--min-retweets {n}] [--min-faves {n}] [--min-replies {n}] [--since YYYY-MM-DD] [--until YYYY-MM-DD] [--author {handle}] [--in-reply-to {handle}] [--mentioning {handle}] [--only-verified] [--only-blue-verified] [--only-image] [--only-video] [--only-quote] [--exclude-retweets] [--exclude-replies] [--geocode LAT,LON,RADIUSmi] [--near 'PLACE'] [--within 15mi])`

Parameters:
- `raw_query` (positional): free-form X advanced search expression; all standard operators are supported and may already include any clause below. Operators added via flags are skipped if the same operator is already present in `raw_query`.
- `--sort`: `Latest` (default) or `Top`. Maps to URL `f=live` and `f=top`, which select GraphQL `product=Latest` vs `product=Top`.
- `--language`: ISO 639-1 code, appended as `lang:CODE`.
- `--min-retweets` / `--min-faves` / `--min-replies`: integer thresholds, appended as `min_retweets:N` / `min_faves:N` / `min_replies:N`.
- `--since` / `--until`: `YYYY-MM-DD`, appended as `since:` / `until:`.
- `--author`: only tweets from this handle, appended as `from:HANDLE`.
- `--in-reply-to`: only tweets replying to this handle, appended as `to:HANDLE`.
- `--mentioning`: only tweets mentioning this handle, appended as `@HANDLE`.
- `--only-verified` / `--only-blue-verified`: appended as `filter:verified` / `filter:blue_verified`.
- `--only-image` / `--only-video` / `--only-quote`: appended as `filter:images` / `filter:native_video` / `filter:quote`.
- `--exclude-retweets` / `--exclude-replies`: appended as `-filter:retweets` / `-filter:replies`.
- `--geocode`: `LAT,LON,RADIUSmi` or `LAT,LON,RADIUSkm`, appended as `geocode:LAT,LON,RADIUS`.
- `--near` / `--within`: appended as `near:"PLACE"` and `within:VALUE`.

Step 2 — navigate and capture the first page:

1. `network requests --clear`
2. `navigate "$URL"`
3. `wait stable --timeout 25000` (a timeout is normal on X; proceed even if it fires — the GraphQL response usually completes before the page reaches network-idle)
4. `network requests --type xhr,fetch --filter SearchTimeline` → take the latest entry's `request_id`
5. `network request <request_id>` → save full output to a file (e.g. `tmp/x-search-page-1.txt`)
6. `python scripts/parse-tweets.py --json-file tmp/x-search-page-1.txt --source search` → emits JSON `{tweets, count, cursor_top, cursor_bottom}`

Endpoint characteristic: URL contains `/i/api/graphql/<hash>/SearchTimeline`. The query hash rotates over time, so always filter by name `SearchTimeline`, never by hash.

Step 3 — paginate via scroll until the desired tweet count is reached or `cursor_bottom` no longer advances:

1. `network requests --clear`
2. `scroll down --amount 5000`
3. `wait stable --timeout 10000`
4. `network requests --type xhr,fetch --filter SearchTimeline` → take the newest entry's `request_id` (its variables now carry the cursor returned by the previous page)
5. `network request <request_id>` → save to `tmp/x-search-page-N.txt`
6. `python scripts/parse-tweets.py --json-file tmp/x-search-page-N.txt --source search`

Repeat Step 3 until any of these termination conditions is met:
- The accumulated unique tweet count reaches the user's target `max_items`.
- `count` returns `0` for the current page.
- `cursor_bottom` is identical to the previous page (the timeline has been fully consumed).

Error handling: if `network requests --filter SearchTimeline` returns no entry after a scroll, the request may have been throttled or the timeline may have ended — wait 3 s and retry the scroll once. If still no new request appears after the second attempt and `cursor_bottom` did not change, stop the loop. If the response file's body is missing (only headers present), the URL is likely stale — re-`navigate` to the same URL and retry.

Output example:
```json
{
  "tweets": [
    {
      "type": "tweet",
      "id": "2068333045510291908",
      "url": "https://x.com/NASA/status/2068333045510291908",
      "twitter_url": "https://twitter.com/NASA/status/2068333045510291908",
      "text": "The official FIFA World Cup ball went to space! ...",
      "created_at": "Thu Jun 20 18:30:11 +0000 2026",
      "lang": "en",
      "source": "Twitter Web App",
      "retweet_count": 4586,
      "reply_count": 1812,
      "like_count": 24499,
      "quote_count": 240,
      "bookmark_count": 1730,
      "view_count": 2098235,
      "is_reply": false,
      "is_retweet": false,
      "is_quote": false,
      "quote_id": null,
      "quote_url": null,
      "in_reply_to_id": null,
      "in_reply_to_user": null,
      "in_reply_to_user_id": null,
      "conversation_id": "2068333045510291908",
      "hashtags": ["FIFAWorldCup"],
      "mentions": [],

From browser-act/skills, MIT.

More from browser-act/skills