data-acquisition-browser

Files (4)

• SKILL.md
• patchright_cookie_endpoint_probe.mjs · scripts/
• patchright_page_dom_probe.mjs · scripts/
• playwright_probe.mjs · scripts/

SKILL.md (first lines)

# Data Acquisition Browser

Act as the browser acquisition specialist. Use Patchright for warm-session capture when a normal browser must mint cookies or storage state before API endpoints are visible. Use Playwright for ordinary rendered-DOM fallback when structured HTTP probes are insufficient and no warm browser context is needed.

When the user asks for page loads only or "no API", use Patchright as a renderer and extract DOM/JSON-LD/meta/visible rows only. Do not harvest, replay, or recommend structured endpoints in that mode.

## Shared Core

Read from `../data-acquisition-core/references/`:

- `source-access.md`
- `playwright-rendered-dom.md`
- `probing.md`
- `compliance-boundaries.md`
- `output-contracts.md`

## Helpers

Use `scripts/patchright_cookie_endpoint_probe.mjs` for warm-session cookie/storage generation and endpoint discovery. Use `scripts/patchright_page_dom_probe.mjs` for Patchright page-load-only DOM extraction. Use `scripts/playwright_probe.mjs` only for ordinary public rendered-DOM fallback.

From the repo root:

```powershell
npm install
npx patchright install chrome
npm run probe:patchright -- "https://example.com/public-category" "outputs/example-patchright-endpoints.json"
```

The Patchright helper opens a persistent Chrome context, lets the page create ordinary browser-issued cookies/storage state, records JSON/API/XHR-looking requests and responses, saves local storage state under `auth/`, and writes a redacted endpoint report.

Page-load-only mode:

```powershell
$env:PATCHRIGHT_HEADLESS = "0"
npm run probe:patchright-page -- "https://example.com/category" "outputs/example-page-dom.json"
```

This mode records rendered page metadata, JSON-LD, canonical URL, visible text, candidate links, product/listing-like DOM nodes, and a screenshot. It sets `api_endpoint_discovery: false` and `replay_attempted: false`.

Useful options:

```powershell
$env:PATCHRIGHT_STORAGE_STATE = "auth\target-storage-state.json"
$env:PATCHRIGHT_USER_DATA_DIR = "auth\target-profile"
$env:PATCHRIGHT_HEADLESS = "0"
npm run probe:patchright -- "https://example.com/category" "outputs/target-endpoints.json"
```

Playwright fallback:

```powershell
npx playwright install chromium
npm run probe:playwright -- "https://example.com/public-page" "outputs/example-playwright-probe.json"
```

Never print or commit storage state, cookies, account pages, or local browser profiles.

## Autonomous Browser Fallback Rules

Use this ladder for tiny feasibility probes:

1. Try headless only when it is likely to behave like an ordinary public browser.
2. If headless returns a block page but the user allows browser probing, retry once with Patchright non-headless and a persistent local profile.
3. If non-headless loads the public page, extract a sample of at most 20 visible rows and classify the route as browser-context dependent.
4. If non-headless also returns access denied, CAPTCHA, login, or verification, stop. Do not add stealth, CAPTCHA solving, identity rotation, or bypass logic.
5. If the user requested no API, use `probe:patchright-page`; do not run endpoint discovery or replay.

## Warm Session Capture

Use this when a normal public page, such as a retail category page, issues browser-context XHR/fetch requests that cold HTTP probes cannot reproduce.

Capture locally:

- user-owned browser cookies/storage state generated by Patchright
- observed endpoint URL and query params
- safe request headers such as `Accept`, `Accept-Language`, `Origin`, and `Referer`
- response field shape, pagination params, and sample rows

Do not capture or publish cookies, auth tokens, CAPTCHA tokens, account state, cart state, or fingerprint material.

Required sequence:

1. Run `npm run probe:patchright` against the public page or explicitly authorized owned-session page.
2. Review `endpoint_candidates` and `network.structured_*` in the report.
3. Convert promising URLs into `EndpointPlan` templates with path params, query params, pagination, and safe headers.
4. Replay only tiny probes in the same user-owned context when cookies/storage are required.
5. Mark the source access as `owned_session` and output label as `non_public_authorized_result` whenever replay depends on local state.

See `../data-acquisition-core/references/execution-adapters.md` for optional runtime adapters, including Jacob Padilla's Stealth-Requests and Google-Colab-Selenium projects.

From Pranjay-kumar/universal-data-acquisition-pipeline-skill, MIT.

More from Pranjay-kumar/universal-data-acquisition-pipeline-skill