bbb-harvest

Files (1)

• SKILL.md

SKILL.md (first lines)

# BBB Harvest

National data source — county-agnostic. The executable harvester is
`skills/use-oracle/runtime/src/enrichment/bbb.mjs`, exposed by the
`elephant-county bbb-harvest` command and the AWS Batch worker. County market and
category URLs must come from a validated enrichment profile. Output feeds
contractor reputation/quality enrichment and is joined to permits in a later stage.

BBB's `robots.txt` disallows general crawler access to query-string URLs (`/*?`), which
includes category pagination. Treat HTTP 403 as a source-access boundary: stop all
remaining BBB requests, preserve the exact evidence, and request approved access through
the official BBB API at `https://developer.bbb.org/`. Do not change egress, proxies,
browser fingerprints, or challenge behavior to evade the block.

## Run

```bash
cd skills/use-oracle/runtime
node bin/elephant-county.mjs bbb-harvest \
  --county duval \
  --category roofing-contractors \
  --job-id duval-roofing-probe \
  --max-pages 2 --max-profiles 5 --max-requests 100 \
  --max-duration-minutes 30 --output downloads/bbb/duval-roofing-probe
```

Production runs use an approved immutable Batch request whose county and category keys
resolve to the same reviewed enrichment profile.

Important bounds:

- `maxPages` / `maxProfiles` / `maxRequests` / `maxDurationMinutes` — hard crawl bounds
- `profileSubpages` (array) — capture extra tabs per profile
- `pageDelayMs` / `profileDelayMs` — conservative pacing
- `includeHtml` — retain source HTML when access is permitted

AWS Batch writes immutable content-addressed artifacts and a handoff per category.
Checkpoints permit an exact resume after transient infrastructure failure. Permanent
source blocks are not retried.

## Output layout

Under `data/artifacts/enrichment/bbb/<jobId>/` (probes can just use a scratch
subdir there):

- `profiles/profiles-part-NNNN.jsonl` — extracted profile records (chunked)
- `failures/failed-profiles.jsonl` — only classified PERMANENT profile failures are
  recorded here (rather than aborting the crawl); re-run with the failed URLs after a
  reviewed source-access change. TRANSIENT failures (network/5xx) never silently count
  as complete; the Batch job fails and its checkpoint remains available for exact resume
- `manifest/summary.json` — counters for reconciliation

## Workflow

1. Pick the category relevant to the enrichment goal (e.g. construction/contractor
   categories for permit contractor matching).
   - **Multi-Trade Harvesting**: Expand contractor collection across all high-value building trades:
     - Roofing Contractors (`roofing-contractors`)
     - Solar Energy Contractors (`solar-energy-contractors`)
     - Heating and Air Conditioning / HVAC (`heating-and-air-conditioning`)
2. Run a small probe (`maxPages: 2`, scratch output subdir). If BBB returns 403, stop
   without further requests and retain the status, URL, timestamp, category, request
   digest, and Batch job ID as source-access evidence.
3. Estimate total crawl time from probe latency, page/profile counts, delays, retries,
   and safe concurrency. If the estimate is more than 48 hours, ask whether to download
   BBB artifacts anyway, ingest them into the query DB, or retrieve BBB profile data at
   runtime from the owning app/service. For runtime retrieval, capture the expected API
   or scrape path, cache/freshness needs, and failure behavior. Stop here if the
   operator chose ingest-only or runtime retrieval — do not proceed to step 4.
4. Run the full category with conservative delays only when the bounded probe is allowed.
5. Reconcile `summary.json` counts vs profiles parts, retry failures.
6. Load into the `bbb_*` tables per the `query-db-loading-matching` skill.

Reconcile a completed profile-category directory set using the profile's category keys:

```bash
node bin/elephant-county.mjs bbb-reconcile \
  --county duval \
  --harvest-root <root-containing-category-key-directories> \
  --input-coverage <dataset-coverage.json> \
  --output-dir <reconciled-dir>
```

After permits are available, link BBB businesses to properties through the retained
private permit-contractor source. Do not join a BBB office address directly to a property:
`has_bbb_contractor` means that the BBB business appears as contractor on a permit already
linked to that property.

```bash
node bin/elephant-county.mjs bbb-link \
  --county duval \
  --input-parquet <query-table.parquet> \
  --input-coverage <dataset-coverage.json> \
  --bbb-profiles <bbb-profiles.jsonl> \
  --bbb-reconciliation-manifest <bbb-reconciliation-manifest.json> \
  --permit-source <private/jaxepics-bid-map.jsonl.gz> \
  --permit-artifact-manifest <permit-artifact-manifest.json> \
  --output-dir <linked-dir>
```

The linker automatically accepts unique exact license, phone, and normalized-name
matches. Jaro-Winkler matches are review candidates only and must not set accepted property
flags without explicit review. Preserve accepted links and review candidates as private
audit artifacts. The linker must reproduce the permit reconciliation exclusions and
deduplication, reconcile source/loaded/linked/excluded permit counts exactly, and prove
that every BBB-flagged property also has `has_permits = true` before lexicon validation.
The flag links the current BBB snapshot to historical permit contractor
identities; it does not claim that a contractor held BBB accreditation when an older
permit was issued.

For a reviewed 403 outcome, submit browser-free zero-profile artifacts and reconciliation:

```bash
npm run batch:recover-blocked -- \
  --config <request.json> \
  --receipt <prior-submission.json> \
  --evidence <blocked-evidence.json>
```

The evidence must match the exact run, request digest, category, and failed Batch job.

From soofi-xyz/soofi-xyz-team-kit, MIT.

More from soofi-xyz/soofi-xyz-team-kit