corp-osint

Files (4)

• SKILL.md
• mcp.md
• report-schema.md
• sources.md

SKILL.md (first lines)

# Corporate OSINT — Entity Expansion (v3)

Passive public-source recon only. **Plugin architecture:**

- One entrypoint: `uv run python main.py`
- One abstract base: `corp_osint.plugin.Plugin`
- All stages live in `corp_osint/plugins/` and inherit `Plugin`
- Orchestrator: `CorpOsintPipeline(Plugin)` runs registered plugins

## Install

From the **repo root** (`corp-OSINT/`), one-time:

```bash
cd /path/to/corp-OSINT

# https://docs.astral.sh/uv/getting-started/installation/
uv sync
uv run playwright install chromium

# optional (RDAP fallback / richer rate limits)
sudo apt install whois             # or: brew install whois
export SEC_USER_AGENT='CorpOSINT/3.0 (research; you@example.com)'
export URLSCAN_API_KEY=...         # optional — higher urlscan limits
export DNSLYTICS_API_KEY=...       # optional — paid GTM reverse API
# optional proxy (HTTP or SOCKS) for aiohttp + Playwright:
export CORP_OSINT_PROXY='socks5://127.0.0.1:1080'
# export HTTP_PROXY=... HTTPS_PROXY=... ALL_PROXY=...   # also honored
# export NO_PROXY=localhost,127.0.0.1,.local
```

**Requirements:** [uv](https://docs.astral.sh/uv/), Python 3.10+, network access.
MCP hosts launch `corp-osint-mcp` via `uv run` (see `.mcp.json`, `docs/mcp.md`).

**Verify install:**

```bash
uv run python main.py --list-plugins
uv run python main.py --plugin azure_tenant --domain microsoft.com
```

Agents: run the Install block if `uv` is missing or Playwright/Chromium is not installed.

## Agent rules (quality)

1. **Pipeline JSON is raw evidence, not truth.** Regex/EDGAR parsers can emit
   header junk (`Name of the Subsidiary`) or invent slug domains. Critically
   filter before presenting findings.
2. **Present the compact `summary` first** (also `summary_text`). Each domain row:
   `domain`, `sources`, `evidence`, `confidence`, `verified`.
3. **Do not invent subsidiary `.com`s.** Subsidiaries are names until urlscan,
   Entra tenant domains, DNSLytics, or a verified homepage links them.
   If `--domain` / MCP `domain` is omitted, Playwright runs first to seed the
   parent site; inventing parent `.com`/`.net` is only a last-resort fallback.
   If `--domain` / MCP `domain` is omitted, Playwright runs first to seed the
   parent site; inventing parent `.com`/`.net` is only a last-resort fallback.
   If `--domain` / MCP `domain` is omitted, Playwright runs first to seed the
   parent site; inventing parent `.com`/`.net` is only a last-resort fallback.
4. Prefer AI judgment over brittle string matches when classifying parking pages,
   brand vs legal-shell names, and whether a domain belongs to the company.

## Quick run

```bash
uv run python main.py "Example Corp" --domain example.com --out report.json
uv run python main.py -v "Example Corp" --domain example.com --summary-only
uv run python main.py --list-plugins
uv run python main.py --plugin dnslytics_gtm --domain example.com
```

Logging (stderr via logger only — no ``print``): default INFO stage progress;
`-v` / `-vv` DEBUG; `-q` warnings for progress (result JSON still emitted).
Reports are **compact by default** (no WHOIS dumps); `summary` + `summary_text`
are always first keys. Use `--summary-only` for summary JSON alone.

Flags: `--skip STAGES`, `--only STAGES` (comma-separated:
`edgar,urlscan,playwright,asn,dnslytics,tenant_domains,entra_verify,dangling,resolve_sites`),
`--summary-only`, `--proxy URL`, `--direct`, `--no-proxy HOSTS`, `--plugin NAME`,
`--tenant-id`, `-v`/`-vv`, `-q`. Tuning caps live in `PipelineConfig.limits`
(defaults); not exposed on the CLI.

## Proxy (HTTP + SOCKS)

All aiohttp traffic and Playwright Chromium share one proxy setting:

```bash
# SOCKS5
uv run python main.py "Example Corp" --domain example.com --proxy socks5://127.0.0.1:1080

# HTTP proxy + bypass list
uv run python main.py --plugin azure_tenant --domain microsoft.com \
  --proxy http://127.0.0.1:8080 --no-proxy localhost,127.0.0.1

# Force direct (ignore env proxies)
uv run python main.py --plugin azure_tenant --domain microsoft.com --direct
```

Also: `CORP_OSINT_PROXY`, `ALL_PROXY`, `HTTPS_PROXY`, `HTTP_PROXY`, and
`NO_PROXY` / `CORP_OSINT_NO_PROXY`. Requires `aiohttp-socks` (in `requirements.txt`).

## Architecture

| Piece | Role |
|-------|------|
| `uv run python main.py` | **Single entrypoint** |
| `corp_osint.plugin.Plugin` | **Abstract base** — every module inherits this |
| `corp_osint.plugins.*` | Concrete plugins (`edgar`, `urlscan`, …) |
| `corp_osint.plugins.pipeline.CorpOsintPipeline` | Orchestrator plugin (`name="pipeline"`) |
| `corp_osint.http.AsyncHttpClient` | aiohttp + throttle + 429 retries |

To add a stage: subclass `Plugin`, implement `async def run(**kwargs)`, register in
`plugins/__init__.py` → `PLUGIN_REGISTRY`.

## Checklist

```
- [ ] 1 Parent (EDGAR CIK / legal name / domain)
- [ ] 2 Subsidiaries / brands (names; filter Exhibit 21 headers)
- [ ] 3 Resolve parent domain only (no mass subsidiary .com guessing)

From Salto7/Peon, Apache-2.0.

More from Salto7/Peon