analytics-instrumentation-check

Files (2)

• SKILL.md
• detection-patterns.md · references/

SKILL.md (first lines)

# Analytics Instrumentation Check

## What this skill is for

The CI gate enforces that any event passed to `trackServerEvent` is in the typed catalog (`packages/analytics/src/event-types.ts`). It does NOT enforce that an event fires when a state-changing action runs. A developer can ship a new server action that mutates the database and fires no event, and `pnpm typecheck` will pass.

This skill closes the coverage gap. Given a diff, branch, or directory, it:

1. Detects every **state-changing action** introduced or touched.
2. For each action, looks for a `trackServerEvent` / `tracker.track*` / `analytics.trackEvent` call in the same code path.
3. Classifies each action as **instrumented**, **missing**, **ambiguous**, or **intentionally untracked**.
4. Produces a reviewer-ready report so the missing instrumentation is fixed before the feature ships.

The brief that introduced the CI gate explicitly framed this as the manual half of enforcement. The gate ensures shipped events are typed; this skill ensures shipped actions are tracked.

## When to apply

Apply this skill when:

- A user asks for an instrumentation audit before a PR ("is this tracked?", "do we fire an event for X?", "audit analytics coverage on this branch").
- A user is reviewing a PR that touches `apps/web/app/**/_lib/server/`, `apps/ai/src/modules/**`, `apps/sync/src/modules/**`, or `apps/horde-v2/src/delivery/**`.
- A new domain entity (brief, task, blueprint, skill, method, connection, plan, alignment vote) is being created or mutated by code introduced in the diff.
- The user mentions a feature being "ready to ship" or "ready for review" and you can see uninstrumented mutation paths.

## When NOT to apply

- Read-only queries, repository getters, list endpoints — these are never expected to fire events.
- Changes to existing **typed** events (event-name renames, property-shape edits) — that work belongs in the catalog, not in instrumentation coverage.
- Webhook-receipt code that only acks (`200 OK`) without acting — instrumentation belongs in the consumer that processes the queued job, not the controller that enqueues.
- Unit-test files, fixtures, mocks (`*.spec.ts`, `*.test.ts`, `__tests__/**`).
- Backfill scripts in `apps/web/supabase/scripts/` — those are one-off runs and have their own audit-log mechanism via `backfill-script-creator`.

If the user's request fits one of these, say so explicitly and stop.

## Required inputs

The skill needs ONE of:

1. **A diff range** — `origin/dev...HEAD`, `HEAD~3...HEAD`, or a path to a saved diff (`/tmp/foo-diff.txt`). Most common case.
2. **A PR number** — fetched via `gh pr diff <number>`.
3. **A directory or file list** — for example `apps/web/app/home/[account]/briefs/_lib/server/` after a feature lands but before instrumentation is verified.

If none is provided, ask the user which scope to audit. Do not guess.

## Workflow

### 1. Bound the scope

Pick the input form (diff / PR / paths) and produce a list of changed files. Exclude:

- `*.spec.ts`, `*.test.ts`, files under `__tests__/`
- `*.md`, `*.json` (except for analytics dashboard configs in `apps/ai/scripts/`, see Pitfalls)
- `packages/analytics/**` itself (we're auditing consumers, not the library)
- `*.sql`, `*.config.ts`, `*.config.mjs`

For a diff, get the file list:

```bash
git diff --name-only origin/dev...HEAD \
  | grep -vE '\.(spec|test)\.ts$|^__tests__/|\.(md|json|sql)$|^packages/analytics/'
```

### 2. Load the live event catalog

Before scanning for instrumentation, extract the authoritative event name list from `event-types.ts` so the search patterns are never stale:

```bash
# Pull every quoted string from EventPropertiesMap — one event name per line
grep -oP "(?<=')\b[A-Z][^']+(?=')\s*:" \
  packages/analytics/src/event-types.ts \
  | sed "s/[[:space:]]*://"
```

This gives you the full catalog as it exists right now (e.g. `Brief Created`, `Task Completed`, `Integration Connected`, …). Use it as the `<EVENTS>` list in the grep below.

Once you have the list, build a single-pass ripgrep that scans every `trackServerEvent` call site and checks whether the literal string matches a catalog entry:

```bash
# Replace <EVENTS> with the pipe-joined list from the command above,
# e.g. "Brief Created|Task Completed|Integration Connected|..."
rg -n "trackServerEvent\s*\(\s*['\"](<EVENTS>)['\"]" \
  apps/ packages/ \
  --type ts \
  --glob '!*.spec.ts' --glob '!*.test.ts'
```

This tells you, for every call site in scope, which catalog event it fires. Cross-referencing this against the state-changing actions you find in step 3 is how you identify gaps.

> If `grep` is too noisy on `EventPropertiesMap`, you can alternatively extract names with:
> 
> ```bash
> grep -oP "(?<=: )[A-Z][^:']+" packages/analytics/src/event-types.ts | sort -u
> ```

### 4. Detect state-changing actions

Run each pattern below across the scoped files. The four patterns map to the brief's definition of "state-changing action":

| # | Pattern | Detection (ripgrep) | Where it lives |
| --- | --- | --- | --- |
| 1 | Server action (Next.js) | `rg -n "enhanceAction\(" <files>` | `apps/web/app/**/_lib/server/*-actions.ts`, `apps/web/app/**/_lib/server/server-actions.ts` |
| 2 | BullMQ processor / consumer | `rg -n "@Processor\(\ | extends WorkerHost\ |
| 3 | Domain-entity mutation | `rg -nP "\.from\('(briefs?\ | tasks?\ |
| 4 | Direct write through an admin client | `rg -n "getSupabaseServerAdminClient\(\)" <files>` (then narrow to the surrounding mutation) | Rare; deserves extra scrutiny |

For each match, record:

- **File path + line number**
- **Enclosing function/handler name** (`enhanceAction(...)` factory, NestJS method, etc.)
- **Mutation table** (for pattern 3) or **queue name** (for pattern 2)
- **Action verb** (insert / update / delete / upsert) — this hints at the canonical event

### 5. Find the instrumentation, if any

For each detected action, search the same enclosing function/handler for a tracking call. Search ~50 lines before and after the mutation, not just the line itself — analytics calls are often after the mutation on the success path or in a `try`/`catch` block:

```bash
# Adjust line range to enclosing function boundaries
rg -n "trackServerEvent\(\|tracker\.track\|analytics\.trackEvent\(\|trackHordeEvent\(\|trackConnectionEvent\(\|trackFlushEvent\(\|trackRelayEvent\(\|trackSyncEvent\(" <file>

From gethamster/skills, MIT.

More from gethamster/skills