grafel-graph-quality

Files (10)

• SKILL.md
• 01-question-generation.md · prompts/
• 02-without-mcp-run.md · prompts/
• 03-with-mcp-run.md · prompts/
• 03a-rpc-capture.md · prompts/
• 04-quality-judgment.md · prompts/
• 05-report.md · prompts/
• 06-extraction-calibration.md · prompts/
• 07-delta-mode.md · prompts/
• with-mcp-artifact.schema.json · schema/

SKILL.md (first lines)

# grafel-graph-quality

Pre-docgen MCP quality benchmark. The skill answers a single question with data: **does grafel MCP deliver value over grep+read on this group?** It produces a shareable markdown report the user passes back to the grafel coordinator for tuning.

## When to use this skill

Invoke when the user asks for any of:

- "Benchmark grafel on this group."
- "Is the MCP actually saving tokens here?"
- "Quality-check before we run /generate-docs."
- "Run a regression against the last benchmark."
- "/grafel-quality-check" (slash command).
- "Run as a CI gate after indexing — /grafel-graph-quality --since <sha> -- benchmarks only entities changed since that commit."

Do **not** invoke for one-off lookups, ad-hoc grep substitution, or to "test the daemon" - the daemon health-check is a separate concern.

## MCP + grep, paired

grafel MCP gives you a navigable, accurate map of the code; grep gives you raw pattern matches.
Use MCP for structural questions: who calls X? what is the flow? where does Y live in the graph?
Use grep for raw enumeration: every `if err != nil`, every import line, every TODO.
Pair them: MCP narrows the search space; grep verifies edge-property questions MCP can't answer yet.

This benchmark exists precisely to measure whether grafel MCP earns its place in that pairing:
if it does not outperform grep+read on structural questions, the pair is unbalanced and docgen will
burn tokens for no quality gain. Phases 2 and 3 operationalise the pairing by running each side
independently then comparing outcomes.

## Why this skill exists

A predecessor MCP tool ("Tool A") was empirically found to consume **3-6× more tokens than grep+read** on representative questions, while not improving answer quality. This skill is the gate that confirms grafel does not have the same failure mode. It is the formal step between "we built grafel" and "we have evidence grafel helps."

It runs **before** `/generate-docs` because docgen amplifies whatever bias the MCP has - if the MCP is slower and less accurate than grep on this group, docgen will burn tokens producing worse documentation than a grep-only pipeline would.

## Inputs the skill expects

- A resolved grafel group (the skill calls `grafel_orient (view=me)` first to confirm).
- A running grafel daemon (per the user's `grafel install` setup). **The skill never spawns a daemon.** If `grafel_orient (view=me)` fails, stop and tell the user to start their daemon.
- Optional flags:
  - `--output <path>` - report destination. Default: `~/private/benchmarks/mcp-quality-bench-<YYYY-MM-DD>.md`.
  - `--iterations N` - run each question N times for noise reduction. Default 1. Reports median + stddev when N > 1.
  - `--focus <category>` - only generate questions from one of the nine categories (see Phase 1). Useful for stress-testing a known weak area.
  - `--question-set <path>` - JSON file with a user-curated question set instead of auto-generated. Schema documented in Phase 1.
  - `--baseline <path>` - prior report markdown to diff against. Surfaces deltas in token/quality/speed.
  - `--no-calibration` - skip Phase 6 (the extraction over/under audit). By default calibration always runs.
  - `--since <sha>` — delta mode: restrict Phase 1 question generation and Phase 6 calibration to entities whose source files changed since `<sha>`. Reads `grafel_index_status(since=<sha>)` to get the entity set. Full benchmark phases 2/3/4/5 still run on the restricted question set. See `prompts/07-delta-mode.md` for the complete delta procedure.

## Daemon discipline

This skill **runs against the user's existing daemon**. It does not start, restart, or touch the daemon. The user's `grafel install` setup determines which daemon is in use.

- The agent never sets `GRAFEL_DAEMON_ROOT` or spawns `grafel daemon`.
- The agent never kills any `grafel` process.
- If MCP tool calls fail with "no daemon" errors, the skill stops and asks the user to run their own start command.

## Pass numbering (Phase 1 through Phase 6)

The skill is a strict pipeline. Each phase has a dedicated prompt under `prompts/`. A subagent reads the prompt and follows it; the orchestrator (this skill) tracks progress and gates each phase on the previous one's output.

**Critical ordering: grep-only runs FIRST (Phase 2), MCP runs SECOND (Phase 3).** This prevents cross-run contamination — the grep agent answers before it can see which files or entities MCP found. Each of Phases 2 and 3 must run in a **separate, fresh subagent context**; the orchestrator spawns independent agents so neither phase sees the other's transcript.

| Phase | Prompt | Purpose |
|------|--------|---------|
| 1 | `prompts/01-question-generation.md` | Call `grafel_orient (view=me)`, learn the group's real entities, generate ~10-15 questions across nine categories. Persist as `questions.json`. |
| 2 | `prompts/02-without-mcp-run.md` | Answer every question using **only** `rg` / `grep` / `Read` / `Bash`. No grafel MCP. Record tokens / time / tool calls / confidence per question. Persist as `without-mcp.json`. **Runs FIRST** — fresh context, no MCP results in scope. |
| 3 | `prompts/03-with-mcp-run.md` | Answer the **same** questions using grafel MCP tools only. Same metrics. Persist as `with-mcp.json`. **Runs SECOND** — fresh context that has not seen Phase 2's transcript. See `prompts/03a-rpc-capture.md` for the daemon RPC capture protocol. |
| 3a | `prompts/03a-rpc-capture.md` | Daemon RPC capture protocol: log-offset snapshot procedure, CLI helper invocation, field semantics, error handling. Referenced by Phase 3 per-question metric collection. |
| 4 | `prompts/04-quality-judgment.md` | Determine ground truth by reading source code directly (independent grep+read pass, no MCP). Judge both runs against ground truth: full / partial / wrong / unknown plus per-question misses. Persist as `judgment.json`. |
| 5 | `prompts/05-report.md` | Render the markdown report at `--output`. Tables, findings, issues, recommendations, raw-data appendix. |
| 6 | `prompts/06-extraction-calibration.md` | Evaluate whether grafel is OVER- or UNDER-extracting on this group. Quantify phantom/duplicate nodes, noise, and missing relationships against grep+read ground truth. Persist as `calibration.json` and append an "Extraction calibration" section to the report. |
| 7 | `prompts/07-delta-mode.md` | Delta mode (--since): restrict question generation and calibration to entities changed since a given commit SHA. Skip this phase when --since is not provided. |

Phase 6 runs after Phase 5 (it consumes the same group + ground-truth passes and appends to the report). It is **not** gated on the benchmark questions — it is a structural audit of the graph itself, independent of the head-to-head. Run it whenever the benchmark runs; skip only if the user passes `--no-calibration`.

### Subagent isolation requirement

The orchestrator **must** enforce the following context boundaries:

- **Phase 2 subagent** — spawned with only `questions.json` in scope. Writes `without-mcp.json` and terminates. Must not have access to any MCP tool results.
- **Phase 3 subagent** — spawned after Phase 2 completes, also with only `questions.json` in scope. Must not receive Phase 2's transcript, tool-call log, or any side-channel that reveals which files or entities the grep run visited. Writes `with-mcp.json` and terminates.
- **Phase 4 subagent (judge)** — independent context. Establishes ground truth via its own fresh grep+read pass **before** opening `with-mcp.json` or `without-mcp.json`, so scoring is not anchored by either prior run.

If the orchestrator cannot guarantee full context isolation, it must at minimum instruct the Phase 3 agent explicitly: "Do not read `without-mcp.json`. Answer every question from scratch using only MCP tools."

Each phase reads its predecessor's output from the run directory:

```
~/.grafel/quality-check/<YYYY-MM-DD-HHMMSS>/
  questions.json       # Phase 1
  without-mcp.json     # Phase 2 (grep-only, runs FIRST)
  with-mcp.json        # Phase 3 (MCP, runs SECOND)
  judgment.json        # Phase 4
  report.md            # Phase 5 (also copied to --output)
  calibration.json     # Phase 6 (extraction over/under audit; report section appended)
```

## Question categories

Phase 1 generates questions across nine categories. Each question is adapted to the registered group's actual entities (e.g., "what is UserService?" only if a `UserService`-like entity exists; otherwise substitute a real entity discovered via `grafel_find`).

1. **Entity lookup** - "what is `<ClassName>`?"
2. **Reference finding** - "what calls `<Class>.<method>`?"
3. **Cross-stack tracing** - "how does the frontend `<feature>` flow reach the backend?"
4. **Pattern discovery** - "what's the convention for adding a new `<thing>`?"
5. **Architecture overview** - "what are the main subsystems in this group?"
6. **Subsystem deep-dive** - "describe the `<subsystem>`."
7. **Specific traces** - "trace from `<ui-entity>` to the DB row written."
8. **Data access** - "where does `<Entity>.<field>` get read or written?"
9. **HTTP cross-repo** - "what endpoints does the `<client-repo>` app actually call?"

## Token tracking

The host (Claude Code or compatible) provides `usage_info` per message: `{ input_tokens, output_tokens, cache_read_input_tokens, cache_creation_input_tokens }`. The skill's per-question records must capture these directly from the host's reported usage for each agent message inside that question's scope, not estimate them.

- A question's `input_tokens` = sum of `input_tokens + cache_creation_input_tokens` across all agent messages emitted while answering that question.
- A question's `output_tokens` = sum of `output_tokens`.
- A question's `cache_read_tokens` = sum of `cache_read_input_tokens` (reported separately - cached reads are effectively free, so the report shows total-with-cache and total-without-cache).
- If the host does not surface usage info, the skill **falls back** to char-count / 4 as a rough estimate and clearly labels the report as "estimated tokens, host did not provide usage info".

From cajasmota/grafel, MIT.

More from cajasmota/grafel