improve-system-prompt

Files (1)

• SKILL.md

SKILL.md (first lines)

# improve-system-prompt Skill

Benchmark system prompt variants for Sheets Chat by routing test scenarios through the
real GAS-side `ClaudeConversation` pipeline. Compares content variants (V2/V2a/V2b/V2c)
and placement modes (system param vs user message prepend) with dual evaluation:
heuristic scoring (ABTestHarness 8-dim rubric) + LLM-as-judge.

## Product-specific prerequisites

- **Project selector**: the user must provide `--gas-project <id-or-configured-selector>`.
  It is never defaulted from this skill or another checkout.
- **GAS execution**: an authorized GAS execution integration exposing an equivalent of
  `mcp__gas__status`, `mcp__gas__ls`, and `mcp__gas__exec` for that selected project.
  This is a product-specific MCP dependency, not a general marketplace capability.
- **Scenarios**: `require('sheets-chat/ABTestHarness').SCENARIOS` — 12 scenarios total (indices 0–11)
- **Variants**: `require('sheets-chat/SystemPrompt')['buildSystemPromptV2|a|b|c'](null, null, SP.gatherEnvironmentContext())`
- **Variant map**: `{ V2: 'buildSystemPromptV2', V2a: 'buildSystemPromptV2a', V2b: 'buildSystemPromptV2b', V2c: 'buildSystemPromptV2c' }`

This skill is unavailable until the integration can prove authorization and the selected
project exposes the named modules. A static package check or a visible MCP connection is not
evidence that inference or the benchmark completed.

## Argument Reference

Arguments are free-form text after `/improve-system-prompt`. Parse them using the table below:

| Parameter | Default | Format | Notes |
|-----------|---------|--------|-------|
| `--variants` | `V2,V2a` | comma list | Any of: V2 V2a V2b V2c |
| `--scenarios` | `0-9` | range `0-9` or comma `0,1,5` | Indices into 12-scenario array |
| `--placement` | `both` | `system` \| `user` \| `both` | Placement mode to test |
| `--runs` | `1` | integer | Runs per cell (increases averaging) |
| `--gas-project` | — | configured selector or ID | Required target for the authorized GAS integration |
| `--model` | selected project's configured default | configured model ID | Model for GAS-side inference |
| `--judge-model` | selected project's configured default | configured model ID | Model for LLM-as-judge |

**Default matrix**: 2 variants × 2 placements × 10 scenarios × 1 run = **40 cells**

---

## Step 0 — Parse & Config Banner

Parse `$ARGUMENTS` free-form. Extract parameters from the table above. `--gas-project` is
required; if it is absent, stop before any integration call with an example that includes it.

Run an **authorization preflight** before building the matrix:

1. Query the configured GAS integration's status and select only `--gas-project`.
2. Confirm the caller is authorized for the selected project and that its reported identity
   matches the user-supplied selector or ID.
3. Run a non-mutating module check through the selected project to confirm
   `sheets-chat/ABTestHarness`, `sheets-chat/SystemPrompt`, and `chat-core/ClaudeConversation`
   are readable.
4. Confirm any supplied `--model` or `--judge-model` is accepted by that configured project;
   otherwise use only the project's configured default and report that choice.

If any check fails, print the failed prerequisite and stop. Do not substitute another GAS
project, deployment, account, or model.

Emit config table:

```
╔══════════════════════════════════════════╗
║     improve-system-prompt Benchmark      ║
╠══════════════════════════════════════════╣
║  Variants : V2, V2a                      ║
║  Scenarios: 0-9 (10 scenarios)           ║
║  Placement: both (system + user)         ║
║  Cells    : 40                           ║
║  Project  : {gas_project}                ║
║  Model    : {configured_model}           ║
║  Judge    : {configured_judge_model}     ║
╚══════════════════════════════════════════╝
```

---

## Step 1 — Build Matrix

Enumerate all `(contentVariant × placement × scenarioIndex × run)` cells.
If `--scenarios 0-9`: indices 0 through 9 inclusive.
If `--placement both`: expand each variant×scenario into two cells (system + user).

Print matrix summary:

```
Matrix: 2 variants × 2 placements × 10 scenarios × 1 run = 40 cells
  Variants : V2, V2a
  Placement: system, user
  Scenarios: 0 (Fast Path), 1 (Destructive Op), 2 (Batch Fetch), ...
```

---

## Step 2 — Execute Cells (Sliding Window, 3 Parallel)

For each cell, construct an inline JS string and call `mcp__gas__exec`.
Use a sliding window of **3 parallel** `mcp__gas__exec` calls.
Do NOT fire all cells at once — GAS execution quota.

**Per-cell inline JS — placement='system':**

```javascript
(function() {
  var AB = require('sheets-chat/ABTestHarness');
  var scenario = AB.SCENARIOS[<N>];
  var SP = require('sheets-chat/SystemPrompt');
  var promptText = SP['<variantFnName>'](null, null, SP.gatherEnvironmentContext());
  var CC = require('chat-core/ClaudeConversation');
  var claude = new CC(null, '<model>', { system: promptText });
  var result = claude.sendMessage({ messages: [], text: scenario.message, enableThinking: false });
  var ev = AB.evaluateResponse(scenario, result.response || '');
  return {
    scenarioId: scenario.id, category: scenario.category, validates: scenario.validates,
    contentVariant: '<variant>', placement: 'system',
    response: result.response || '', promptLength: promptText.length,
    usage: result.usage || {}, composite: ev.composite, scores: ev.scores
  };
})()

From whichguy/skill-craft, MIT.

More from whichguy/skill-craft