diagnose-refusal

Files (8)

• SKILL.md
• evals.json · evals/
• trigger-evals.json · evals/
• refusal-classes.md · references/
• report-template.md · references/
• collect-environment.sh · scripts/
• init-bundle.sh · scripts/
• redact-secrets.py · scripts/

SKILL.md (first lines)

# Diagnose Refusal

**Diagnostics only.** Isolate *where* a refusal originated in the request path. Do not
retry, reinterpret, weaken, jailbreak, or continue the refused task.

## Hard invariants

1. Do **not** retry the refused operation.
2. Do **not** bypass, weaken, reinterpret, jailbreak, or circumvent any safety policy.
3. Do **not** continue the task that triggered the refusal.
4. Do **not** invent invisible instructions — write `Not visible from this session.`
5. Do **not** expose hidden reasoning or private chain-of-thought.
6. Do **not** copy credential files (`.env`, `.env.*`, `credentials.json`, `*.pem`, `*.key`).
7. Do **not** modify application source or configuration.
8. Redact secrets: replace API keys, tokens, cookies, passwords, Authorization headers,
   and private keys with `***REDACTED***`. For env vars, record **names and SET/UNSET
   only** — never values.

If the current model is itself refusing, this skill still does not ask it to continue —
it switches the job entirely to benign inspection of config, routing, and the refusal text.

## Why this skill exists

A refusal sentence alone is almost useless. The same surface text can come from the
harness, a project skill, a LiteLLM guardrail, a mis-route, an upstream provider filter,
or the model. Differential diagnosis needs a sanitized bundle and a reconstructed path.

## Bundle layout (v2)

Create or replace under the repo root (or cwd if not a git repo):

```text
.refusal-debug/
├── report.md          # full diagnosis (sections 1–9)
├── request.json       # sanitized request metadata if available; else {"status":"unavailable"}
├── response.json      # sanitized response/error metadata if available; else {"status":"unavailable"}
├── routing.md         # HARNESS → LITELLM → PROVIDER → MODEL
├── environment.md     # safe env / tool versions
└── manifest.txt       # list of files created
```

Initialize with the bundled script, then fill the files:

```bash
bash skills/debugging/diagnose-refusal/scripts/init-bundle.sh
bash skills/debugging/diagnose-refusal/scripts/collect-environment.sh > .refusal-debug/environment.md
```

Resolve the skill directory from whatever path the harness loaded (plugin root,
checkout, or symlink). Prefer `git rev-parse --show-toplevel` then
`skills/debugging/diagnose-refusal/scripts/…`.

After writing any file that may contain pasted error bodies, run:

```bash
python3 skills/debugging/diagnose-refusal/scripts/redact-secrets.py .refusal-debug/report.md
python3 skills/debugging/diagnose-refusal/scripts/redact-secrets.py .refusal-debug/request.json
python3 skills/debugging/diagnose-refusal/scripts/redact-secrets.py .refusal-debug/response.json
```

Classification taxonomy: [references/refusal-classes.md](references/refusal-classes.md).
Report skeleton: [references/report-template.md](references/report-template.md).

## Workflow

Copy and track:

```text
Diagnose-refusal progress:
- [ ] 0. Init bundle + hard-invariant stance
- [ ] 1. Incident capture
- [ ] 2. Conversation state
- [ ] 3. Active repository instructions
- [ ] 4. LiteLLM / proxy path
- [ ] 5. Environment (script)
- [ ] 6. Refusal classification
- [ ] 7. Request-path reconstruction → routing.md
- [ ] 8. Comparison tests (propose only — do not execute refused task)
- [ ] 9. Final assessment (top of report.md)
- [ ] 10. Bundle manifest + completion reply
```

### 0. Init

```bash
bash skills/debugging/diagnose-refusal/scripts/init-bundle.sh
```

Incorporate any user notes from `$ARGUMENTS` / the slash-command argument into section 1.

### 1. Incident → `report.md`

Record:

- current date/time (UTC)
- active model name if visible
- provider/backend if visible
- agent/harness name and version if available
- the user's immediately preceding request that resulted in refusal
- the complete refusal response (redacted)
- whether the refusal appeared before or after any tool call
- whether an HTTP/API error was visible
- status / error / request IDs if available

Populate `request.json` / `response.json` when structured metadata exists (status code,
request id, model id, error type). Otherwise write `{"status":"unavailable","reason":"…"}`.

### 2. Conversation state

Summarize the relevant conversation immediately before the refusal. Include the exact
relevant user message where possible.

Identify relevant:

- system/project instructions visible to you
- `AGENTS.md` / `CLAUDE.md` instructions
- activated skills
- slash commands
- MCP tools

From Tamircohen28/tamirs-superpowers, MIT.

More from Tamircohen28/tamirs-superpowers