triage-failure

Files (1)

• SKILL.md

SKILL.md (first lines)

# Triage Failure

Investigate a failed benchmark task and produce a diagnosis.

## Input

User provides one of:
- Task path: `runs/official/pytorch_opus_.../baseline/.../sgt-005__hash/`
- Suite/config/task: `csb_sdlc_pytorch/baseline/sgt-005`
- Just a task name: `sgt-005` (will search for it)
- Or says "triage the most recent failure"

## Steps

### 1. Locate the task directory

If user gave a full path, use it directly. Otherwise, find it:

```bash
cd ~/CodeScaleBench

# If "most recent failure" — get failures from aggregate_status.py
python3 scripts/aggregate_status.py --failures-only --format json | python3 -c "
import sys, json
data = json.load(sys.stdin)
tasks = [t for t in data['tasks'] if t['status'] in ('errored', 'completed_fail', 'timeout')]
tasks.sort(key=lambda t: t.get('dir_mtime', ''), reverse=True)
if tasks:
    t = tasks[0]
    print(f'Task: {t[\"task_name\"]}')
    print(f'Suite: {t[\"suite\"]}')
    print(f'Config: {t[\"config\"]}')
    print(f'Status: {t[\"status\"]}')
    print(f'Dir: {t[\"task_dir\"]}')
else:
    print('No failures found')
"
```

If user gave a task name, search for it:
```bash
# Find all dirs matching the task name
find runs/official -type d -name "*TASKNAME*" | head -10
```

### 2. Read key files in order

Read the files in the task directory to understand what happened:

```bash
TASK_DIR="<resolved path>"

# 1. result.json — primary source of truth
cat "$TASK_DIR/result.json" | python3 -m json.tool

# 2. status.json if it exists (from aggregate_status --write-status)
cat "$TASK_DIR/status.json" 2>/dev/null | python3 -m json.tool

# 3. Agent transcript (last 100 lines)
tail -100 "$TASK_DIR/agent/claude-code.txt" 2>/dev/null

# 4. Verifier output
cat "$TASK_DIR/verifier/test-stdout.txt" 2>/dev/null | tail -50
cat "$TASK_DIR/verifier/reward.txt" 2>/dev/null

# 5. Task log
cat "$TASK_DIR/../$(basename $TASK_DIR | sed 's/__.*//').log" 2>/dev/null | tail -50
```

### 3. Run error fingerprinting

```bash
python3 scripts/status_fingerprints.py "$TASK_DIR/result.json"
```

### 4. Classify the failure

Categorize into one of:
- **infrastructure**: API errors, rate limits, token refresh, network, Docker issues
- **timeout**: Task exceeded time limit (check if agent was stuck or task is inherently slow)
- **verifier_bug**: Error in the verifier/scorer script (KeyError, parse error, etc.)
- **task_setup**: Missing dependencies, git clone failures, Docker compose issues
- **agent_bug**: Agent produced invalid output format or took wrong approach
- **task_difficulty**: Agent tried but task is genuinely hard (completed_fail with effort)
- **flaky**: Passed in other configs or previous runs, failed now with no clear cause
- **mcp_related**: Failure specific to MCP-enabled configs

### 5. Check for known patterns

```bash
# Check error catalog
cat ~/CodeScaleBench/docs/ERROR_CATALOG.md
```

Cross-reference the error with known patterns in the catalog.

### 6. Produce diagnosis

Present findings in this format:

```
## Diagnosis: <task_name>

**Classification**: <category from step 4>
**Status**: errored / completed_fail / timeout
**Error**: <one-line summary of what went wrong>
**Root Cause**: <explanation of why it failed>
**Known Pattern?**: Yes (see ERROR_CATALOG.md#section) / No

**Evidence**:
- result.json exception: <type + message>
- Agent transcript: <relevant excerpt>
- Verifier output: <relevant excerpt>

**Suggested Fix**:
- File: <path to file that needs changing>
- Change: <description of the fix>

**Repro Command**:

From sourcegraph/CodeScaleBench, Apache-2.0.

More from sourcegraph/CodeScaleBench