rca-annotator

Files (4)

• SKILL.md
• schema.json · schemas/
• cli.py · scripts/
• jumpbox_io.py · scripts/

SKILL.md (first lines)

# RCA Annotator

A structured annotation tool that presents the `root-cause-analysis` agent's diagnosis to the user and guides them through labeling it — capturing whether the diagnosis is correct, evidence quality, difficulty, and alternative hypotheses.

| | `root-cause-analysis` (Agent) | `rca-annotator` (Annotation tool) |
|---|---|---|
| **Purpose** | Diagnose failures for users | Capture human-labeled ground-truth data |
| **Reads** | Logs, Splunk, GitHub (live) | Step 1/3/4/5 output files (offline) |
| **Output** | Human-readable diagnosis | Structured `annotation.json` |

**Use when**: a `root-cause-analysis` run is complete and you want to annotate its output as correct, incorrect, or partially correct — for evaluation, benchmarking, or dataset building.

**Do NOT use** to perform initial RCA (use `root-cause-analysis`).

## Prerequisites

Verify root-cause-analysis has been completed.

- `JUMPBOX_URI` (optional) — SSH connection string (e.g. `"user@host -p 2222"`). If unset, uses local `.analysis/` only.
- SSH keys configured in `~/.ssh/config` if using jumpbox; `ssh` and `rsync` installed.
- **Required files** in `.analysis/<job_id>/`:
  - `step5_summary.json` — Agent's final diagnosis (primary input)
  - `step1_job_context.json` — Job metadata, failed tasks
  - `step3_correlation.json` — Timeline with AAP + Splunk events
  - `step4_github_fetch_history.json` — Configuration and code context

If missing, run `root-cause-analysis` skill first.

## Workflow

0. Download from jumpbox (if `JUMPBOX_URI` set) or verify local files
1. Read `step5_summary.json` — present the agent's diagnosis to the user
2. Walk through annotation questions interactively — the user labels each section
3. Write `annotation.json` with the user's labels
4. Upload to jumpbox (if `JUMPBOX_URI` set)

---

## Step 0: Download Analysis Files

```bash
cd skills/rca-annotator
python scripts/cli.py download --job-id <job_id>
```

Downloads from jumpbox `/usr/local/mlflow/<job_id>/` to local `.analysis/<job_id>/`. If `JUMPBOX_URI` unset, validates local files only. Errors on missing remote directory, missing required files, or connection failure.

---

## Step 1: Read Agent Diagnosis

Read `step5_summary.json` and present the agent's diagnosis clearly to the user:

- Root cause category and summary
- Confidence level
- Key evidence cited
- Difficulty score (if present)
- Recommendations
- Alternative diagnoses (if any)

This is the starting point for annotation. The user is reviewing the agent's work.

---

## Step 2: Interactive Annotation

Walk through each question below with the user. Present the relevant section from `step5_summary.json` before asking each question. Wait for the user's response before continuing.

### 1. Root Cause Category

Present the agent's category and summary. Ask:

> **Is the root cause category correct?** *(e.g. `configuration`, `infrastructure`, `credential` — or should it be something else?)*

Valid categories: `configuration` | `infrastructure` | `application_bug` | `dependency` | `network` | `resource` | `cloud_api` | `credential` | `secrets` | `unknown`

### 2. Summary Accuracy

Present the agent's summary sentence. Ask:

> **Is the summary accurate and specific?** *(Does it clearly describe what failed and why?)*

### 3. Evidence

Present the evidence items the agent cited. Ask:

> **Is any evidence missing or wrong?** *(Any key log lines, config values, or Splunk events that were overlooked or incorrectly cited?)*

If the user wants to cross-check, read step1/step3/step4 and compare against what the agent cited. This is reference material for validation — not a re-analysis.

**Evidence traceability format** (for any new or corrected evidence items the user provides):

```json
{
  "source": "step1 | step3 | step4",
  "source_file": ".analysis/<job_id>/step1_job_context.json",
  "json_path": "failed_tasks[0].duration",
  "exact_value": 917.565567,
  "exact_quote": "optional — literal text for code/config",
  "line_number": 5,
  "github_path": "owner/repo:path/to/file.yml:line",
  "message": "The relevant log line or config snippet.",
  "confidence": "high | medium | low",
  "is_root_cause": true
}
```

### 4. Difficulty Rating

Present the agent's difficulty score (or estimate one from the evidence). Present the calibration rubric to help the user score:

| Criterion | Points |
|---|---|
| Requires cross-source correlation (AAP + Splunk + GitHub) | +3 |
| Requires understanding code behavior | +2 |
| Error message is generic or misleading | +2 |
| Requires variable precedence/override knowledge | +1 |
| Requires domain knowledge (K8s, Ansible, cloud APIs) | +1 |
| Multiple plausible alternatives exist | +1 |

From redhat-et/rhdp-rca-plugin, Apache-2.0.

More from redhat-et/rhdp-rca-plugin