root-cause-analysis

Files (13)

• README.md
• SKILL.md
• requirements.txt
• correlation.schema.json · schemas/
• job_context.schema.json · schemas/
• summary.schema.json · schemas/
• __init__.py · scripts/
• cli.py · scripts/
• config.py · scripts/
• correlator.py · scripts/
• job_parser.py · scripts/
• splunk_client.py · scripts/
• step4_fetch_github.py · scripts/

SKILL.md (first lines)

# Root Cause Analysis

Investigate failed jobs by correlating Ansible Automation Platform (AAP) job logs with Splunk OCP pod logs and analyzing AgnosticD/AgnosticV configuration to identify root causes.

## Automatic Execution

When a user asks to analyze a failed job, execute these steps automatically.
The skill's base path is provided when this skill is invoked. Run scripts relative to this folder.

### Setup (run once per session if .venv doesn't exist)

```bash
# Create virtual environment and install dependencies
python3 -m venv .venv && .venv/bin/pip install -q -r requirements.txt
```

### Step 1-3: Run the analysis CLI

```bash
# Option 1: By job ID (searches JOB_LOGS_DIR automatically)
.venv/bin/python scripts/cli.py analyze --job-id <JOB_ID>

# Option 2: By explicit path
.venv/bin/python scripts/cli.py analyze --job-log <path-to-job-log>
```

The skill automatically searches for job logs in the configured `JOB_LOGS_DIR` (set in `.env`).

### Step 4: Fetch GitHub Data (Automated)

```bash
# Fetch all GitHub files automatically via GitHub API
.venv/bin/python scripts/step4_fetch_github.py --job-id <JOB_ID>
```

This script automatically parses job metadata, fetches AgnosticV configs and AgnosticD workload code, and reports success/404 status.

**Output**: `.analysis/<job-id>/step4_github_fetch_history.json`

**MANDATORY: GitHub MCP Tools for 404 Verification**:
- **ALWAYS check 404 files via MCP server** when step4 reports `"error": "all_paths_failed"` or any `"status": "404"` in `paths_tried`
- **DO NOT skip 404 verification** - the script may have incorrect path parsing
- **Process**:
  1. Check parent directory first using `mcp__github__get_file_contents` to list actual folder/file names (reveals case sensitivity, hyphens vs underscores)
  2. If parent check fails, use `mcp__github__search_code` to locate files by filename or partial path
  3. Document findings: wrong format → parser bug, missing → truly missing, empty → rare
  4. **No assumptions**: "empty", "test", "demo" in names are just labels - always verify 404s

### Step 5: Analyze and Generate Summary

Read outputs from `.analysis/<job-id>/` and perform root cause analysis. See Step 5 section below for detailed requirements.

## Manual Usage

From this skill's directory:

```bash
# Setup virtual environment (one time)
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

# Analyze by job ID (uses JOB_LOGS_DIR to find the file)
.venv/bin/python scripts/cli.py analyze --job-id 1234567

# Or analyze with explicit path
.venv/bin/python scripts/cli.py analyze --job-log /path/to/job_123.json.gz
```

## Configuration

**IMPORTANT**: All credentials (GitHub token and Splunk credentials) should be set in `.claude/settings.json` (project-level) or `~/.claude/settings.json` (global), NOT in `.env` files.

Add the following environment variables to your Claude Code settings file:

**Project-level**: `.claude/settings.local.json` in your project root
**Or global**: `~/.claude/settings.json`

```json
{
  "env": {
    "JOB_LOGS_DIR": "/path/to/your/extracted_logs",
    "GITHUB_TOKEN": "your-github-token",
    "SPLUNK_HOST": "host-url",
    "SPLUNK_USERNAME": "your-username",
    "SPLUNK_PASSWORD": "your-password",
    "SPLUNK_INDEX": "your_aap_index",
    "SPLUNK_OCP_APP_INDEX": "your_ocp_app_index",
    "SPLUNK_OCP_INFRA_INDEX": "your_ocp_infra_index",
    "SPLUNK_VERIFY_SSL": "false"
  }
}
```

---

## Steps 1-3: Automated Analysis (Python Scripts)

**Step 1**: Parse job log → Extract job ID, GUID, namespace, failed tasks, time window
**Step 2**: Query Splunk → Fetch pod logs from namespace within job time window
**Step 3**: Correlate → Merge AAP and Splunk events into unified timeline

**Outputs**: `.analysis/<job-id>/step1_job_context.json`, `step2_splunk_logs.json`, `step3_correlation.json`

---

## Step 4: Fetch GitHub Files (Automated)

Script `step4_fetch_github.py` automatically parses job metadata, fetches AgnosticV configs and AgnosticD workload code, and reports success/404 status.

**Output**: `.analysis/<job-id>/step4_github_fetch_history.json`


**GitHub MCP Verification for 404 Errors**:

When step4 reports `"error": "all_paths_failed"` or any `"status": "404"` in `paths_tried`, you **MUST** verify using MCP tools:

2. **If parent check fails**: Use `mcp__github__search_code` to locate files by filename or partial path. Use `mcp__github__search_repositories` only if task_path parsing failed.

4. **No assumptions**: "empty", "test", "demo" in names are just labels - always verify 404s before concluding files are missing.

From redhat-et/rhdp-rca-plugin, Apache-2.0.

More from redhat-et/rhdp-rca-plugin