root_cause_analysis

Files (13)

• README.md
• SKILL.md
• requirements.txt
• correlation.schema.json · schemas/
• job_context.schema.json · schemas/
• summary.schema.json · schemas/
• __init__.py · scripts/
• cli.py · scripts/
• config.py · scripts/
• correlator.py · scripts/
• job_parser.py · scripts/
• parse_github_paths.py · scripts/
• splunk_client.py · scripts/

SKILL.md (first lines)

# Root Cause Analysis

Investigate failed jobs by correlating Ansible Automation Platform (AAP) job logs with Splunk OCP pod logs and analyzing AgnosticD/AgnosticV configuration to identify root causes.

This version uses Python scripts to parse job metadata and build GitHub investigation paths (Steps 1-4a). Then, use GitHub MCP tools to fetch configuration and workload files from GitHub repositories.

## Automatic Execution

When a user asks to analyze a failed job, execute these steps automatically.

The skill's base path is provided when this skill is invoked. Run scripts relative to this folder.

### Setup (run once per session if .venv doesn't exist)

```bash
# Create virtual environment and install dependencies
python3 -m venv .venv && .venv/bin/pip install -q -r requirements.txt
```

### Step 1-4a: Run the analysis CLI

```bash
# Option 1: By job ID (searches JOB_LOGS_DIR automatically)
.venv/bin/python scripts/cli.py analyze --job-id <JOB_ID>

# Option 2: By explicit path
.venv/bin/python scripts/cli.py analyze --job-log <path-to-job-log>
```

The CLI automatically runs Steps 1-4a:
- **Step 1**: Parse job log → extract identifiers, failed tasks
- **Step 2**: Query Splunk → fetch correlated OCP pod logs
- **Step 3**: Build correlation timeline
- **Step 4a**: Parse GitHub paths → **ONLY generates candidate paths** (does NOT fetch files)

**Output files**:
- `.analysis/<job-id>/step1_job_context.json`
- `.analysis/<job-id>/step2_splunk_logs.json`
- `.analysis/<job-id>/step3_correlation.json`
- `.analysis/<job-id>/step4a_github_paths.json` (candidate paths only - files NOT fetched yet)

### Step 4b-4e: Analyze with AgnosticD/V context (You do this)

**Input**: Read the following files in order (optimized to avoid redundancy):
1. **REQUIRED**: `step1_job_context.json` - Job metadata and failed task details
2. **REQUIRED**: `step4a_github_paths.json` - Parsed GitHub paths and investigation targets
3. **REQUIRED**: `step3_correlation.json` - Correlated timeline with relevant pod logs (DO NOT read step2 unless needed)
4. **CONDITIONAL**: `step2_splunk_logs.json` - Only read if step3 indicates errors needing deeper investigation

Then perform enhanced analysis following these steps:

**Step 4b: Fetch AgnosticV Configuration using github mcp server**

Read `step4a_github_paths.json` and for EACH failed task, iterate through `investigation_targets.configuration.hierarchy`:

For EACH config in the hierarchy:

1. **If config has `path` (single path)**:
   - Use `mcp__github__get_file_contents(owner=config['owner'], repo=config['repo'], path=config['path'])`
   - If 404: Use GitHub MCP search tools (see below)

2. **If config has `paths_to_try` (multiple candidate paths)**:
   - Try ALL paths in the list using `mcp__github__get_file_contents`
   - **CRITICAL**: Don't stop after first 404 - try all variations (underscores vs hyphens, etc.)
   - If ALL paths return 404: Use GitHub MCP search tools

3. **When candidate paths fail, use GitHub MCP search**:
   - **Check parent directory first** (`mcp__github__get_file_contents`): List parent to see actual names/formats (reveals case sensitivity, hyphens vs underscores). Document findings: wrong format → parser bug, missing → truly missing, empty → rare.
   - **If parent check fails**: Use `mcp__github__search_code` to locate files by filename or partial path
   - **No assumptions**: "empty", "test", "demo" in names are just labels - always verify 404s

4. Store fetched content with metadata:
   - Which path worked (for recommendations later)
   - Order number (for precedence analysis)
   - Purpose (base_defaults, platform_config, env_overrides)
   - Any `check_for` hints (secrets, vault)

**Validation Checklist Before Step 4c:**
- [ ] All candidate paths from step4a were attempted (not just the first one)
- [ ] If candidate paths failed, GitHub MCP search tools were used to find actual paths
- [ ] Parent directory was checked first before doing wild searches
- [ ] All fetched configs stored with metadata (order, purpose, working path)

**Step 4c: Fetch AgnosticD Workload Code**

Read `investigation_targets.workload_code` from step4a for each failed task.

For EACH workload file:

1. **If file has `paths_to_try`** (role defaults with extension variations):
   ```
   Try each path until one succeeds
   ```

2. **If file has `path`** (task file - always direct):
   ```
   Fetch directly
   ```

3. Note any `warning` fields (e.g., "Could not extract role name")

Store fetched workload code with:
- File path that worked
- Purpose (role_defaults, failed_task_code)
- Line context information (target_line, context_before, context_after)

**Step 4d: Apply Investigation Rules**

Now you have ALL data:
- step1_job_context.json (basic failure info)
- step2_splunk_logs.json (pod logs, if available)
- step3_correlation.json (timeline)
- Fetched config files (from step 4b)
- Fetched workload files (from step 4c)

Apply these investigation rules:

1. **Variable Precedence Analysis**:
   - Use the config hierarchy ORDER from step4a (order 1 → 2 → 3)

From redhat-et/rhdp-rca-plugin, Apache-2.0.

More from redhat-et/rhdp-rca-plugin