root-cause-analysis

Files (11)

• README.md
• SKILL.md
• correlation.schema.json · schemas/
• job_context.schema.json · schemas/
• summary.schema.json · schemas/
• __init__.py · scripts/
• cli.py · scripts/
• config.py · scripts/
• correlator.py · scripts/
• job_parser.py · scripts/
• splunk_client.py · scripts/

SKILL.md (first lines)

# Root Cause Analysis

Investigate failed jobs by correlating Ansible Automation Platform (AAP) job logs with Splunk OCP pod logs and analyzing AgnosticD/AgnosticV configuration to identify root causes.
## Automatic Execution

When a user asks to analyze a failed job, execute these steps automatically.

The skill's base path is provided when this skill is invoked. Run scripts relative to this folder.

### Setup (run once per session if .venv doesn't exist)

```bash
# Create virtual environment and install dependencies
python3 -m venv .venv && .venv/bin/pip install -q -r requirements.txt
```

### Step 1-3: Run the analysis CLI

```bash
# Option 1: By job ID (searches JOB_LOGS_DIR automatically)
.venv/bin/python scripts/cli.py analyze --job-id <JOB_ID>

# Option 2: By explicit path
.venv/bin/python scripts/cli.py analyze --job-log <path-to-job-log>
```

The skill automatically searches for job logs in the configured `JOB_LOGS_DIR` (set in `.env`).

### Step 4: Read outputs and analyze with AgnosticD/V context

After running the CLI, read the generated files from the `.analysis/` folder in this skill directory:

```
.analysis/<job-id>/step1_job_context.json
.analysis/<job-id>/step2_splunk_logs.json
.analysis/<job-id>/step3_correlation.json
```

Then perform enhanced analysis:
1. Parse job metadata to identify platform/demo/env from job_name pattern
2. Fetch AgnosticV configuration hierarchy from rhpds/agnosticv
3. Fetch AgnosticD workload code from redhat-cop/agnosticd
4. Apply investigation rules to identify root cause
5. Generate summary with actionable recommendations

Provide a summary to the user with:
1. **Job Details**: ID, status, GUID, namespace, platform, demo
2. **Failed Task(s)**: **IMPORTANT** - Preserve ALL fields from step1 failed_tasks:
   - Task: task name
   - Play: play name
   - Role: role name
   - Task Action: task_action (Ansible module)
   - Error: error_message
   - Task Duration: duration seconds
   - Location: Both original task_path AND derived GitHub path
     * Original: `/home/runner/.ansible/collections/...` (from task_path field)
     * GitHub: `repository:file_path:line` (parsed from original)
     * Example: `agnosticd/core_workloads:roles/ocp4_workload_gitops_bootstrap/tasks/workload.yml:74`
3. **Configuration Context**: Relevant variables, missing vars, secrets references
4. **Correlation**: How AAP logs link to Splunk pod logs (GUID, namespace, time overlap)
5. **Correlated Pods**: Pods found in Splunk during the job window
6. **Root Cause**: Analysis of why the job failed (configuration vs. infrastructure vs. workload bug)
7. **Recommendations**: Specific file changes with paths and suggested values
---

## Manual Usage

From this skill's directory:

```bash
# Setup virtual environment (one time)
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

# Analyze by job ID (uses JOB_LOGS_DIR to find the file)
.venv/bin/python scripts/cli.py analyze --job-id 1234567

# Or analyze with explicit path
.venv/bin/python scripts/cli.py analyze --job-log /path/to/job_123.json.gz
```

## Configuration

Create a `.env` file (see `.env.example`):

```bash
# Default directory to search for job log files
# The skill will look for files matching job_<ID>.* in this directory
JOB_LOGS_DIR=/path/to/extracted_logs

```

---

## Step 1: Parse Local Job Log

The script extracts key identifiers from the local AAP job log:

- **Job ID**: The AAP job number
- **GUID**: Unique deployment identifier
- **Namespace**: OCP namespace
- **Time Window**: Job start/end times for Splunk query filtering
- **Failed Tasks**: Tasks that failed with error messages
- **Pod References**: Any pod names mentioned in the job output

**Output**: `.analysis/<job-id>/step1_job_context.json`

---

## Step 2: Query Splunk for Pod Logs

Using the identifiers from Step 1, query Splunk for:

1. **OCP Application Logs**: Pod logs from the namespace
2. **Error Events**: Filtered for errors, failures, exceptions
3. **Time-Bounded**: Only logs within the job execution window

**Queries used**:
```spl

From redhat-et/rhdp-rca-plugin, Apache-2.0.

More from redhat-et/rhdp-rca-plugin