root-cause-analysis

Files (30)

• README.md
• SKILL.md
• github_mcp_verification.md
• requirements.txt
• correlation.schema.json · schemas/
• github_context.schema.json · schemas/
• job_context.schema.json · schemas/
• summary.schema.json · schemas/
• __init__.py · scripts/
• bastion_resolver.py · scripts/
• cli.py · scripts/
• config.py · scripts/
• correlator.py · scripts/
• github_fetcher.py · scripts/
• job_parser.py · scripts/
• jumpbox_io.py · scripts/
• log_fetcher.py · scripts/
• setup.py · scripts/
• splunk_client.py · scripts/
• tracing.py · scripts/
• __init__.py · tests/
• conftest.py · tests/
• correlator_fixtures.json · tests/data/
• sample_job.json · tests/data/
• test_config.py · tests/scripts/
• test_correlator.py · tests/scripts/
• test_fetch_correlator.py · tests/scripts/
• test_job_parser.py · tests/scripts/
• test_splunk_client.py · tests/scripts/
• test_splunk_request.py · tests/scripts/

SKILL.md (first lines)

# Root Cause Analysis

Investigate failed jobs by correlating Ansible Automation Platform (AAP) job logs with Splunk OCP pod logs and analyzing AgnosticD/AgnosticV configuration to identify root causes.

## Automatic Execution

When a user asks to analyze a failed job, execute these steps automatically.
The skill's base path is provided when this skill is invoked. Run scripts relative to this folder.

### Preflight Check (run before first analysis)

```bash
# Create virtual environment and install dependencies (if .venv doesn't exist)
python3 -m venv .venv && .venv/bin/pip install -q -r requirements.txt

# Check all prerequisites (use --json for structured output)
.venv/bin/python scripts/cli.py setup --json
```

Review the JSON output. Some settings are required, others are optional:

**Required** (skill will not proceed without these):
- **JOB_LOGS_DIR** -- Local directory for job log files
- **JUMPBOX_URI** -- SSH jumpbox connection for uploading analysis results and feedback

**Recommended** (analysis works without these but functionality is reduced):
- **MLFlow** -- Tracing configuration for recording analysis runs (MLFLOW_TRACKING_URI, MLFLOW_EXPERIMENT_NAME, MLFLOW_TAG_USER)

**Optional** (skill runs with reduced functionality when missing):
- **SSH / REMOTE_HOST** not configured: `--fetch` flag won't work (user must provide logs in JOB_LOGS_DIR manually)
- **Splunk** not configured: Steps 2-3 (log correlation) will be skipped
- **GitHub token** not configured: Step 4 (config fetching) will be skipped

#### Interactive Setup for Missing Configs

If any checks have `"status": "missing"` and `"configurable": true`, offer to help the user configure them:

1. List the missing configurable items grouped by check name
2. Ask: "Would you like me to help configure these? I'll walk you through each one."
3. If yes, for each missing check with `"configurable": true`:
   - Show the check name and each `env_vars[].prompt` to explain what's needed
   - If the env var has a `"default"`, mention it (user can press enter to accept)
   - If the env var has `"optional": true`, let the user know they can skip it
   - Ask the user for the value
   - **SSH special handling**: If the SSH check has `"ssh_setup_needed": true`:
     - Ask the user for their SSH host alias name
     - Check if that alias already exists in `~/.ssh/config` -- if so, use it as `REMOTE_HOST`
     - If it doesn't exist, ask: do you want to create a new SSH config entry? If yes, ask for: hostname, username, port (default 22), and optional identity file path
     - Read `~/.ssh/config`, append the new `Host` block, and write it back
     - Then set `REMOTE_HOST` to the alias name
4. After collecting all values, read the project's `.claude/settings.json` file
5. Merge the new values into the `"env"` block (create it if it doesn't exist)
6. If MLflow env vars were configured (MLFLOW_TRACKING_URI, MLFLOW_EXPERIMENT_NAME), also add the required MLflow hooks to the `"hooks"` block (create it if it doesn't exist):
   ```json
   "hooks": {
     "SessionStart": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "INPUT=$(cat); SESSION_ID=$(echo \"$INPUT\" | jq -r '.session_id'); echo \"export CLAUDE_SESSION_ID='$SESSION_ID'\" >> \"$CLAUDE_ENV_FILE\""
          },
          {
            "type": "command",
            "command": "if [ \"$MLFLOW_CLAUDE_TRACING_ENABLED\" = \"true\" ]; then if ! pip show mlflow >/dev/null 2>&1; then pip install mlflow; fi; fi"
          }
        ]
      }
    ],
    "Stop": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "python -c \"from mlflow.claude_code.hooks import stop_hook_handler; stop_hook_handler()\""
          }
        ]
      }
    ]
   }
   ```
7. Write the updated settings file
8. Tell the user to **restart the Claude Code session** for env vars and hooks to take effect
9. **Important**: Write secrets (tokens, passwords) to `.claude/settings.json` -- ensure this file is in `.gitignore`

If checks show non-configurable errors (e.g., venv issues, rsync not found), provide the fix command instead.

#### MLFlow Server Startup

The `MLFlow server` preflight check automatically handles server connectivity:
- If the server is unreachable and `JUMPBOX_URI` is configured, it starts an SSH tunnel automatically
- If the tunnel is already running, it skips startup
- If the tunnel fails, it reports the error but the skill can still proceed (MLFlow is recommended, not required)

If any **required** checks (JOB_LOGS_DIR, JUMPBOX_URI) are still missing after the setup flow, do **not** proceed to analysis -- tell the user what's still needed. If MLFlow is missing, warn that tracing won't be recorded but proceed. If all required checks pass (recommended/optional items may remain missing), proceed to analysis.

### Step 1-4: Run the analysis CLI

Always use `--fetch` when analyzing by job ID. This automatically downloads the log from the remote server if it's not already present locally, and skips fetching if the log is already there.

```bash
# By job ID (auto-fetches log from remote if not found locally)
.venv/bin/python scripts/cli.py analyze --job-id <JOB_ID> --fetch

# By explicit path (when you already have the log file)
.venv/bin/python scripts/cli.py analyze --job-log <path-to-job-log>
```

The `cli.py analyze` command automatically runs all steps:

- **Step 1**: Parse job log → Extract job ID, GUID, namespace, failed tasks, time window
- **Step 2**: Query Splunk → Fetch pod logs from namespace within job time window
- **Step 3**: Correlate → Merge AAP and Splunk events into unified timeline
- **Step 4**: Fetch GitHub files → Parse job metadata, fetch AgnosticV configs and AgnosticD workload code (requires `GITHUB_TOKEN` to be configured)

**Outputs**: `.analysis/<job-id>/step1_job_context.json`, `step2_splunk_logs.json`, `step3_correlation.json`, `step4_github_fetch_history.json`

This skill automatically searches for job logs in the configured `JOB_LOGS_DIR`.

From redhat-et/rhdp-rca-plugin, Apache-2.0.

More from redhat-et/rhdp-rca-plugin