debug-e2e-tests

Files (6)

• SKILL.md
• pac-webhook-debugging.md
• prow-reference.md
• reference.md
• download-logs.sh · scripts/
• download-prow-logs.sh · scripts/

SKILL.md (first lines)

# Debug E2E Tests

Debug failed e2e test runs from **GitHub Actions** or **OpenShift CI (Prow)**. Downloads test logs and cluster artifacts, identifies root causes, and suggests fixes to this repo or upstream dependencies.

**When this skill is activated, tell the user:** "Using the **debug-e2e-tests** skill to investigate this failure." Then show the checklist from the Workflow section below so the user can track progress.

## Prerequisites

- `gh` CLI authenticated with access to `konflux-ci/konflux-ci`
- `curl` available (for Prow artifact downloads)
- `tar` and `unzip` available (standard on Linux/macOS)

## Workflow

Copy this checklist and track progress:

```
- [ ] Step 1: Identify the CI system and failed run
- [ ] Step 2: Download artifacts
- [ ] Step 3: Analyze test output (JUnit + Ginkgo logs)
- [ ] Step 4: Analyze cluster logs
- [ ] Step 5: (Optional) Inspect the tested commit
- [ ] Step 6: Determine root cause and suggest fix
```

### Step 1: Identify the CI System and Failed Run

First, determine which CI system the failure is from:

| Signal | CI System |
|--------|-----------|
| User provides a GitHub Actions run ID or PR number | **GitHub Actions** |
| URL contains `prow.ci.openshift.org` | **OpenShift CI (Prow)** |
| URL contains `gcsweb-ci.apps.ci` | **OpenShift CI (Prow)** |
| Job name starts with `pull-ci-` or `periodic-ci-` | **OpenShift CI (Prow)** |
| User mentions "Prow", "OpenShift CI", or "OCP e2e" | **OpenShift CI (Prow)** |

#### GitHub Actions

If the user provides a PR number, run ID, or branch name, use that. Otherwise, find recent failures:

```bash
# List recent failed e2e runs
gh run list --repo konflux-ci/konflux-ci \
  --workflow "Operator E2E Tests" \
  --status failure --limit 10

# For a specific PR
gh run list --repo konflux-ci/konflux-ci \
  --workflow "Operator E2E Tests" \
  --branch <branch> --limit 5
```

Get the **run ID** from the output (first column).

To see which jobs failed within a run:

```bash
gh run view <run-id> --repo konflux-ci/konflux-ci
```

The e2e workflow has these jobs:
- **Run Operator E2E Tests (AMD64)** — main e2e on amd64
- **Run Operator E2E Tests (ARM64)** — main e2e on arm64

#### OpenShift CI (Prow)

The user will typically provide a Prow URL like:
```
https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/konflux-ci_konflux-ci/<PR>/<job-name>/<build-id>
```

Key Prow jobs for this repo:
- `pull-ci-konflux-ci-konflux-ci-main-konflux-e2e-v420-optional` — PR presubmit (OCP 4.20)
- `periodic-ci-konflux-ci-konflux-ci-main-ocp420-konflux-e2e-v420` — periodic x86_64
- `periodic-ci-konflux-ci-konflux-ci-main-ocp420-arm64-konflux-e2e-v420-arm64` — periodic ARM64

You can view recent Prow job runs at:
- [x86_64 periodic](https://prow.ci.openshift.org/?job=periodic-ci-konflux-ci-konflux-ci-main-ocp420-konflux-e2e-v420)
- [ARM64 periodic](https://prow.ci.openshift.org/?job=periodic-ci-konflux-ci-konflux-ci-main-ocp420-arm64-konflux-e2e-v420-arm64)

### Step 2: Download Artifacts

#### GitHub Actions

Use the helper script to download and extract artifacts:

```bash
bash skills/debug-e2e-tests/scripts/download-logs.sh <run-id> [arch]
```

- `arch` defaults to `amd64`. Use `arm64` if that's the failing job.
- Artifacts are extracted to `/tmp/e2e-debug-<run-id>/`

If the script is unavailable, download manually:

```bash
WORKDIR="/tmp/e2e-debug-<run-id>"
mkdir -p "$WORKDIR" && cd "$WORKDIR"

# Download artifacts (name is "logs-amd64" or "logs-arm64")
gh run download <run-id> --repo konflux-ci/konflux-ci \
  --name logs-amd64 --dir "$WORKDIR/logs"
```

Also download the raw job log for the failing job:

```bash
# Get job IDs
gh api "repos/konflux-ci/konflux-ci/actions/runs/<run-id>/jobs" \
  --jq '.jobs[] | select(.conclusion=="failure") | "\(.id) \(.name)"'

# Download the full job log
gh api "repos/konflux-ci/konflux-ci/actions/jobs/<job-id>/logs" > "$WORKDIR/job.log"
```

#### OpenShift CI (Prow)

Use the Prow download script:

From konflux-ci/konflux-ci, Apache-2.0.

More from konflux-ci/konflux-ci