newman-report-analyzer

Files (1)

• SKILL.md

SKILL.md (first lines)

# Newman Report Analyzer

Analyze Newman run output in any format — terminal logs, JSON exports, JUnit XML — to give clear summaries, diagnose failures, spot slow requests, and recommend fixes.

---

## Supported Input Formats

| Format | How to get it |
|--------|--------------|
| Terminal (CLI) output | Copy/paste from terminal |
| JSON export | `--reporter-json-export results.json` |
| JUnit XML | `--reporter-junit-export results.xml` |
| HTMLextra data | User describes or pastes key sections |

Accept whichever format the user provides. Parse all available information.

---

## What to Extract and Report

### 1. Run Summary
Always lead with the high-level outcome:
- Total requests run
- Total assertions (tests) run
- ✅ Passed / ❌ Failed counts
- Total run duration
- Any skipped requests

### 2. Failed Tests
For each failure, report:
- Request name and folder path
- The `pm.test()` description that failed
- Expected vs. actual (if available in output)
- HTTP status code received
- Response time

### 3. Request-Level Errors (non-test failures)
Newman can fail at the request level before tests even run:
- **Connection errors** — ECONNREFUSED, ENOTFOUND, ETIMEDOUT
- **SSL errors** — cert validation failures
- **Redirect issues**
- **Script errors** — syntax errors in pre-request or test scripts

Distinguish these from test assertion failures.

### 4. Slow Requests
Flag any request exceeding a reasonable threshold (default: 2000ms unless user specifies). Report:
- Request name
- Response time
- Whether a `--timeout-request` would have killed it

### 5. Variable/Chaining Issues
Common symptom: a later request fails because a previous one didn't set an environment variable. Look for:
- `undefined` values in request URLs or bodies
- `{{variable}}` not replaced (appears literally in URLs)
- Tests that set variables followed by tests that fail with unexpected nulls

---

## Common Newman Error Patterns & Diagnoses

### `Error: ECONNREFUSED`
**Cause:** The server isn't running or the wrong port/host is configured.
**Fix:** Check `baseUrl` environment variable. Confirm the server is up.

### `Error: ENOTFOUND`
**Cause:** DNS resolution failed — hostname doesn't exist.
**Fix:** Check the URL for typos. Confirm environment is set correctly.

### `Error: connect ETIMEDOUT` / `Request timeout`
**Cause:** Server took too long to respond.
**Fix:** Increase `--timeout-request`, or investigate server-side latency.

### `AssertionError: expected 401 to equal 200`
**Cause:** Auth failed — token missing, expired, or wrong.
**Fix:** Check that the login/auth request ran first and set `{{authToken}}`. Check token expiry.

### `TypeError: Cannot read property 'X' of undefined`
**Cause:** `pm.response.json()` returned something unexpected — often an HTML error page or empty body.
**Fix:** Add a status code check before accessing body properties. Log `pm.response.text()` to see raw response.

### `Error: Request Failed: self signed certificate`
**Cause:** SSL cert is self-signed (common in staging/local).
**Fix:** Add `-k` / `--insecure` flag to Newman command.

### `{{variable}}` appearing literally in URL
**Cause:** The environment variable was never set or the wrong environment file was loaded.
**Fix:** Check `-e` flag points to correct environment. Verify the variable name matches exactly.

### `Script execution error`
**Cause:** Syntax error in a test or pre-request script.
**Fix:** Check the script in the failing request. Common issues: missing semicolons, unclosed brackets, referencing `pm.response.json()` when response has no body.

---

## JSON Report Structure (for reference)

Key fields in a Newman JSON export:
```json
{
  "run": {
    "stats": {
      "requests": { "total": 10, "failed": 2 },
      "assertions": { "total": 25, "failed": 3 }
    },
    "timings": {
      "started": "...",
      "completed": "...",
      "responseAverage": 432
    },
    "executions": [
      {
        "item": { "name": "Get User" },
        "response": { "responseTime": 312, "code": 200 },
        "assertions": [
          {
            "assertion": "Status is 200",
            "error": null

From LambdaTest/agent-skills, MIT.

More from LambdaTest/agent-skills