system-performance

Files (1)

• SKILL.md

SKILL.md (first lines)

# System Performance Analysis

## Overview

System performance analysis is a methodology for investigating and solving performance problems in computing systems. This skill provides structured approaches based on Brendan Gregg's performance analysis methodologies, including the USE Method, RED Method, flame graphs, and other systematic debugging techniques.

## When to Use

Use this skill when:

- Investigating performance bottlenecks
- Debugging slow systems or applications
- Analyzing CPU, memory, disk, or network performance
- The user mentions "performance analysis", "USE method", "flame graphs", or "Brendan Gregg"
- Conducting performance testing or benchmarking
- Optimizing system or application performance

## Core Methodologies

### 1. USE Method (Utilization, Saturation, Errors)

**For every resource, check utilization, saturation, and errors.**

The USE Method is most effective for finding resource bottlenecks early in a performance investigation.

#### Resource Types

**Physical Resources:**

- CPUs: sockets, cores, hardware threads (virtual CPUs)
- Memory: capacity
- Network interfaces
- Storage devices: I/O, capacity
- Controllers: storage, network cards
- Interconnects: CPUs, memory, I/O

**Software Resources:**

- Mutex locks
- Thread pools
- Process/thread capacity
- File descriptor capacity

#### Metrics

| Resource         | Utilization                              | Saturation                            | Errors                           |
| ---------------- | ---------------------------------------- | ------------------------------------- | -------------------------------- |
| CPU              | CPU utilization (per-CPU or system-wide) | Run-queue length or scheduler latency | Correctable CPU cache ECC events |
| Memory           | Available free memory (system-wide)      | Anonymous paging or thread swapping   | Failed malloc()s                 |
| Network          | RX/TX throughput / max bandwidth         | Dropped packets, overruns             | Interface errors                 |
| Storage I/O      | Device busy percent                      | Wait queue length                     | Device errors                    |
| Storage Capacity | Used vs total capacity                   | Allocation failures                   | Filesystem errors                |

#### Interpretation Guidelines

**Utilization:**

- 100% utilization is usually a sign of a bottleneck
- High utilization (>70%) can begin to be a problem:
  - Long time periods can hide short bursts of 100%
  - Some resources (disks) cannot be interrupted during operations

**Saturation:**

- Any degree of saturation can be a problem (non-zero)
- Measured as queue length or time spent waiting

**Errors:**

- Non-zero error counters are worth investigating
- Especially if they are still increasing while performance is poor

#### USE Method Flowchart

```
Start
  ↓
For each resource:
  ↓
Check errors → If errors found → Investigate
  ↓
Check utilization → If high utilization → Check saturation
  ↓                    ↓
Check saturation → If saturation → Investigate
  ↓
Next resource
```

### 2. RED Method (Rate, Errors, Duration)

**For every service or microservice, check request rate, errors, and duration.**

The RED Method is designed for monitoring services and microservices, focusing on request-driven metrics.

#### Metrics

| Metric   | Description                  | Measurement               |
| -------- | ---------------------------- | ------------------------- |
| Rate     | Requests per second          | Counter over time window  |
| Errors   | Failed requests per second   | Error count / time        |
| Duration | Request latency distribution | Histogram (p50, p95, p99) |

#### Application

For each service:

1. **Rate**: Measure requests per second
2. **Errors**: Count failed requests
3. **Duration**: Track latency percentiles

### 3. TSA Method (Thread State Analysis)

**For each thread of interest, measure time in operating system thread states.**

#### Thread States

- **Executing**: Running on-CPU
- **Runnable**: Ready to run, waiting for CPU
- **Swapping**: Waiting for memory pages

From ntk148v/skills, MIT.

More from ntk148v/skills