performance-benchmarking

Files (4)

• SKILL.md
• benchmark-template.R · references/
• benchmark-template.sh · references/
• speedup-metrics.md · references/

SKILL.md (first lines)

# Performance Benchmarking

Design and execute reproducible benchmarks to measure scaling, compare implementations, and analyze speedup.

## Quick Start

Test a function's scaling with parameter n:

```bash
# Save this as bench.R
benchmark_test <- function(n) {
  # Setup test data
  test_data <- seq(1, n)
  
  # Warm-up (1 run, don't time)
  sum(test_data)
  
  # Timed runs (3 runs, report mean)
  times <- c()
  for (i in 1:3) {
    t <- system.time(sum(test_data))[3]
    times <- c(times, t)
  }
  
  mean(times)
}

# Run for scaling parameter
for (n in c(1000, 10000, 100000, 1000000)) {
  t <- benchmark_test(n)
  cat(sprintf("n=%d: %.4f s\n", n, t))
}
```

Run: `Rscript bench.R`

## Benchmark Anatomy

Every reproducible benchmark has these components:

### Warm-up Run
Run the function once before timing to stabilize JIT, caches, and memory allocation. Don't measure this run -- just let the system settle.

### Multiple Timed Runs
Execute the function >=3 times per configuration. Record all times, report mean and standard deviation. Single-run benchmarks are noise.

### Timeout Protection
Skip configurations that exceed 60 seconds per run. This prevents expensive branches (e.g., O(n^2) algorithms on master) from running excessively long tests.

```r
mean_time <- mean(times)
if (mean_time > 60) {
  cat(sprintf("TIMEOUT: skipping n=%d (%.1f s/run)\n", n, mean_time))
  next
}
```

### CSV Output
Save results to CSV for comparison across runs and branches:

```r
results <- data.frame(
  param = param_values,
  time_sec = timing_results
)
write.csv(results, "benchmark_results.csv", row.names = FALSE)
```

### Visualization
Generate plots for scaling analysis. Log-log plots reveal asymptotic behavior (O(n), O(n^2), etc.).

## Example 1: Hash Map Benchmarks

**Functions**: `classify()`, `subst()` -- value lookup and substitution **Parameter**: Lookup table size (n_lookup) **Expected scaling**: O(1) average with hash map, O(n^2) with nested loops

From archive `benchmark_hash.R` pattern:

```r
n_lookup_values <- c(10, 50, 100, 500, 1000, 5000, 10000)
results <- data.frame()

for (n_lookup in n_lookup_values) {
  # Generate test data
  r <- terra::rast(nrows=100, ncols=100, vals=sample(1:100, 10000, replace=TRUE))
  from_vals <- sample(1:100, n_lookup, replace=FALSE)
  to_vals <- seq(1000, 1000 + n_lookup - 1)
  
  # Warm-up
  terra::classify(r, cbind(from_vals, to_vals))
  
  # Timed runs
  times <- c()
  for (i in 1:3) {
    t <- system.time(terra::classify(r, cbind(from_vals, to_vals)))[3]
    if (t > 60) break  # Timeout check
    times <- c(times, t)
  }
  
  if (length(times) > 0) {
    results <- rbind(results, data.frame(
      n_lookup = n_lookup,
      mean_time = mean(times),
      sd_time = sd(times)
    ))
  }
}

# Plot log-log
png("scaling_plot.png", width=800, height=600)
plot(log(results$n_lookup), log(results$mean_time),
     main="Hash Map Scaling",
     xlab="log(n_lookup)", ylab="log(time_sec)",
     pch=19, cex=1.5)
dev.off()
```

**Result interpretation**: Flat slope = O(1), 45 deg slope = O(n), steep slope = O(n^2).

## Example 2: Other Function Benchmarks

From AndrewBrown-USDA/aigb-skills-usda, MIT.

More from AndrewBrown-USDA/aigb-skills-usda