taguchi

Files (10)

• SKILL.md
• analyze.py · scripts/
• arrays.py · scripts/
• design.py · scripts/
• experiment.py · scripts/
• run.py · scripts/
• test_analyze.py · scripts/
• test_design.py · scripts/
• test_experiment.py · scripts/
• test_run.py · scripts/

SKILL.md (first lines)

Replace one-factor-at-a-time debugging with a designed experiment. An orthogonal array covers **every pair of factor levels** in a handful of runs, so any behavior caused by one factor or by a two-factor interaction is guaranteed to show up in at least one run. 11 binary factors need 12 runs instead of 2048.

Every run changes several factors at once. That feels wrong to debugging intuition — resist the urge to "change only one thing". The analysis is column-wise (compare all runs where factor X was at level 1 vs level 2), not row-wise, and the balance of the array is what makes that comparison fair.

## Step 0 — gate

Confirm all of these before proceeding; otherwise use the cheaper tool and say so:

- **3+ candidate factors.** One factor → plain bisection. Two factors → just run the 2×2 (4 runs).
- **Runs are expensive.** If a run is seconds, brute-force the full factorial instead.
- **Factors are independently settable.** If setting A=2 forces B=2, merge them into one factor.
- **At most ~11 factors** — that is the 2-level ceiling; 3-level factors cap out at 7 (plus one 2-level) in L18. More than that means the suspect list was never narrowed — shrink it first (the shrink skill in this plugin), then design the array for the survivors.
- **The array has to be meaningfully cheaper than the factorial.** Three 2-level factors are 4 runs against 8 — a 2x saving that does not pay for the ceremony or for confounding interactions. Below roughly 3x, run the factorial and read the interactions directly; `design.py` prints the ratio and says so.

Sibling skills cover the neighboring shapes: an expected single culprit among many boolean toggles → shrink; level counts that fit no array below → pairwise; optimizing knobs rather than hunting a culprit → tune.

## Step 1 — factors and levels

Build the factor table with the user (or from the debugging context). Force every factor to 2 or 3 discrete levels:

- Boolean or on/off → 2 levels.
- Versions → current vs suspected-bad (2 levels), add a third only if a middle version genuinely discriminates.
- Continuous values (timeout, batch size, memory limit) → the two extremes of the plausible range; a midpoint only as a third level.
- A factor nobody can articulate a level for is not a factor — drop it or fix it at its current value.
- A factor you can observe but not set (region, node shape, neighbor load) is a covariate, not a column: record its value for every run and check it during analysis (step 5).

## Step 2 — pick the array

Smallest array that fits all factors:

| Factors | Array | Runs | Full factorial would be |
| --- | --- | --- | --- |
| ≤3 × 2-level | L4 | 4 | 8 |
| ≤7 × 2-level | L8 | 8 | 128 |
| ≤4 × 3-level | L9 | 9 | 81 |
| ≤11 × 2-level | L12 | 12 | 2048 |
| 1 × 2-level + ≤7 × 3-level | L18 | 18 | 4374 |

Mixed 2- and 3-level factors: dummy treatment — a 2-level factor sits in a 3-level column with level 3 repeating level 1 (coverage holds, the repeated level just gets more runs). Its level means stay unbiased but rest on unequal run counts, so its Δ is noisier than the rest of the table — rank a dummy-treated factor against the others only when its effect clears the noise floor by a margin. Up to 4 factors total fit L9 this way at 9 runs; bigger mixes take L18. Extra columns beyond your factor count are simply left unassigned. Level counts that fit none of these shapes (a 4-level factor, uneven mixes) → the pairwise skill generates a covering array for arbitrary levels.

Do not assemble the sheet by hand. `scripts/design.py` picks the array, assigns the columns, applies the dummy treatment and prints the run sheet with concrete values:

```bash
python3 scripts/design.py "os=ubuntu,macos" "node=18,20" "lockfile=frozen,fresh"
python3 scripts/design.py "gc=on,off" "pool=8,16,32" "batch=1,10,100" --json
```

## The arrays

These tables are pairwise-balanced, and `scripts/arrays.py` re-checks that at import — a transposed digit raises instead of silently destroying the balance the analysis relies on. They are printed here to be read, not to be retyped: transcribing an L12 by hand is exactly the step the script exists to remove.

### L4 (3 factors × 2 levels, 4 runs)

```text
run  A B C
1    1 1 1
2    1 2 2
3    2 1 2
4    2 2 1
```

### L8 (7 factors × 2 levels, 8 runs)

```text
run  A B C D E F G
1    1 1 1 1 1 1 1
2    1 1 1 2 2 2 2
3    1 2 2 1 1 2 2
4    1 2 2 2 2 1 1
5    2 1 2 1 2 1 2
6    2 1 2 2 1 2 1
7    2 2 1 1 2 2 1
8    2 2 1 2 1 1 2
```

### L9 (4 factors × 3 levels, 9 runs)

```text
run  A B C D
1    1 1 1 1
2    1 2 2 2
3    1 3 3 3
4    2 1 2 3
5    2 2 3 1
6    2 3 1 2
7    3 1 3 2
8    3 2 1 3
9    3 3 2 1
```

### L12 (11 factors × 2 levels, 12 runs)

```text
run  A B C D E F G H I J K
1    1 1 1 1 1 1 1 1 1 1 1
2    1 1 1 1 1 2 2 2 2 2 2
3    1 1 2 2 2 1 1 1 2 2 2
4    1 2 1 2 2 1 2 2 1 1 2
5    1 2 2 1 2 2 1 2 1 2 1
6    1 2 2 2 1 2 2 1 2 1 1
7    2 1 2 2 1 1 2 2 1 2 1
8    2 1 2 1 2 2 2 1 1 1 2
9    2 1 1 2 2 2 1 2 2 1 1
10   2 2 2 1 1 1 1 2 2 1 2
11   2 2 1 2 1 2 1 1 1 2 2
12   2 2 1 1 2 1 2 1 2 2 1
```

### L18 (1 factor × 2 levels + 7 factors × 3 levels, 18 runs)

```text
run  A B C D E F G H
1    1 1 1 1 1 1 1 1
2    1 1 2 2 2 2 2 2
3    1 1 3 3 3 3 3 3
4    1 2 1 1 2 2 3 3
5    1 2 2 2 3 3 1 1
6    1 2 3 3 1 1 2 2
7    1 3 1 2 1 3 2 3

From lexfrei/ccc, BSD-3-Clause.

More from lexfrei/ccc