benchmark-and-mms-planner

Files (5)

• CHANGELOG.md
• SKILL.md
• evals.json · evals/
• vv_patterns.md · references/
• benchmark_mms_planner.py · scripts/

SKILL.md (first lines)

# Benchmark And MMS Planner

## Goal

Design a verification and validation plan before trusting simulation results. The skill helps agents choose manufactured solutions, benchmark cases, refinement protocols, uncertainty checks, and pass/fail criteria.

## Requirements

- Python 3.10+
- No external dependencies
- Works on Linux, macOS, and Windows

## Inputs to Gather

| Input | Description | Example |
|-------|-------------|---------|
| PDE or model class | Governing family | `diffusion`, `elasticity`, `phase-field` |
| Quantity of interest | Metric to validate | `interface velocity`, `L2 temperature error` |
| Dimension | 1, 2, or 3 | `2` |
| Expected order | Formal discretization order | `2` |
| Reference availability | Analytic, benchmark, or none | `analytic` |
| Risk level | Cost or consequence of wrong result | `high` |

## Decision Guidance

- Use **MMS** when code correctness is uncertain and an analytic solution can be injected.
- Use **canonical benchmarks** when physical model validation matters more than code verification.
- Use **grid/time refinement** whenever the result is used for a claim, design decision, or comparison.
- Use **uncertainty propagation** when inputs are calibrated, noisy, or experimentally measured.

## Script Outputs

`scripts/benchmark_mms_planner.py` emits `inputs` and `results` with:

- `verification_strategy`
- `effective_model` — the resolved model family actually used; unknown families fall back to `general`.
- `mms_plan`
- `benchmark_cases`
- `refinement_protocol` (`dimension`, `levels`, `spacing_ratio`, `expected_order`, `accept_observed_order_min`, `include_time_refinement`)
- `uncertainty_plan` (`propagate_inputs`, `report_error_bars`, `separate_discretization_and_model_error`) — propagation/error-bar guidance driven by risk level and reference type.
- `acceptance_criteria`
- `warnings`

The `accept_observed_order_min` is an **engineering screening heuristic**, not a certified bound: it is the formal `expected_order` reduced by a fractional tolerance (10% for high risk, 20% otherwise) and floored at first-order convergence (`1.0`). The relative band keeps strictness consistent across formal orders. See `references/vv_patterns.md`.

## Workflow

1. Collect the governing model, quantity of interest, and risk level.
2. Run `benchmark_mms_planner.py --json`.
3. Treat warnings as blockers for high-risk claims.
4. Convert the returned protocol into tests, simulation runs, or review checklist items.

```bash
python3 skills/verification-validation/benchmark-and-mms-planner/scripts/benchmark_mms_planner.py \
  --model diffusion \
  --quantity "L2 error in temperature" \
  --dimension 2 \
  --expected-order 2 \
  --reference analytic \
  --risk high \
  --json
```

## Error Handling

- If the dimension or expected order is invalid, stop and correct the model description.
- If no reference exists, use conservation and convergence checks but do not call the result validated.

## Limitations

This skill plans verification work; it does not run the solver or prove that a physical model is appropriate for an experiment.

## Verification checklist

Before trusting a result that used this planner, record concrete evidence for each item:

- [ ] Ran `benchmark_mms_planner.py --json` and saved the `inputs` block, confirming the echoed `dimension`, `expected_order`, `reference`, and `risk` match the actual run (a fallback to `effective_model: general` was intentional, not a typo in `--model`).
- [ ] Executed the `refinement_protocol`: used the reported `levels` (3, or 4 for high risk) of systematically refined grids at `spacing_ratio` 2, and recorded the observed order of accuracy from those runs.
- [ ] Confirmed the observed order is at or above `accept_observed_order_min`; if below, logged the investigation (mesh not yet asymptotic, boundary/source errors, limiter activation) rather than treating the result as passed.
- [ ] When `include_time_refinement` is `true`, ran a separate time-step refinement study and recorded the temporal observed order, not just the spatial one.
- [ ] When `mms_plan.manufacture_solution` is `true`, derived the symbolic source/forcing term, applied the matching boundary terms, and recorded the L2 and Linf error norms versus the manufactured solution.
- [ ] Checked every `acceptance_criteria` item with a number: conservation/balance closes within a documented tolerance, the quantity of interest plateaus under refinement, and any benchmark discrepancy from `benchmark_cases` is explained before production use.
- [ ] Treated all entries in `warnings` as blockers for high-risk claims and recorded how each was resolved (e.g. an independent analytic/published reference was added when `reference` was `none` or `experimental`).

## Common pitfalls & rationalizations

| Tempting shortcut | Why it's wrong / what to do |
|-------------------|-----------------------------|
| "The planner ran and printed a plan, so the result is verified." | The script only *plans* V&V; it never runs the solver. Verification comes from executing the `refinement_protocol`, MMS, and `acceptance_criteria`, not from generating the plan. |
| "Two grids converged, so the observed order is fine." | `refinement_protocol.levels` is 3 (4 for high risk) for a reason: you need >=3 systematically refined grids to estimate observed order and confirm the solution is in the asymptotic range before quoting it. |
| "Observed order beats `accept_observed_order_min`, so it's certified." | That threshold is an engineering *screening* heuristic (formal order minus a 10%/20% relative tolerance, floored at 1.0), not a certified bound. For rigorous order verification run a Richardson/GCI study. |
| "Steady-looking model, so I can skip `include_time_refinement`." | If the planner set `include_time_refinement: true` (any time-dependent or `general` fallback family), spatial refinement alone hides temporal error — run the time-step study too. |
| "We matched a benchmark, so the code is validated." | Matching `benchmark_cases` or converging shows the code approaches *some* solution; it does not prove the physical model is correct. Validation needs an independent reference plus model-error separation, not convergence alone. |
| "`reference none` is fine, the runs look physical." | With `reference: none` the strategy is verification-only; `warnings` says so. You may report convergence and conservation but must NOT call the result validated. |
| "Unknown model name, so I'll ignore the `general` fallback." | An unrecognized `--model` silently resolves to `effective_model: general`; confirm that fallback is intended, since it changes both `benchmark_cases` and the time-refinement decision. |

## Security

### Input Validation

All inputs are command-line arguments parsed by `argparse`; validation happens in `plan_vv` (and partly in the parser). Any rejected input causes the script to print the error to stderr and exit with code `2`.

- `dimension` must be exactly `1`, `2`, or `3`; any other integer is rejected.
- `expected_order` must be a positive, finite number (NaN, infinity, zero, and negatives are rejected).
- `risk` must be one of the allowlist `low`, `medium`, `high` (enforced both as an `argparse` choice and re-checked in `plan_vv`).
- `reference` must be one of the allowlist `analytic`, `benchmark`, `experimental`, `none` (enforced both as an `argparse` choice and re-checked in `plan_vv`).
- `model` and `quantity` are capped at 256 characters (`MAX_FIELD_LEN`); longer strings are rejected.
- The string content of `model` and `quantity` is otherwise not allowlisted or sanitized: `quantity` is echoed verbatim into the output, and an unrecognized `model` family is silently resolved to `general` rather than rejected.

### File Access

- The script reads and writes no files; all I/O is command-line args -> stdout JSON (or a short plain-text summary), with errors on stderr.
- Because no filesystem paths are accepted or constructed, there is no path-traversal surface and no path-sandboxing logic is needed.
- The only DoS-relevant size limit is the 256-character cap on `model` and `quantity`; numeric outputs are bounded by the validated inputs.

### Tool Restrictions

The frontmatter declares `allowed-tools: Read, Bash, Write, Grep, Glob`.

From HeshamFS/materials-simulation-skills, Apache-2.0.

More from HeshamFS/materials-simulation-skills