asc-ml-system-design

Files (2)

• SKILL.md
• openai.yaml · agents/

SKILL.md (first lines)

# Asc Ml System Design

Use this for the ML version of an architecture pass.
It should stay generic until the task proves otherwise.

## The Scaffold

Use these eight sections in order.

1. Functional requirements.
2. Non-functional requirements.
3. System shape.
4. Success metrics.
5. Model adaptation strategy.
6. Evaluation and release gating.
7. Monitoring and iteration.
8. Tradeoffs and open questions.

## What To Put In Each Section

### 1. Functional requirements

- What must the model or ML system do.
- What inputs it gets.
- What outputs it must produce.
- What constraints are part of the task, such as policy, format, latency, or domain language.

### 2. Non-functional requirements

- Latency, cost, robustness, throughput, and maintainability.
- Quality across slices, not just average quality.
- Deployment and fallback constraints.

### 3. System shape

- Where the model sits in the product.
- What the control flow looks like.
- What is deterministic and what is learned.
- What guardrails sit inline.

### 4. Success metrics

- Pick task metrics that reflect actual user value.
- Include slice-level metrics.
- Separate offline metrics from production proxies when needed.

### 5. Model adaptation strategy

Use `asc-ml-adaptation` for the detailed choice.
At this level, decide the smallest adaptation that can plausibly move the task.

- `CPT` for domain or language coverage.
- `SFT` for task behavior and output shape.
- `RL-based preference tuning` for ranking among multiple valid outputs.

### 6. Evaluation and release gating

Use `asc-ml-eval` for the detailed rubric.
State what must pass before release.

- held-out offline evals
- slice checks
- regression checks
- canary or rollback gates when relevant

### 7. Monitoring and iteration

- What will you watch after launch.
- What failure modes matter.
- What signal triggers another round of data or tuning.

### 8. Tradeoffs and open questions

- What you are optimizing for.
- What you are explicitly not solving yet.
- What needs more evidence before you commit.

## Design Rule

Keep the first pass simple.
If the problem can be solved with a smaller, more boring ML shape, prefer that.
Only specialize once the task repeats or the evidence forces it.

From cavaunpeu/agent-skills-closet, MIT.

More from cavaunpeu/agent-skills-closet