define-experiment

Files (2)

• SKILL.md
• openai.yaml · agents/

SKILL.md (first lines)

# Define Experiment

Treat each Experiment as one concrete training, evaluation, or train-then-evaluate declaration. It selects reusable components and fixes the phase behavior, configuration, resources, checkpoint flow, and outputs for one run design. It does not define Model, Dataset, or Benchmark components and does not execute the Experiment.

## Workflow

1. Inspect the target project's instructions, selected component paths, training and evaluation entry points, framework conventions, configuration and implementation conventions, and dependency conventions.
2. Before defining or changing an Experiment, invoke the `grill-with-docs` skill. Resolve the question and comparison, component paths, environment configuration, declared phases, configuration, resources, checkpoint flow, outputs, and failure boundaries. Continue only after the design is explicit and confirmed. If it is unavailable, stop.
3. Create or update `EXPERIMENT.md` inside `experiments/<experiment-id>/`.
4. Implement the declared phases using the target project's existing configuration or code conventions. Create or update project-native configuration, custom phase code, or both as required. When the confirmed design uses a supported framework, use its optional integration skill if available or follow the project's established convention directly. Run a bounded lifecycle smoke test only when the Experiment contract requires it.
5. Implement the declared phases, then record the explicit Python file or directory scope containing the code created or modified by this invocation. Keep `EXPERIMENT.md`, configuration or phase code, and commands consistent. If the question, components, phases, configuration, implementation, resources, checkpoint flow, outputs, or failure boundaries change, pause and repeat the `grill-with-docs` skill before continuing.
6. After the implementation and Experiment contract agree, invoke `improve-python-documentation` with that explicit Python scope. Then invoke `fix-python-quality` with the same explicit scope. Do not let either utility infer scope from Git status or the diff.
7. List the changed files, documentation changes, quality commands and results, and unverified checks, then show the diff. Do not create a commit or execute the Experiment.

If a selected Model, Dataset derivation, or Benchmark must change, hand the work to the `define-model`, `define-dataset`, or `define-benchmark` skill.

## Experiment Directory

Use one directory for each concrete Experiment:

```text
experiments/
  <experiment-id>/
    EXPERIMENT.md
    configs/
      <configuration files>
    <implementation files>
```

The optional `configs/` directory contains configuration consumed directly by the target project or framework. The Experiment may also contain custom phase implementation files when the target project does not provide the required training or evaluation entry point. Use existing project conventions; do not require a framework, schema, filename, or one configuration file per phase. Keep Experiment-specific training and evaluation configuration and code inside the Experiment directory; keep architecture and model-behavior configuration in the Model component.

## EXPERIMENT.md

`EXPERIMENT.md` is the human-readable contract for the Experiment. Keep it concise and consistent with its configuration and commands.

Use these sections:

### Description

Describe the research question, comparison, purpose, and explicit boundaries of the Experiment.

### Components

Identify the Model, Dataset derivations, Benchmark, and other inputs used by the declared phases. Record their logical identities, roles, repositories when external, and paths. Record external locators or digests when they are part of the Experiment's input contract. Version fixation and submission membership belong to `run-experiment`, not this specification.

A Train phase requires a Model and its training Dataset derivations. A Benchmark is optional unless training consumes its protocol. An Evaluate phase requires a Model, Benchmark, and checkpoint; the Benchmark identifies its Dataset derivations and evaluation inputs.

### Environment

Identify the environment configuration files or directory used by the Experiment, such as `pyproject.toml`, `requirements.txt`, `uv.lock`, `poetry.lock`, or `environment.yml`. Record the paths and use the target project's existing environment convention. Do not duplicate configuration content in `EXPERIMENT.md` or require a setup command here. The configuration files express the Python and package versions used by the Experiment; this section does not define CUDA or hardware requirements.

### Phases

Use `#### Train` and `#### Evaluate` subsections for the phases that apply. Declare at least one phase.

For Train, record the data inputs, command or entry point, configuration paths, seed, resources, checkpoint selection, and continuation or failure conditions. Validation used for early stopping, checkpoint selection, or training monitoring remains part of Train. When a framework-native checkpoint applies, identify how its learned weights map to the Model's canonical weight contract and distinguish model-weight loading from full training-state restoration.

For Evaluate, record the Benchmark inputs, command or entry point, configuration paths when needed, resources, checkpoint source, and failure conditions. An evaluation-only Experiment may consume an external checkpoint or a checkpoint from a prior Result. Record its locator, digest when available, provenance, and compatibility requirements. In a train-then-evaluate Experiment, reference the selected checkpoint produced by Train; the Result records its final path and digest.

Evaluation configuration controls execution parameters such as batch size, workers, precision, devices, checkpoint selection, and output locations. It may select a declared Benchmark input or an explicitly allowed metric subset, but it must not redefine the Benchmark's metrics, masking, reduction, aggregation, or visualizations. Omit evaluation configuration when the target project does not need it.

### Outputs

Describe the checkpoints, training metrics, Benchmark metrics, visualizations, reports, logs, and other outputs produced by the declared phases, including their expected locations or location rules.

### Implementation

Record the phase commands, configuration files, custom phase code, environment files, and implementation choices needed to run the Experiment. Keep optimizer, scheduler, learning rate, batch size, epochs, seed, trainer, precision, resources, and other run-specific settings in the Experiment configuration or implementation when the target project consumes them there.

The specification must remain consistent with the configuration and commands. If implementation changes the Experiment contract, stop and realign the design before continuing.

## Run Handoff

`define-experiment` finishes with a validated Experiment contract and reports its changed files and checks. It does not create commits, determine the submission set, or execute the Experiment. `run-experiment` owns version fixation immediately before execution, including the declared dependency closure, user-confirmed control commit, external immutable identities, and pre-run `HEAD` and worktree checks.

From peoxin/skills, MIT.

More from peoxin/skills