easy-auto-research

Files (33)

• SKILL.md
• VERSION
• research_spec.filled_example.json · reference/
• _stub.py · scripts/Skills/_shared/
• SKILL.md · scripts/Skills/analyze-hpo/
• analyze_hpo.py · scripts/Skills/analyze-hpo/scripts/
• SKILL.md · scripts/Skills/arxiv-verified-search/
• arxiv_verified_search.py · scripts/Skills/arxiv-verified-search/scripts/
• SKILL.md · scripts/Skills/disciplined-edit/
• disciplined_edit.py · scripts/Skills/disciplined-edit/scripts/
• SKILL.md · scripts/Skills/estimate-vram/
• estimate_vram.py · scripts/Skills/estimate-vram/scripts/
• LICENSE · scripts/Skills/humanizer/
• SKILL.md · scripts/Skills/humanizer/
• SKILL.md · scripts/Skills/inspect-batch/
• inspect_batch.py · scripts/Skills/inspect-batch/scripts/
• SKILL.md · scripts/Skills/inspect-checkpoint/
• inspect_checkpoint.py · scripts/Skills/inspect-checkpoint/scripts/
• SKILL.md · scripts/Skills/reset-project/
• SKILL.md · scripts/Skills/validate-before-run/
• validate_code.py · scripts/Skills/validate-before-run/scripts/
• harness.py · scripts/
• init.py · scripts/
• watch_session.py · scripts/
• evaluator.md · templates/
• executer.md · templates/
• group_duty.md · templates/
• interviewer.md · templates/
• orchestrator.md · templates/
• planner.md · templates/
• research_spec.template.json · templates/
• secretary.md · templates/
• verifier.md · templates/

SKILL.md (first lines)

# Easy-Auto-Research for Deep Learning — Research Loop

Easy-Auto-Research for Deep Learning provides a human-supervised research loop in Claude Code. Six role-specialized Claude Code sessions run in a fixed cycle — **Planner → Executer → Secretary →
Verifier → Evaluator**, with a stateless **Orchestrator** routing between them — each using a
role-specific prompt. It improves the user's **own local ML
codebase** against a goal they define and versions every experiment. It may stop on verified
success, user action, abort, configured cycle/stall limits, or an unrecoverable failure.

This skill is self-contained. Everything lives under this skill's own directory. The skill
agent resolves **`$SKILL`** internally to the absolute directory containing this SKILL.md;
this path and all commands are implementation details, not part of the human interface:
- `scripts/` — the runtime controllers and `scripts/Skills/` helper skills used by the research roles.
- `templates/` — six runtime role templates, `interviewer.md`, `group_duty.md`, and `research_spec.template.json`.
- `reference/research_spec.filled_example.json` — an illustrative filled spec to adapt.
- `VERSION` — the release version.

---

## Mental model you must hold (before doing anything)

**Three layers, state flows downward only:**
1. **`$SKILL`** — read-only source bundle (this skill). Never mutated by a run.
2. **`$PROJ`** — one folder **per research task**. Gets a *copy* of the runtime and owns
   project state (`goal.md`, `agents/`, ledgers, sessions). Disposable and independent.
3. **`$RUN_ROOT/WorkSpace/` and `$RUN_ROOT/CycleReport/`** — siblings of `$PROJ`; hold
   versioned experiments (including per-version logs) and human reports.

The reason each task gets its own copy: **`harness.py` and `init.py` resolve every path
relative to their own file location.** `harness.py` treats *its own directory* as the project
root. So a `$PROJ` folder is fully self-contained and two tasks never collide — which is
exactly what makes runs portable across different Claude Code sessions.

**Two invariants — never work around them:**
- **Only the Evaluator may emit `EVAL_VERDICT: GOAL_MET`**, and the harness rejects it unless
  the cited evidence files exist on disk. No other Claude Code role (not even the Orchestrator) can end
  the run as a success.
- **`goal.md` is the single source of truth.** Never silently relax the goal to make a run
  "succeed."

**Safety gate:** `harness.py` runs **real training jobs** — launches background processes,
can **saturate GPUs**, edits code inside version copies, runs for **hours to days**. Setup and
history-reading are safe; **before launching or resuming the loop, state this plainly and get
explicit user confirmation.**

## User-facing interaction contract

The human interface is natural language only. Ask for goals, paths, constraints, approval,
guidance, or a request to continue in ordinary prose. Never instruct the user to run, copy,
watch, resume, or paste a shell command; never expose internal CLI syntax as a user procedure.
The skill agent performs every filesystem and process operation itself, then reports outcomes
in natural language. Commands may appear below only inside blocks explicitly labeled
**INTERNAL IMPLEMENTATION — SKILL AGENT ONLY** and must never be quoted to the user as steps.

---

## The six Claude Code roles (to explain the system if asked)

| Role | Does | Never does |
|-------|------|-----------|
| **Planner** | Reads log + prior versions + `goal.md` + `human_comments.txt`; proposes ONE next experiment (version + step budget). | Edits code, runs training. |
| **Executer** | Edits the harness-created current version, launches training in background, confirms step 1, exits (<5 min). | Creates version dirs, monitors past step 1, judges results. |
| **Secretary** | Writes the human-readable cycle report; supports interactive chat; forwards feedback to `human_comments.txt`. | Edits code, kills processes, judges goals. |
| **Verifier** | Polls training ~30s, kills doomed runs, emits `STATUS:` (`DONE_NORMAL`/`CRASHED`/`KILLED_EARLY_STOP`/…). | Claims the goal is met. |
| **Evaluator** | Re-reads `goal.md`, checks evidence on disk, emits `EVAL_VERDICT`. | Monitors training, edits code. |
| **Orchestrator** | Stateless router: one JSON routing decision per turn. | Reads files, runs commands. |

---

# The three tasks this skill handles

Figure out which one the user wants, then follow that section.

- **No `$PROJ` yet / "start a new research"** → **Task 1**.
- **A loop is running and the user gives an instruction** ("tell it to try X", "stop touching
  the data pipeline", "pause and explain") → **Task 2**.
- **A run exists but the harness process is not running** ("resume", "continue", "pick it back
  up") → **Task 3**.

---

## Task 1 — START a new research run

Goal: stand up a fresh `$PROJ`, generate its goal and Claude Code role prompts, and launch the loop.

### 1.1 Gather the research spec
The whole run is defined by one file: `research_spec.json`. Open
`$SKILL/templates/research_spec.template.json` (every field + hints) and
`$SKILL/reference/research_spec.filled_example.json` (an illustrative example). Collect from the
user — ask with AskUserQuestion if anything is unclear — at minimum the **REQUIRED** fields:
- `codebase_path` — absolute path to their existing ML codebase (must exist).
- `what` — the concrete task a Claude Code role can act on; **include the exact training launch command**.
- `dos` / `donts` — allowed vs forbidden actions.

Strongly recommended (else the framework delegates a guess): `how` (metric + direction),
`baseline` (current number), `terminate` (the success bar), `search_posture`
(`hparam` | `structural` | `mixed`), and optionally `method_priors` / `signal_horizon_hint`.
**Preserve the user's exact numbers, metric names, and paths — never invent them.**

### 1.2 Prepare and initialize the project
Agree with the user on an **empty** project folder and the model choice. The skill agent then
creates the self-contained run, writes the filled spec, initializes the role prompts, verifies
that all six prompts are non-empty, and reviews `goal.md` with the user. This review is the
cheapest point to correct the objective.

> **INTERNAL IMPLEMENTATION — SKILL AGENT ONLY.** Execute this block yourself. Never show it
> as instructions for the user or ask the user to copy any part of it.
>
> ```bash
> mkdir -p "$PROJ"
> cp "$SKILL/scripts/init.py" "$SKILL/scripts/harness.py" "$PROJ/"
> cp -r "$SKILL/templates" "$PROJ/templates"
> cp -r "$SKILL/scripts/Skills" "$PROJ/Skills"
> cd "$PROJ"
> python3 init.py --output-dir "$PROJ" --spec "$PROJ/research_spec.json" \
>         --model <model>
> ```
>
> Before execution, write the filled spec to `$PROJ/research_spec.json`, confirm `claude` is
> available on `PATH`, and select a capable model. Afterward, verify

From wjc2830/Easy-AutoResearch-for-DeepLearning, MIT.