llm4ad-task-builder

Files (20)

• README.md
• SKILL.md
• api-contract.md · reference/
• README.md · reference/templates/
• config.yaml · reference/templates/lunarlander/
• instance_001.json · reference/templates/lunarlander/data/sample/
• instance_002.json · reference/templates/lunarlander/data/sample/
• debug_run.py · reference/templates/lunarlander/
• lunarlander_evaluator.py · reference/templates/lunarlander/
• choose_action.py · reference/templates/lunarlander/policy/
• requirements.txt · reference/templates/lunarlander/
• test_evaluator.py · reference/templates/lunarlander/
• config.yaml · reference/templates/tsp/
• instance_001.json · reference/templates/tsp/data/sample/
• instance_002.json · reference/templates/tsp/data/sample/
• debug_run.py · reference/templates/tsp/
• requirements.txt · reference/templates/tsp/
• test_evaluator.py · reference/templates/tsp/
• solve.py · reference/templates/tsp/tsp_algorithm/
• tsp_evaluator.py · reference/templates/tsp/

SKILL.md (first lines)

# Building an LLM4AD_Next Task Package

## What LLM4AD_Next is

LLM4AD_Next is an automated algorithm-design platform: it combines an LLM with
evolutionary optimization to discover and improve algorithms automatically. The
user describes a problem (e.g. "evolve a solver for the Traveling Salesman
Problem"), marks the code region to evolve with `# EVOLVE_START` / `# EVOLVE_END`,
and the platform repeatedly asks an LLM to propose better code inside that region,
scores each candidate with a custom evaluator, and keeps the best across
generations. Better algorithms emerge through guided search rather than manual
trial and error.

Your job with this skill: turn a user's problem into a **complete, runnable task
package**, then verify it actually runs before handing it over.

## AutoDiscovery runtime mode

When this Skill is installed together with `algorithm-discovery`, first help that
Skill resolve the source-grounded function boundary, I/O contract, objective,
constraints, evaluator data, and reproducibility requirements. Then call the
runtime's `build_algorithm_task` tool with one complete description. Do not
hand-write package files and do not leave package construction for project
management: the tool invokes this project's official builder and validation
pipeline and returns the exact runnable package to publish.

Only a package whose returned validation status is `passed` may be published.
Project management imports that same package and starts no second requirements or
build conversation. Evolution itself still starts only when the user requests it.

## What a task package is

A task package is a self-contained directory with everything the platform needs to
evolve algorithms for one problem:

| File | Purpose |
|---|---|
| `config.yaml` | Master config for the whole pipeline (evolution params, providers, evaluator, dataset). This is what gets run. |
| `<name>_evaluator.py` | A `BaseEvaluator` subclass that runs a candidate algorithm on test data and returns a score + metrics. Defines what "better" means. |
| `<algo_dir>/<algo>.py` | The algorithm, with the function to evolve wrapped in `# EVOLVE_START` / `# EVOLVE_END`. Reads input as a JSON CLI arg, prints a JSON result. |
| `requirements.txt` | Third-party dependencies (auto-generated by build engine based on task description and code analysis). |
| `debug_run.py` | Runs the full pipeline once (`LLM4AD("config.yaml").run()`) — a smoke test. |
| `test_evaluator.py` | Loads the evaluator via the dispatcher to confirm it imports and is wired correctly. |
| `data/sample/*.json` | 2-3 small test instances the evaluator scores algorithms against. |

## The contracts each file must satisfy

**Algorithm file** (`<algo_dir>/<algo>.py`):
- The algorithm directory is the one named in `version_control.local_path`; the
  platform copies it into a git worktree for each candidate and edits only the code
  between the markers.
- The function to evolve is wrapped exactly in `# EVOLVE_START` and `# EVOLVE_END`
  comment markers (the platform only edits code between them).
- It reads its input as a single JSON string from `sys.argv[1]`, and prints a JSON
  result to stdout — so the evaluator can run it as a subprocess.
- The file name must match the name the evaluator looks for (see below). In the TSP
  example both sides use `solve.py`.

**Evaluator** (`<name>_evaluator.py`):
- **Before writing evaluator code**, read `reference/api-contract.md` for the complete
  BaseEvaluator contract and common pitfalls.
- Subclasses `BaseEvaluator`, decorated `@BaseEvaluator.register("<name>")`, with a
  no-argument `__init__(self)` (the base `__init__` takes no config).
- Implements `metrics` (list of `Metric(name, type=MetricType.MINIMIZE, weight, description)`
  — note `MetricType.MINIMIZE`, not bare `MINIMIZE`), `name`, and `async def evaluate(self, cfg: EvalContext) -> EvaluationResult`.
- `cfg: EvalContext` carries `cfg.project_root` (the candidate's worktree dir),
  `cfg.data_path` (the current data file), and `cfg.timeout` (seconds per instance).
- `evaluate` locates the algorithm file **by hardcoded name** under
  `cfg.project_root` (e.g. `Path(cfg.project_root) / "solve.py"`), runs it as a
  subprocess on `cfg.data_path`, parses its JSON output, validates it, and returns
  `EvaluationResult(score, metrics, success, ...)` (negative cost for a minimization
  objective). That hardcoded name **must** equal the algorithm file's actual name —
  this is the most common wiring mistake.

**config.yaml** (10 sections — generated programmatically, not hand-written):
- `providers` use `${LLM_BASE_URL}` / `${LLM_API_KEY}` / `${LLM_MODEL}` env
  placeholders (the platform fills them in).
- The evaluator `module` reference (`<file>.py:<ClassName>`), the `metrics` list,
  the `dataset` path (`data/sample`), and the coder `prompt_template`'s EVOLVE block
  must all be consistent with the other files.

**debug_run.py** / **test_evaluator.py**: standard boilerplate that runs
`config.yaml` end-to-end and loads the evaluator, respectively.

See `reference/api-contract.md` for the BaseEvaluator contract and
`reference/templates/` for two complete, copyable template packages demonstrating
different problem patterns (see `reference/templates/README.md`).

## The information to gather before building

A correct package needs these decided (ask the user; let the agent fill sensible
defaults where the user has no preference):

1. **Problem description** — what is being optimized. (Required — ask.)
2. **Function/algorithm to evolve + input/output format** — ask if the user has any
   constraints; if not, the agent designs it.
3. **Evaluation metric(s)** — what to minimize/maximize. (Required — ask clearly.)
4. **Reuse existing code/evaluator?** — user provides existing code, or generate from
   scratch. (Ask.)
5. **Data source** — user provides data, or generate sample instances. (Ask.)
6. **Programming language** — default Python. (Offer as a choice.)
7. **Project name** — the agent proposes one from the description; confirm with user.

## Build workflow

1. Understand the problem and the 7 items above (inspect any user-provided code/data).
2. Write the algorithm file first; verify it runs standalone and prints valid JSON.
3. Write 2-3 sample data files under `data/sample/`.
4. Write the evaluator; keep its metrics consistent with `config.yaml`.
5. Write `config.yaml`, `debug_run.py`, `test_evaluator.py`.
6. **Self-test (required):** run `test_evaluator.py` (evaluator loads — no LLM, no
   network, cheap and always runnable). Then run `debug_run.py`, which executes the
   full pipeline and **does call the LLM**: it needs the `LLM_BASE_URL` /
   `LLM_API_KEY` / `LLM_MODEL` env vars set (the config uses these placeholders) and
   consumes tokens over the network. Read every error, fix the relevant file,
   re-run. Not done until both run cleanly. If no provider credentials are available,
   say so — a missing-credential failure of `debug_run.py` is not a package defect.

## Automated algorithm design methods (reference — only when the user asks)

From Optima-CityU/LLM4AD_Next, BSD-3-Clause.

More from Optima-CityU/LLM4AD_Next