AGENTS.md · alibaba/skill-up

Sections

1. Project overview
2. Setup commands
3. Build & run
4. Testing
5. Code style & static analysis
6. Repository layout
7. Configuration schema
8. Commit & PR guidelines
9. Security considerations
10. Release
11. What agents should NOT do

First lines (shown under the repo's Apache-2.0 license)

# AGENTS.md

> This file follows the [agents.md](https://agents.md) convention and provides
> machine-readable guidance for AI coding agents (Qoder, Claude Code, Codex,
> Cursor, etc.) working in this repository. For a human-oriented overview,
> see [`README.md`](README.md) and [`CONTRIBUTING.md`](CONTRIBUTING.md).

## Project overview

**skill-up** is a CLI evaluation framework for Agent Skill developers, written
in Go. It loads declarative eval configs (`evals/eval.yaml` + `cases/*.yaml`),
provisions an isolated workspace, invokes an Agent Engine (Qoder CLI / Claude
Code / Codex), runs Judges (`rule_based` / `script` / `agent_judge`), and emits
structured reports (JSON / JUnit / HTML / Anthropic `grading.json`).

- Module path: `github.com/alibaba/skill-up`
- Entry point: [`cmd/skill-up/main.go`](cmd/skill-up/main.go)
- Go version: **1.26+** (see [`go.mod`](go.mod))
- License: Apache-2.0

## Setup commands

```bash
# Fetch dependencies (only needed after cloning or go.mod changes)
go mod download

# Install git hooks (pre-commit + commit-msg). Idempotent; auto-runs on first build.
make hooks

# Install pinned lint tools into .tools/bin/ (golangci-lint v2.11.4, revive v1.10.0)
make lint-tools
```

If you are in mainland China and `go install` is slow, set
`GOPROXY=https://goproxy.cn,direct` before running the commands above.

On Windows, `make` is unavailable by default; use the PowerShell equivalents
in `scripts/windows/` (`hooks.ps1`, `lint-tools.ps1`, `verify.ps1`). See the
[Windows support guide](docs/guide/windows.md) for supported features and
known limitations.

## Build & run

```bash
# Standard build with version injection via ldflags -> bin/skill-up
make build

# Fast iteration build (no version injection)
go build -o bin/skill-up ./cmd/skill-up

# Run the CLI
./bin/skill-up --help
./bin/skill-up run ./evals/eval.yaml
./bin/skill-up validate ./evals/eval.yaml
```

## Testing

```bash
# Unit tests (race detector enabled — always use this)
make test

# equivalent: go test -race ./...
go test -race ./...

# GitHub Action adapter tests (requires Python 3)
make test-action

# Run a single test
go test -race -run TestFoo ./internal/config/

# End-to-end tests (build-tag gated, slower)
make e2e
# equivalent: go test -tags e2e -v ./e2e
```

Rules for agents:

- **Always run `make test` before declaring a change complete.** New code
  must not break any existing tests.
- Add or update tests alongside any behavior change. Table-driven tests are
  the dominant pattern in this repo — follow it.
- Avoid network-dependent tests; use fixtures under `internal/*/testdata/`
  or `e2e/testdata/` for inputs.
- E2E tests live in [`e2e/`](e2e/) and are gated by the `e2e` build tag;
  do not add them to regular unit packages.

## Code style & static analysis

The CI gate (and `pre-commit` hook) is `make verify`, which chains:

```
fmt-check   -> gofmt must be clean (run `make fmt` to auto-fix)
vet         -> go vet ./...
revive      -> revive -config revive.toml ./... (incremental linter)
lint        -> golangci-lint run ./... (config: .golangci.yml)
```

Agents MUST:

- Run `make fmt` before committing Go files.
- Run `make verify` before declaring a change complete. If any step fails,
  fix the findings — do not disable lints to pass CI.
- Keep the pinned tool versions consistent between [`Makefile`](Makefile)
  (`GOLANGCI_LINT_VERSION`, `REVIVE_VERSION`), CI workflows under
  [`.github/workflows/`](.github/workflows), and documentation.
- Not introduce new abstractions, helpers, or files unless the task
  actually requires them. Bug fixes should stay local.

Language & comments:

- Public (exported) identifiers need Go doc comments starting with the name.
- Inline comments are sparse — only explain non-obvious intent.
- Comment and documentation language: **English** for code-level artifacts;
  user-facing docs may be bilingual (`README.md` + `README.zh.md`).

## Repository layout

```
cmd/skill-up/       CLI main (keep minimal; delegate to internal/cli)
internal/           Private implementation — never import from outside the module
  cli/              Cobra subcommands (run / validate / list-cases / report / import / debug)
  config/           eval.yaml + cases/*.yaml loader, schema, validator (v1alpha1)
  credential/       API key resolution (flags / env / ~/.skill-up/credentials.yaml)
  runtime/          Workspace runtime: none / opensandbox
  agent/            Agent Engine adapters (qoder_cli / claude_code / codex)
  mcp/              MCP provisioner (mock / real)
  skill/            Install Skill files into Engine's conventional path (excluding evals/)
  evaluator/        Evaluator: iterates cases, calls agent.Run, returns CaseResult
  evalevent/        Internal evaluation event model, publisher, lifecycle, and JSONL sink
  judge/            Judges: rule_based, script, agent_judge
  report/           Report generators: JSON / JUnit / HTML / Anthropic grading & benchmark
  runner/           End-to-end orchestration for `skill-up run`
  logging/ observability/ shellquote/ ui/    Cross-cutting utilities
pkg/                Publicly importable APIs (semver-stable; change with care)
  skillup/          Embeddable evaluation API
  transcript/       Transcript parsing helpers
plugins/            Agent host plugin sources; complete bundles are generated under ignored dist/
  codex-skill-up/    Codex hooks/MCP adapter; packaging adds shared Skill/schema
  dsh-skill-up/     DeepSeek Harness bundle with a generated skill-upper copy under dist/
skills/skill-upper/ Canonical source for the distributable Agent Skill that guides AI agents through the
                    skill-up eval workflow (scaffolding, running, interpreting).
                    Contains SKILL.md, assets/
                    templates, references/, and its own evals/ suite. Not part of
                    `go build`; consumed by Agent Engines (Cursor, Claude Code,
                    Qoder, etc.) at runtime.
e2e/                End-to-end tests (build-tag gated) + testdata/
examples/           Example fixtures and debug inputs
docs/               Design docs, user manuals, and the VitePress site
                    (built & deployed to GitHub Pages by .github/workflows/docs.yml)
schemas/evalevent/  Versioned machine-readable evaluation event protocol schemas
schemas/skill-observation/ Host-neutral observation schema and conformance fixtures
```

### Boundary rules

- `internal/` is **private**. Never import it from outside this module; never
  re-export its types through `pkg/` without a conscious API-stability review.
- `pkg/skillup/` and `pkg/transcript/` are **public, semver-stable**.
  Breaking changes here require a new module major version.