AGENTS.md · microsoft/waza

Sections

1. Overview
2. Squad Team
3. Project Tracking
4. Code Structure (Go)
5. Go Naming Conventions
6. Key Go Patterns
7. Review-derived Engineering Standards
8. Building and Testing
9. CI/CD
10. Fixture Isolation
11. Documentation Requirements
12. Documentation Maintenance
13. Adding New Features
14. Code Ownership and Review
15. Commit Messages
16. Files to Ignore
17. Quick Reference
18. Epics and Priorities

First lines (shown under the repo's MIT license)

# Agent Instructions for waza Repository

## Overview

This repository contains `waza`, a CLI tool for evaluating Agent Skills. **Go is the only implementation**, and it lives in the repository root. (An early Python implementation was removed in February 2026.)

When making changes, follow these guidelines to maintain consistency and quality.

## Squad Team

See [Meet the Squad](meet-the-squad.md) for current owners, the pinned Squad
tooling, and upgrade/verification procedures. `.squad/team.md` and
`.squad/routing.md` define the active team; `.squad/governance.md` defines bounded
collaboration and independent review. Persisted GPT-first role assignments live
in `.squad/config.json`, with code-review and fallback rules in
`.squad/model-policy.md`. Historical model directives are superseded.

## Project Tracking

**Keep issues and tracking up to date:**
- **Tracking Issue:** [#66 - Waza Platform Roadmap](https://github.com/microsoft/waza/issues/66)
- **PRD:** [docs/PRD.md](docs/PRD.md)
- When completing work, update the relevant GitHub issue
- Reference issue numbers in commit messages (e.g., `feat: Add tokens command #47`)

## Code Structure (Go)

```
cmd/waza/                  # CLI entrypoint
└── main.go                # Command parsing and execution
internal/
├── config/                # Configuration with functional options
├── execution/             # AgentEngine interface and implementations
│   ├── engine.go          # Core engine interface
│   ├── mock.go            # Mock engine for testing
│   └── copilot.go         # Copilot SDK integration
├── graders/               # Grader interface and Create() factory
│   ├── grader.go          # Grader interface + Create() switch over grader kinds
│   ├── text_grader.go     # Text grader (one file per grader kind)
│   └── file_grader.go     # File grader
├── models/                # Data structures
│   ├── spec.go            # EvalSpec (eval configuration)
│   ├── testcase.go        # TestCase (task definition)
│   ├── grader_params.go   # models.*GraderParameters types
│   └── outcome.go         # EvaluationOutcome (results)
├── orchestration/         # EvalRunner for coordinating execution
│   └── runner.go          # Eval orchestration
└── scoring/               # Skill-adherence heuristic scoring
    └── scoring.go         # Scorer / HeuristicScorer
go.mod
go.sum
Makefile                   # Build and test commands
.golangci.yml
```

## Go Naming Conventions

The Go implementation uses idiomatic Go naming:

| Concept | Go Name | Defined in |
|---------|---------|------------|
| Eval configuration | `EvalSpec` | `internal/models/spec.go` |
| Executor | `AgentEngine` | `internal/execution/engine.go` |
| Grader | `Grader` | `internal/graders/grader.go` |
| Task | `TestCase` | `internal/models/testcase.go` |
| Result | `EvaluationOutcome` | `internal/models/outcome.go` |

## Key Go Patterns

### Functional Options for Configuration
```go
cfg := config.NewEvalConfig(spec,
    config.WithSpecDir("evals"),
    config.WithVerbose(true),
)
```

### Builder for Engine Construction
```go
// options may be nil in production; tests pass a NewCopilotClient factory
engine := execution.NewCopilotEngineBuilder(modelID, nil).Build()
```

### Interface-based Design
```go
type AgentEngine interface {
    Initialize(ctx context.Context) error
    Execute(ctx context.Context, req *ExecutionRequest) (*ExecutionResponse, error)
    Shutdown(ctx context.Context) error
    SessionUsage(sessionID string) *models.UsageStats
}
```

### Grader Factory
```go
// Create() switches over the models.*GraderParameters type to build a Grader
g, err := graders.Create(identifier, params)
```

## Review-derived Engineering Standards

These standards come from the last 150 merged PRs reviewed on 2026-10-15
(merged 2026-07-27 through 2026-10-15): 106 dependency PRs and 44 other PRs,
including 310 review threads and the review bodies and conversation comments.
The links below identify representative feedback, not a claim that every
suggestion was accepted or every merged PR was defect-free. Check the final
disposition and current implementation before reusing a proposed fix.

### Preserve wire contracts and field presence

- Treat Go loaders, eval/task JSON Schemas, preflight, result writers, API DTOs,
  TypeScript guards, exports, and documentation as separate consumers. Update
  every affected consumer, including task/checkpoint graders and regrading.
- Distinguish omitted, null, empty, explicit zero/false, and populated values
  wherever the contract does. Preserve that distinction through copies and
  read/regrade/write cycles; do not repair invalid types into valid defaults.
  For strict identity-bearing profiles, validate original wire types, duplicate
  keys/aliases, Unicode, and numeric precision before lossy decoding.
- Keep legacy reader defaults separate from new writer versions. Eval, result,
  snapshot, and sidecar versions are independent; task YAML has no independent
  version negotiation. Do not apply a new strict profile to unrelated historical
  payloads or tighten an intentionally tolerant loader without a reviewed
  compatibility decision.

Sources: [nested grader/schema parity](https://github.com/microsoft/waza/pull/573#pullrequestreview-5237174677),
[presence lost during regrade](https://github.com/microsoft/waza/pull/652#discussion_r4266695291),
[legacy admission regression](https://github.com/microsoft/waza/pull/666#discussion_r4236249110).

### Resolve effective configuration once, then use it consistently

- Reuse shared resolution and validation instead of adding command-specific
  copies. Run, grade, preflight, trigger, migration, and API discovery must agree
  on the supported filenames, spec-relative paths, task overrides, filtered
  skills, and original CSV row identities. Honor configured file/glob settings.
- Resolve agent selection and policy from the same effective paths as execution.
  Preserve explicit disable/deny-all behavior, source-qualified tool namespaces,
  and per-run policy across initial/resumed/follow-up turns; do not leak mutable
  policy between tasks or baseline passes.
- When adding behavior-affecting configuration, audit `internal/cache/cache.go`.
  Hash the relevant effective configuration/content, including interacting flags,
  or explicitly bypass incompatible cached results. Test both cache-key changes
  and actual request/session propagation.

Sources: [run/grade CSV divergence](https://github.com/microsoft/waza/pull/671#discussion_r4252290593),
[effective skill selection](https://github.com/microsoft/waza/pull/670#discussion_r4262603609),
[cache invalidation](https://github.com/microsoft/waza/pull/627#discussion_r4122534791),
[policy namespaces](https://github.com/microsoft/waza/pull/596#discussion_r4096082837).

### Report only observed, complete evidence

- Missing scores, credits, provenance, or observations are unavailable, not zero
  or success. Preserve explicit recorded zero. Keep mock, authored, cached, and
  live evidence distinguishable; a digest or configured mock does not prove
  execution, reconstruction, dependency isolation, or real-world assurance.
- Use authoritative final SDK usage, not token-price estimates. Count each
  session once, distinguish cumulative snapshots from deltas, include discarded
  retries/judges/responders, and exclude cached historical billing from the new
  evaluation. Incomplete inputs must not become authoritative partial totals;
  offline regrading must preserve execution billing.
- Bind evidence to its actual eval/task/run/attempt, including cache-source