AGENTS.md · promptfoo/promptfoo

Sections

1. Project Overview
2. Project Structure
3. Build Commands
4. Testing in Development
5. Running Evaluations
6. End-to-End Work Expectations
7. Debugging & Troubleshooting
8. Git Workflow (CRITICAL)
9. Pull Request Creation
10. Security-Sensitive PRs
11. Screenshots for Pull Requests
12. Code Style Guidelines
13. Logging
14. Testing
15. Project Conventions
16. Before Writing Code
17. Knip (dead-code CI check)
18. Adversarial and Redteam Bias
19. Review Guidelines
20. Documentation Testing

First lines (shown under the repo's MIT license)

# AGENTS.md

Guidance for AI agents working on this TypeScript codebase.

## Project Overview

Promptfoo is an open-source framework for evaluating and testing LLM applications.

## Project Structure

| Directory           | Purpose                         | Local Docs                   |
| ------------------- | ------------------------------- | ---------------------------- |
| `.agents/`          | Codex metadata and repo skills  | `.agents/AGENTS.md`          |
| `.github/`          | GitHub Actions and workflows    | `.github/AGENTS.md`          |
| `code-scan-action/` | Code scan GitHub Action wrapper | `code-scan-action/AGENTS.md` |
| `docs/agents/`      | Reusable coding-agent docs      | `docs/agents/AGENTS.md`      |
| `plugins/`          | Agent plugin bundles            | `plugins/AGENTS.md`          |
| `src/`              | Core library                    | -                            |
| `src/app/`          | Web UI (React 19/Vite/MUI v7)   | `src/app/AGENTS.md`          |
| `src/assertions/`   | Assertion handlers              | `src/assertions/AGENTS.md`   |
| `src/codeScan/`     | Code scan scanner               | `src/codeScan/AGENTS.md`     |
| `src/commands/`     | CLI commands                    | `src/commands/AGENTS.md`     |
| `src/contracts/`    | Public package contracts        | `src/contracts/AGENTS.md`    |
| `src/database/`     | SQLite/libSQL persistence       | `src/database/AGENTS.md`     |
| `src/matchers/`     | Assertion matcher helpers       | `src/matchers/AGENTS.md`     |
| `src/models/`       | Eval/result persistence models  | `src/models/AGENTS.md`       |
| `src/prompts/`      | Prompt loading & processors     | `src/prompts/AGENTS.md`      |
| `src/providers/`    | LLM providers                   | `src/providers/AGENTS.md`    |
| `src/redteam/`      | Security testing                | `src/redteam/AGENTS.md`      |
| `src/scheduler/`    | Concurrency & rate limits       | `src/scheduler/AGENTS.md`    |
| `src/server/`       | Backend server                  | `src/server/AGENTS.md`       |
| `src/tracing/`      | OpenTelemetry trace storage     | `src/tracing/AGENTS.md`      |
| `src/types/`        | Config/API types & Zod schemas  | `src/types/AGENTS.md`        |
| `src/util/`         | Shared utilities                | `src/util/AGENTS.md`         |
| `src/validators/`   | Config validation schemas       | `src/validators/AGENTS.md`   |
| `test/`             | Tests (Vitest)                  | `test/AGENTS.md`             |
| `site/`             | Docs site (Docusaurus)          | `site/AGENTS.md`             |
| `examples/`         | Example configs                 | `examples/AGENTS.md`         |
| `drizzle/`          | DB migrations                   | `drizzle/AGENTS.md`          |

**Read the relevant AGENTS.md when working in that directory.**

## Build Commands

```bash
# Core commands
npm run build              # Build the project
npm run build:clean        # Clean the dist directory
npm run build:watch        # Watch and rebuild TypeScript files
npm test                   # Run all tests
npm run tsc                # Run TypeScript compiler

# Linting & Formatting
npm run lint               # Run Biome linter (alias for lint:src)
npm run lint:src           # Lint src directory
npm run lint:tests         # Lint test directory
npm run lint:site          # Lint site directory
npm run format             # Format all files (Biome + Prettier)
npm run format:check       # Check formatting without changes
npm run l                  # Lint only changed files
npm run f                  # Format only changed files

# Testing
npm run test:watch         # Run tests in watch mode
npm run test:integration   # Run integration tests
npm run test:redteam:integration  # Run red team integration tests
npm run test:app -- src/pages/path/to/test.test.tsx --run  # Run a specific frontend test file from repo root
npx vitest path/to/test    # Run a specific backend test file

# Development
npm run dev                # Start both server and app
npm run dev:app            # Start only frontend (localhost:3000)
npm run dev:server         # Start only server/API (localhost:15500)
npm run local -- eval      # Test with local build

# Database
npm run db:generate        # Generate Drizzle migrations
npm run db:migrate         # Run database migrations
npm run db:studio          # Open Drizzle studio

# Other
npm run jsonSchema:generate  # Generate JSON schema for config
npm run citation:generate    # Generate citation file
```

## Testing in Development

When testing changes, use the local build:

```bash
npm run local -- eval -c path/to/config.yaml
```

**Important:** Always use `--` before flags with `npm run local`:

```bash
npm run local -- eval --max-concurrency 1  # Correct
npm run local eval --max-concurrency 1     # Wrong - flags go to npm
```

**Don't run `npm run local -- view`** unless explicitly asked. Assume the user already has `npm run dev` running. The `view` command serves static production builds without hot reload.

When starting `npm run dev`, keep it attached in a live terminal session; backgrounding with `&`/`nohup` can exit silently in agent shells. The expected local URLs are `http://localhost:3000/` for the Web UI and `http://localhost:15500` for the server/API. Do not assume Vite's default `5173`; confirm the actual ports from startup output or with `lsof -nP -iTCP:3000 -iTCP:15500 -sTCP:LISTEN`.

### Using Environment Variables

The repository includes a `.env` file for API keys. To use it:

```bash
# Use --env-file flag to load environment variables
npm run local -- eval -c config.yaml --env-file .env

# Or set specific variables inline
OPENAI_API_KEY=sk-... npm run local -- eval -c config.yaml

# Disable remote generation for testing
PROMPTFOO_DISABLE_REMOTE_GENERATION=true npm run local -- eval -c config.yaml
```

**Never commit the `.env` file or expose API keys in code or commit messages.**

## Running Evaluations

**Always run from the repository root**, not from subdirectories.

**Always use `--no-cache` during development** to ensure fresh results:

```bash
npm run local -- eval -c examples/my-example/promptfooconfig.yaml --no-cache
```

**Export and inspect results** to verify pass/fail/errors:

```bash
npm run local -- eval -c path/to/config.yaml -o output.json --no-cache
```

Add `--env-file .env` or another explicit env file only when the eval needs local
secrets and the file exists.

Review the output file for `success`, `score`, and `error` fields. With the default
pass-rate threshold, exit code 0 means the eval met the threshold; still inspect the
JSON for per-test failures, errors, and scores, especially when the threshold has been
lowered. This is the standard command for verifying a PR end-to-end.

Keep local secrets in the repo's gitignored `.env` (or another path the user points at
with `--env-file`); never echo them into logs or commit messages.

## End-to-End Work Expectations

When asked only to review or audit a PR, keep the work read-only: inspect the branch, diff, PR comments, and CI as needed; run non-mutating tests or QA when useful; and report findings without committing, pushing, or changing files unless the user explicitly asks for fixes.

When asked to fix, improve, or land a PR, own the full loop: check out the branch, inspect the diff and PR comments, merge or rebase on current `origin/main` when requested, run focused tests, run the relevant real workflow, commit, push, and watch CI until it is green or the remaining failure is clearly unrelated.

**Standing commit/push authorization on feature branches.** When the user has asked you to fix, improve, or land work on a non-`main` branch, you have durable authorization to `git commit` and `git push` to that branch's tracking remote without per-step confirmation. Do not pause to ask "want me to commit?" — committing and pushing is part of the requested work. The safety constraints in _Git Workflow (CRITICAL)_ below (no commits to `main`, no `--force` without approval, no `--no-verify`, etc.) still apply.

**After landing a PR, watch `main` until its CI is green.** Merging is not the end of the loop. The squash commit kicks off a fresh CI run on `main` that can fail for reasons the PR's own checks never surfaced — a base that advanced, a flaky job, or a real regression from the merge. After you merge, follow that `main` run to completion and do not leave `main` red:

- **Classify before reacting.** Read the failing job's logs and decide whether it is a flake or a real failure. A failure is a _flake_ when the tests themselves pass and the job dies on infrastructure noise — e.g. a vitest `Worker exited unexpectedly` / `Timeout terminating forks worker` printed _after_ `Test Files N passed`, a cache-cleanup step (`Post Use Node …`), or a transient `Install Dependencies` error. The signature of a flake is non-determinism: different jobs/files fail across consecutive commits. A _real_ failure is deterministic and attributable to the change — the same test, build, or type error fails on re-run.
- **Flake → re-run.** Re-run only the failed jobs (`gh run rerun <run-id> --failed`) and confirm the run goes green. Do not blame the just-merged change for a flake it cannot have caused (e.g. a config-only diff breaking a test worker).