AGENTS.md · mock-server/mockserver-monorepo

Sections

1. Instruction Priority
2. Project Overview
3. Agent Operating Model
4. Git Policy
5. Pre-Commit Workflow
6. Documentation Style
7. Code Comments
8. Diagrams and Formatting
9. Code Navigation
10. Java Compatibility Policy
11. Helm Chart Versioning Policy
12. Fix Placement Policy
13. Temporary Files
14. Local (Uncommitted) Plans
15. Parallel Sessions
16. Code Review Routing
17. Subagent Routing
18. Research-First Problem Solving
19. Release Preparation

First lines (shown under the repo's Apache-2.0 license)

# MockServer — Agent Instructions

## Instruction Priority

1. Direct user instructions (highest)
2. Rules in `.opencode/rules/`
3. This file (`AGENTS.md`)
4. Skills in `.opencode/skills/`

## Project Overview

MockServer is an open-source HTTP(S) mock server and proxy for testing, written in Java. It uses Netty as the HTTP server framework, Maven for builds, and is deployed as Docker containers, JARs, and WARs.

**Tech stack:** Java 17+ (minimum supported), Netty 4.2, Jackson 2.22, Maven (multi-module), Node.js/TypeScript (UI + client), Python 3.9+ (client), Ruby 3.0+ (client), Docker, Helm, Jekyll (documentation site)
**CI/CD:** Buildkite (primary CI — including all Docker image builds), GitHub Actions (CodeQL, scheduled hygiene jobs such as certificate-expiry, and issue/label automation)
**Infrastructure:** AWS (Buildkite build agents, documentation site hosting), Docker Hub (container images)
**Repository:** GitHub (github.com)

### Local Development Environment

**Docker is available locally.** Docker Desktop runs on the developer Mac, so Docker is available to agents in this environment (not only in CI). This means:

- **Docker-gated tests CAN and SHOULD be run locally**, not just in CI. Tests that guard on `Assume.assumeTrue(DockerAvailability.isAvailable(...))` (Testcontainers live-broker tests, `NET_ADMIN` transparent-proxy e2e, QUIC/HTTP-3 client tests, etc.) will actually execute here — validate them by running and passing them, not merely by confirming they skip.
- **Keep the Docker-gating in place anyway.** The `assumeTrue(...)` guard is still the correct design so the suite degrades gracefully on any CI agent or machine without Docker. Docker being present locally changes how we *validate*, not how we *write* the tests.
- **`org.mockserver.test.DockerAvailability` (in `mockserver-testing`) is the canonical availability probe. NEVER call `DockerClientFactory.instance().isDockerAvailable()` directly.** Despite its javadoc ("true if Docker is available, false if not") that method converts only `IllegalStateException` into `false` — it **throws** for everything else, because it also starts the Ryuk reaper and runs version/mount checks. A `BadRequestException` (privileged Ryuk rejected by a user-namespace-remapped daemon), a `ContainerFetchException`, or an `Error` such as `NoClassDefFoundError` from an incomplete test classpath all escape it, turning "no usable Docker" into a hard ERROR and defeating the `assume` guard — and Testcontainers caches that failure and rethrows it for the rest of the JVM. Write:
  ```java
  Assume.assumeTrue("Docker is not available",
      DockerAvailability.isAvailable(() -> DockerClientFactory.instance().isDockerAvailable()));
  ```
  Pass a **lambda, not a method reference** — `instance()` must be evaluated inside the wrapper's try/catch. Testcontainers is the preferred harness, and works with Testcontainers 1.21.4+ (docker-java 3.4.2) on Docker Desktop 4.67 / Engine 29.x / API 1.54. (Earlier versions — Testcontainers 1.20.6 / docker-java 3.4.1 — got a 400 on the info endpoint and the probe returned false even though Docker worked.)
- **A Docker-gated suite that runs in CI needs a fail-closed assertion too.** A fail-safe probe means an unusable Docker SKIPS rather than errors, which is right off-CI but is a silent false positive in CI. Pair it with `.buildkite/scripts/steps/assert-suite-ran.sh` over the suite's surefire reports so a skip fails the build loudly.
- `docker` CLI commands (`docker build`, `docker run`) are also available for Dockerfile smoke checks in the commit workflow.

**Behind a corporate TLS-inspection proxy?** If local dependency downloads fail with TLS / `certificate verify failed` errors, every toolchain needs to trust the corporate root CA (via a combined bundle), configured ONLY in your user/shell environment — never in repo or pipeline files. Set `LOCAL_DOCKER_CA_BUNDLE` for anything run through `.buildkite/scripts/run-in-docker.sh`, and per-toolchain env / `~/.npmrc` for host builds. To build **MockServer's own Docker images** (`docker/*/Dockerfile`) behind the proxy, set `MOCKSERVER_LOCAL_CA_BUNDLE` to the **combined** bundle (system roots + corporate root, e.g. `~/.tesco-ca/ca-bundle-with-tesco.pem`) and run `docker/ensure-ca-bundle.sh <context-dir>` first — it stages the CA into the build context as `ca-bundle.pem` (a scratch file; never commit it) so the alpine stages trust the proxy before fetching the tcnative native `.so`. This is a distinct mechanism from `LOCAL_DOCKER_CA_BUNDLE` (which is only for the `run-in-docker.sh` client-language toolchains). Full reusable setup (new-laptop checklist + per-toolchain table): [docs/operations/build-system.md → Local Development Behind a Corporate TLS-Inspection Proxy](docs/operations/build-system.md#local-development-behind-a-corporate-tls-inspection-proxy).

### Project Documentation

Comprehensive internal documentation is maintained in `docs/`. **Always consult these docs before making changes** to understand architecture, conventions, and dependencies:

| Document | When to consult |
|----------|----------------|
| [docs/README.md](docs/README.md) | Documentation index and quick reference |
| [docs/code/overview.md](docs/code/overview.md) | Before modifying any module — understand module boundaries and dependencies |
| [docs/code/netty-pipeline.md](docs/code/netty-pipeline.md) | Before modifying Netty handlers, protocol detection, or TLS |
| [docs/code/request-processing.md](docs/code/request-processing.md) | Before modifying mock matching, proxy forwarding, or action dispatch |
| [docs/code/event-system.md](docs/code/event-system.md) | Before modifying event logging, verification, or persistence |
| [docs/code/memory-management.md](docs/code/memory-management.md) | Before modifying maxLogEntries, maxExpectations, ring buffer sizing, or memory defaults |
| [docs/code/dashboard-ui.md](docs/code/dashboard-ui.md) | Before modifying the dashboard UI or WebSocket communication |
| [docs/code/domain-model.md](docs/code/domain-model.md) | Before modifying domain model, matchers, codecs, or configuration |
| [docs/code/tls-and-security.md](docs/code/tls-and-security.md) | Before modifying TLS, mTLS, certificates, or authentication |
| [docs/code/client-and-integrations.md](docs/code/client-and-integrations.md) | Before modifying client library, JUnit rules, or Spring integration |
| [docs/code/breakpoints.md](docs/code/breakpoints.md) | Before modifying request breakpoints, BreakpointRegistry, PausedExchange, request/response/stream phases, or the /breakpoint endpoints |
| [docs/code/chaos.md](docs/code/chaos.md) | Before modifying chaos experiments, ChaosExperimentOrchestrator, auto-halt integration, or the /chaosExperiment endpoints |
| [docs/code/drift-detection.md](docs/code/drift-detection.md) | Before modifying mock drift detection, DriftAnalyzer, DriftStore, or the /drift endpoint |
| [docs/code/wasm-rules.md](docs/code/wasm-rules.md) | Before modifying WASM custom rule engine, chicory integration, or WASM REST endpoints |
| [docs/code/extensions.md](docs/code/extensions.md) | Before modifying the extension SPI (`org.mockserver.extension`), ExtensionRegistry, extension loading, or the extension start/stop order in LifeCycle and the servlets |
| [docs/code/telemetry.md](docs/code/telemetry.md) | Before modifying OpenTelemetry integration, OTLP export, GenAI spans, or W3C trace context propagation |
| [docs/code/async-messaging.md](docs/code/async-messaging.md) | Before modifying the AsyncAPI broker mocking module, AsyncApiParser, MessagePublisher adapters, or AsyncApiMockOrchestrator |
| [docs/code/http3.md](docs/code/http3.md) | Before modifying experimental HTTP/3 (QUIC) support, Http3Server, or QUIC native dependencies |
| [docs/code/startup-performance.md](docs/code/startup-performance.md) | Before modifying the startup path, Docker image bases, CDS/AOT archives, `startupWarmup`, or lazy initialization of startup subsystems (BouncyCastle provider, forward client event-loop group) |
| [docs/code/performance-measurement.md](docs/code/performance-measurement.md) | Before interpreting a perf number, adding a new measurement, or claiming CI catches a regression — which k6 scripts run, which are lint-only, what each JMH benchmark actually measures |
| [docs/code/optimisation-safety.md](docs/code/optimisation-safety.md) | Before merging any performance change — the hazard-class table, differential corpus requirement, and the evidence standard that proves a change did not break correctness |
| [docs/code/clustered-state.md](docs/code/clustered-state.md) | Before modifying the StateBackend SPI, InMemoryStateBackend, InfinispanStateBackend, cross-node invalidation, or cluster configuration properties |
| [docs/code/llm-mocking.md](docs/code/llm-mocking.md) | Before modifying the LLM response builder, provider codecs, streaming physics, conversation matchers, isolation, MCP tools, or LLM dashboard |
| [docs/code/ai-protocol-mocking.md](docs/code/ai-protocol-mocking.md) | Before modifying MCP/A2A server mocking, SSE/WebSocket/JSON-RPC mock responses, or AI protocol handlers |
| [docs/code/llm-codec-fixtures.md](docs/code/llm-codec-fixtures.md) | Before modifying LLM provider codec golden files, the codec drift-detection test harness, or normalisation rules |
| [docs/code/llm-security-audit.md](docs/code/llm-security-audit.md) | Before modifying LLM mocking security-sensitive paths (prompt injection guards, credential redaction, isolation boundaries) |
| [docs/code/metrics.md](docs/code/metrics.md) | Before modifying Prometheus metrics, memory monitoring, or CSV metric export |
| [docs/code/cli.md](docs/code/cli.md) | Before modifying the command-line interface or `org.mockserver.cli.Main` |
| [docs/code/configuration-reference.md](docs/code/configuration-reference.md) | Before adding a configuration property or changing property resolution order or the equivalent property forms |
| [docs/operations/build-system.md](docs/operations/build-system.md) | Before changing Maven config, plugins, or build scripts |
| [docs/infrastructure/ci-cd.md](docs/infrastructure/ci-cd.md) | Before modifying Buildkite or GitHub Actions pipelines |
| [docs/infrastructure/aws-infrastructure.md](docs/infrastructure/aws-infrastructure.md) | Before investigating AWS, Terraform, or Buildkite agent issues |
| [docs/infrastructure/docker.md](docs/infrastructure/docker.md) | Before modifying Dockerfiles, images, or Compose configs |
| [docs/infrastructure/service-mesh.md](docs/infrastructure/service-mesh.md) | Before modifying transparent HTTP interception or Kubernetes sidecar deployment |
| [docs/infrastructure/aws-ses-email-forwarding.md](docs/infrastructure/aws-ses-email-forwarding.md) | Before modifying SES email forwarding, DNS mail records, or the Lambda forwarder |
| [docs/infrastructure/helm.md](docs/infrastructure/helm.md) | Before modifying Helm charts or Kubernetes deployment |
| [docs/operations/website.md](docs/operations/website.md) | Before modifying the Jekyll documentation site |
| [docs/operations/security.md](docs/operations/security.md) | Before adding, removing, or upgrading dependencies (Java 17 floor, version ceilings, CodeQL, Dependabot, Snyk) |
| [docs/operations/release-process.md](docs/operations/release-process.md) | When performing or automating releases |
| [docs/operations/release-principles.md](docs/operations/release-principles.md) | Before changing anything under `scripts/release/` or `.buildkite/release-*` |
| [docs/operations/ai-sdlc-integration-spec.md](docs/operations/ai-sdlc-integration-spec.md) | **Authoritative spec** for how AI is used in this repo's SDLC — autonomy, parallelism caps, model/temperature, verification, control integrity, operator halt; the `.opencode/rules/` implement it |
| [docs/operations/ai-native-sdlc-principles.md](docs/operations/ai-native-sdlc-principles.md) | For the principles behind working with AI across the SDLC |
| [docs/operations/ai-assisted-development.md](docs/operations/ai-assisted-development.md) | For understanding the AI development approach, adversarial review, and testing backstop |
| [docs/operations/opencode-configuration.md](docs/operations/opencode-configuration.md) | Before modifying opencode config, agents, rules, skills, or commands |

When making changes to the project, **update the relevant docs/ file** if the change affects architecture, dependencies, build process, CI/CD, infrastructure, or deployment.

### Consumer Documentation

The consumer-facing documentation lives in `jekyll-www.mock-server.com/` and is published to https://www.mock-server.com. **Always consider consumer docs when making changes:**

1. **Read for context** — before changing behaviour, check the consumer docs to understand what users have been told to expect. Key pages:
   - `jekyll-www.mock-server.com/mock_server/configuration_properties.html` — all configuration properties with defaults and examples
   - `jekyll-www.mock-server.com/mock_server/_includes/running_docker_container.html` — Docker usage examples
   - `jekyll-www.mock-server.com/mock_server/_includes/performance_configuration.html` — performance tuning
2. **Update when behaviour changes** — if you change defaults, add properties, modify behaviour, or fix bugs that affect user-visible behaviour, update the consumer docs to match. Keep changes simple and clear — assume users have limited context.
3. **User-friendly language** — consumer docs should explain *what* a setting does and *why* a user might change it, not internal implementation details. Include practical guidance (e.g., "each HTTP request generates 2-3 log entries") rather than code-level details.

### AWS Accounts

| Purpose | CLI Profile |
|---------|-------------|
| Pipeline build agents and infrastructure | `mockserver-build` |
| Website (S3, CloudFront, DNS, TLS) | `mockserver-website` |

Account IDs, SSO portal URLs, and resource identifiers are in `~/mockserver-aws-ids.md` (not committed to the repo). Before any AWS operation, verify that file exists and never hard-code identifiers into the repo — see `.opencode/rules/aws-ids-file.md`.

### AWS Prerequisites

To investigate or manage AWS infrastructure:

1. **Install AWS CLI**: `brew install awscli`
2. **Configure SSO profiles**:
   - Build: `aws configure sso --profile mockserver-build` (SSO region: `eu-west-2`, default region: `eu-west-2`)
   - Website: `aws configure sso --profile mockserver-website` (SSO region: `eu-west-2`, default region: `us-east-1`)
3. **Authenticate**: `aws sso login --profile mockserver-build` and/or `aws sso login --profile mockserver-website`
4. **Corporate TLS proxy**: if behind a TLS inspection proxy, set `AWS_CA_BUNDLE` to a **combined bundle** containing both the system root CAs AND your corporate root CA. Pointing it at the corporate root alone (e.g. `tesco_root_ca.pem`) is **not enough** — AWS endpoints whose certificates don't chain through the corporate proxy (e.g. SNS in us-east-1) will fail TLS validation and the AWS SDK / Terraform's AWS provider will retry silently, appearing to hang. Typical fix: build a combined `ca-bundle-with-tesco.pem` (system roots + corporate CA), then `export AWS_CA_BUNDLE=/path/to/ca-bundle-with-tesco.pem` (the same file commonly used for `REQUESTS_CA_BUNDLE` / `SSL_CERT_FILE`).
5. **macOS + Python 3.14 + Homebrew**: if you get `pyexpat` symbol errors, set `export DYLD_LIBRARY_PATH=/opt/homebrew/opt/expat/lib`

### Reading Buildkite Builds and Logs

**To read a Buildkite build log, use the local `bk` CLI — NOT the API tokens in Secrets Manager.** The tokens at `mockserver-build/buildkite-api-token` (write) and `-readonly` work for build **state**, **triggering**, and **retrying** jobs, but they intentionally lack the `read_build_logs` scope, so fetching `/jobs/<id>/log` with them returns `"doesn't have the read_build_logs scope"`. The developer's locally-authenticated `bk` CLI (`~/.config/bk.yaml`) has full scope.

```bash
# Job log (bk api prepends the org path — pass a RELATIVE endpoint):
bk api "pipelines/mockserver-release/builds/<N>/jobs/<JOB_ID>/log" \
  | python3 -c "import sys,json; print(json.load(sys.stdin).get('content',''))" \
  | sed 's/\x1b\[[0-9;]*m//g; s/_bk;t=[0-9]*//g; s/\r//g'

# Artifacts: use `bk artifacts`, NOT `bk api .../artifacts/<id>/download`.
# The api form returns 0 bytes for BINARY artifacts (a .tgz, a PNG). It appears to
# work for JSON ones, which is how the limitation gets missed.
# Note the PLURAL `artifacts`: `bk artifact download` (singular, as some Buildkite
# docs show) does not exist in this CLI and errors with "unexpected argument artifact".
# The positional is an artifact ID, not a filename glob — list first to get it.
# The two subcommands take the build number DIFFERENTLY: `list` takes it as a
# POSITIONAL, `download` takes it as `--build`. Passing `--build` to `list` prints
# usage, and passing it positionally to `download` errors "unexpected argument".
bk artifacts list <N> -p <pipeline-slug>
bk artifacts download <ARTIFACT_ID> --build <N> -p <pipeline-slug>
```

Strip the bell character too when grepping a log — lines are prefixed with a bare
`\x07`, so `^---` style anchors silently match nothing: `tr -d '\007' < log > clean`.

**Before concluding a perf build is stuck waiting for an agent:** the `perf` queue scales
from zero to at most **three** agents, one per c5.12xlarge, so each perf job has a whole
machine to itself. `mockserver-performance-test` has a **scheduled daily run at 04:00**, a
**weekly `PERF_XL` run on Saturday at 08:00** (which also uses the `perf-xl` agent) and a
**weekly soak on Sunday at 08:00** (UTC). A daily build fans its measurement steps out and
can hold all three agents at once, so a manual run can be legitimately queued behind it,
which is easy to miss because the Buildkite UI lists only the most recent pipeline runs by
default. Skip-intermediate-builds no longer applies to this pipeline's `master` builds
(`skip_intermediate_builds_branch_filter = "!master"`, applied 2026-09-29), but a build on
any other branch can still be `skipped` by a newer one. Check the
agent's `job` field (`bk api "agents?per_page=50"`) to see what is actually holding it, and
check each build's `state` — `bk build create` reports success either way; when querying
builds with `bk api`, filter by the full 40-character commit SHA. Confirm each perf build
reaches `running`. **For a close A/B comparison, measure both arms on one machine** — a