MCP Eval Runner

Install

Works with Claude Code, Codex, VS Code, Cursor, Windsurf, Claude Desktop. Read the repository before you give any tool access to your data.

npx agentmarket install connector/dbsectrainer-mcp-eval-runner

About

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Permissions

What this server can do and what it asks for, from its tool list and install config. Checked Oct 25, 2026.

Tools

6 read · 3 write · 1 delete

Secrets

1 (API keys, tokens or passwords)

Install scripts

Not scanned yet

Runs as

Node.js package

Tools (10)

Read from the source code; the running server may differ. Descriptions are the server’s own words; hidden characters are shown as [U+…].

• compare_resultsread-only

  Diff two named run results by run ID. Shows regressions, fixes, new and removed cases.

• create_test_case

  Create a new YAML fixture file in the fixtures directory. Supported assertion types in each step's expect block: output_contains: "substring" — checks output includes this text output_equals: "exact" — checks exact output match tool_called: "tool_name" — verifies which tool was called latency_under: 500 — asserts response time under N ms schema_match: { type: 'object', ... } — validates output JSON against schema

• discover_fixturesread-only

  Discover fixture files across one or more directories. Returns a list of fixtures with their names, paths, and step counts. If no dirs are provided, uses the configured fixtures directory and FIXTURE_LIBRARY_DIRS env var (colon-separated list of paths).

• evaluate_deployment_gateread-only

  CI gate — evaluate whether recent eval runs meet a minimum pass rate threshold. Fails (passed: false) if the pass rate of recent runs drops below min_pass_rate. Use this in CI pipelines to block deployments on regressions.

• generate_html_reportread-only

  Generate a full single-file HTML report for a completed run. The report includes: suite summary (pass/fail counts, duration), per-case drill-down with per-assertion pass/fail details, and color-coded status (green=pass, red=fail, yellow=error). Use run_suite or run_case first, then pass the returned run_id here.

• list_casesread-only

  Enumerate all available fixtures with their step counts and assertion types.

• regression_reportread-only

  Compare current state to the last run and return what changed (regressions, fixes, new cases).

• run_case

  Run a single named test case from the fixtures directory. Assertion types supported in fixture expect blocks: output_contains: "substring" output_equals: "exact text" tool_called: "tool_name" latency_under: 500 schema_match: { type: "object", required: ["id"], properties: { id: { type: "number" } } }

• run_suite

  Execute all fixtures in the fixtures directory. Returns a pass/fail summary. Execution modes: Live mode: fixture has a 'server' block with command/args — spawns the server and calls tools via real MCP stdio transport. Simulation mode: no server block — assertions run against expected_output fields. Assertion types supported in fixture expect blocks: output_contains: "substring" output_not_contains: "substring" output_equals: "exact text" output_matches: "regex" tool_called: "tool_name" latency_under: 500 schema_match: { type: "object", required: ["r…

• scaffold_fixture

  Generate a boilerplate YAML fixture file in the fixtures directory. Each tool name provided becomes a documented step with placeholder input parameters and all supported assertion types pre-filled as comments. Example: scaffold_fixture({ name: 'search_test', tool_names: ['search', 'summarize'] }) Supported assertion types (pre-filled as TODOs in the generated file): output_contains, output_equals, tool_called, latency_under, schema_match

Signals

Stars

0

Forks

0

Open issues

0

Last commit

Sep 22, 2026 (1 month ago)

Last release

Mar 23, 2026 (7 months ago)

From GitHub (fetched Oct 24, 2026).

Details

Registry name

io.github.dbsectrainer/mcp-eval-runner

Version

1.0.0

Auth

API key

Packages

npm mcp-eval-runner

License

MIT

Repository

github.com/dbsectrainer/mcp-eval-runner

First seen

Oct 24, 2026

Last checked

Oct 25, 2026

Source

Official MCP Registrygithub-graphql