Skip to content
Agentmart
Submit a tool

LLM Eval Harness

Run a golden set and gate LLM model swaps on accuracy, with latency and cost per request.

Connector by razilzul · GitHub account verified by the MCP Registry

IndexedMITLocal (stdio)
Write a review

Install

Works with Claude Code, Codex, VS Code, Cursor, Windsurf, Claude Desktop. Read the repository before you give any tool access to your data.

npx agentmarket install connector/razilzul-llm-eval-harness

Asks which of your tools to add it to (Claude Code, Codex, Cursor, VS Code, Windsurf, Gemini CLI, Claude Desktop) and shows the change first. See https://agentmart.dev/cli.

Get the weekly diff

The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.

We use your email only for this newsletter. Privacy

About

Run a golden set and gate LLM model swaps on accuracy, with latency and cost per request.

Reviews

Write a review

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Permissions

What this server can do and what it asks for, from its tool list and install config. Checked Oct 25, 2026.

Tools
1 read · 1 write · 0 delete
Secrets
2 (API keys, tokens or passwords)
Install scripts
Not scanned yet
Runs as
Docker image

Tools by what they do 2

Read from the source code; the running server may list different tools.

Read 1 · Look things up without changing anything.

  • compare_models

Write 1 · Create, change, send or run things.

  • run_golden_set

Secrets and settings it asks for

Secrets: values you paste in that let it act as you.

  • ANTHROPIC_API_KEY
  • GEMINI_API_KEY

Tools (2)

Read from the source code; the running server may differ. Descriptions are the server’s own words; hidden characters are shown as [U+…].

  • compare_models

    A/B two models on the golden set and return the gate verdict. The candidate must match or beat the baseline on accuracy, overall and on traps; only then do latency and cost matter.

  • run_golden_set

    Run one model over the golden set. Returns accuracy overall and on trap cases, mean latency and cost per request, and the failing cases.

Signals

Stars
0
Forks
0
Open issues
0
Last commit
Oct 18, 2026 (11 days ago)
Last release
Oct 18, 2026 (11 days ago)

From GitHub (fetched Oct 24, 2026).

Details

Registry name
io.github.razilzul/llm-eval-harness
Version
1.0.0
Auth
API key
Packages
oci ghcr.io/razilzul/llm-eval-harness:1.0.0
License
MIT
Repository
github.com/razilzul/llm-eval-harness
First seen
Oct 24, 2026
Last checked
Oct 24, 2026
Source
Official MCP Registrygithub-graphql
Cite this page

LLM Eval Harness MCP server. Agent Mart. https://agentmart.dev/connectors/razilzul-llm-eval-harness (last updated 2026-10-24)