Evalview MCP

Install

Works with Claude Code, Codex, VS Code, Cursor, Windsurf, Claude Desktop. Read the repository before you give any tool access to your data.

npx agentmarket install connector/hidai25-evalview-mcp

About

Use cases: llm-evaluation, testing

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Permissions

What this server can do and what it asks for, from its tool list and install config. Checked Oct 25, 2026.

Tools

3 read · 4 write · 0 delete · 3 not yet classified

Secrets

1 (API keys, tokens or passwords)

Install scripts

None found

Runs as

Python package

Tools (10)

Read from the source code; the running server may differ. Descriptions are the server’s own words; hidden characters are shown as [U+…].

• compare_agents

  Compare two agent endpoints side-by-side on the same test suite. Useful for A/B testing a new model, prompt change, or architecture swap. Returns per-test score deltas, tool diffs, and an optional HTML report.

• create_test

  Create a new EvalView test case YAML file for an agent. Call this when the user asks to add a test, or when you want to capture expected agent behavior. After creating a test, call run_snapshot to establish the baseline. No YAML knowledge required — just describe the test. IMPORTANT: Automatically detect test_path by looking for a 'tests/evalview/' directory in the current project. If found, use it. Otherwise use 'tests'.

• generate_skill_tests

  Auto-generate test cases from a SKILL.md file. Call this when the user asks to create tests for a skill — it reads the skill definition and generates a ready-to-run YAML test suite covering explicit, implicit, contextual, and negative test categories. After generating, call run_skill_test to execute them.

• generate_visual_report

  Generate a beautiful self-contained HTML visual report from the latest evalview check or run results. Opens automatically in the browser. Call this after run_check or run_snapshot to give the user a visual breakdown of traces, diffs, scores, and timelines. Returns the absolute path to the generated HTML file.

• list_tests

  List all available golden baselines in this EvalView project. Shows test names, variant counts, and when each baseline was last updated.

• replay

  Open a trajectory diff viewer for a specific test. Shows a side-by-side HTML comparison of baseline vs current behavior — tool calls, parameters, outputs, and score changes. Opens automatically in the browser.

• run_check

  Check for regressions against the golden baseline. Returns a diff summary for each test: PASSED, OUTPUT_CHANGED, TOOLS_CHANGED, or REGRESSION. REGRESSION means the score dropped significantly — treat this as a blocking failure. TOOLS_CHANGED / OUTPUT_CHANGED are warnings: the agent's behavior shifted but may be intentional. Also returns observability signals: behavioral anomalies (tool loops, stalls), trust scores (benchmark gaming detection), and coherence issues (multi-turn context loss). Use this after any code change (prompt, model, tools) to confirm nothing broke. If you see a regression,…

• run_skill_test

  Run a skill test suite against a SKILL.md. Executes two evaluation phases: Phase 1 (deterministic) checks tool calls, file operations, commands run, output content, and token budgets. Phase 2 (rubric) uses LLM-as-judge to score output quality against a defined rubric. Call this after writing skill tests or after any change to the skill or agent. Use --no-rubric for fast Phase 1-only checks with no LLM cost.

• run_snapshot

  Run tests and save passing results as the new golden baseline. Use this to establish or update the expected behavior after an intentional change. Future `run_check` calls will compare against this snapshot. Call this: (1) after creating a new test with create_test, (2) after confirming a behavioral change is intentional, (3) before making large refactors so you have a clean rollback point. Only passing tests are saved — failing tests are skipped with a warning. IMPORTANT: Automatically detect test_path by looking for a 'tests/evalview/' directory in the current project. If it exists, pass it a…

• validate_skill

  Validate a SKILL.md file for correct structure, naming conventions, and completeness. Call this after writing or editing a SKILL.md before running tests. Returns a list of issues found and whether the skill is valid.

Signals

Stars

140

Forks

27

Open issues

2

Contributors

18

Weekly downloads

2,125

Last commit

Oct 16, 2026 (12 days ago)

Last release

Jul 26, 2026 (3 months ago)

Last package publish

Jul 26, 2026 (3 months ago)

From GitHub (fetched Oct 24, 2026) and the package registry.

Details

Registry name

io.github.hidai25/evalview-mcp

Version

0.6.0

Auth

API key

Packages

pypi evalview

License

Apache-2.0

Repository

github.com/hidai25/eval-view

First seen

Oct 24, 2026

Last checked

Oct 24, 2026

Source

Official MCP Registrygithub-graphql