EvalForge Lite

Install

Works with Claude Code, Codex, VS Code, Cursor, Windsurf, Claude Desktop. Read the repository before you give any tool access to your data.

npx agentmarket install connector/thejaredchapman-evalforge-lite

About

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Permissions

What this server can do and what it asks for, from its tool list and install config. Checked Oct 25, 2026.

Tools

7 read · 2 write · 0 delete

Secrets

1 (API keys, tokens or passwords)

Install scripts

Not scanned yet

Runs as

Python package

Tools (9)

Read from the source code; the running server may differ. Descriptions are the server’s own words; hidden characters are shown as [U+…].

• evaluate_prompt

  Get pre-run feedback on a prompt's clarity/specificity before running a comparison. An explicit, separately-triggered LLM call (uses your credentials) — not run automatically as part of run_comparison. Rate-limited independently from run_comparison's 3-per-8h budget. Pass `creds` as {"openrouter"?: str, "bedrock"?: {...}, "vertex"?: {...}, "foundry"?: {...}} to use Amazon Bedrock, Google Vertex AI, or Microsoft Foundry; a bare `api_key` is treated as an OpenRouter key. `judge_backend` picks which backend runs the evaluation. If the operator has set server-side keys for a backend (see https://…

• get_report

  Get a PDF report (base64-encoded) for a run. Defaults to the most recent run. priority (balanced|quality|fastest|cheapest), if given, overrides the priority the run was made with for the report's priority/best-pick line; otherwise the run's own priority (from run_comparison) is used.

• get_report_csv

  Get a CSV export for a run, one row per (test case x model) cell. Defaults to the most recent run.

• list_availability

  Return the current model-availability snapshot: live OpenRouter listing status (refreshed at most every 6 hours) plus curated Bedrock/Vertex/Foundry region coverage.

• list_models

  List every provider and model in the catalog, plus each provider's frontier (flagship) model. Each model carries a `reasoning` flag (supports extended reasoning) and a `tags` list of curated need/industry tag ids; `tags` in the result gives their labels, one-line "why" explanations and the date they were verified. Tags are a starting point, not benchmarks.

• list_runs

  List metadata for the 5 most recent runs, newest first.

• run_comparison

  Run a set of test-case prompts against a set of models, scoring each response. Each test case may include an optional "rubric" (scored by an LLM judge) and/or "checks" (rule-based checks). Returns per-model grades, cost/latency stats, and an overall verdict. Rate-limited to 3 calls per 8 hours. Models are "<catalog id>" (OpenRouter) or "<catalog id>@bedrock" / "<catalog id>@vertex" / "<catalog id>@foundry"; pass matching creds ({"openrouter"?, "bedrock"?, "vertex"?, "foundry"?}) or a bare OpenRouter api_key. judge_backend picks which backend runs the judge and policy gate. At most 4 models. p…

• set_policy

  Set the company policy text used to gate prompts before any model is called.

• suggest_models

  Suggest sibling models from the same family as the given model id.

Signals

Stars

0

Forks

0

Open issues

0

Weekly downloads

41

Last commit

Oct 14, 2026 (14 days ago)

Last release

Oct 1, 2026 (27 days ago)

Last package publish

Oct 14, 2026 (14 days ago)

From GitHub (fetched Oct 24, 2026) and the package registry.

Details

Registry name

io.github.thejaredchapman/evalforge-lite

Version

1.0.1

Auth

API key

Packages

pypi evalforge-lite

License

MIT

Repository

github.com/thejaredchapman/evalforge-lite

Website

github.com/thejaredchapman/evalforge-lite/blob/main/docs/mcp-server.md

First seen

Oct 24, 2026

Last checked

Oct 24, 2026

Source

Official MCP Registrygithub-graphql