Skip to content
Agentmart
Submit a tool

AI agent tools for LLM evaluation

In this catalog, an LLM evaluation tool is one that measures model or agent quality with evals, benchmarks and test sets. Our enrichment pass tags 91 listings with it: 23 MCP servers, 27 agent skills, 6 plugins, 10 agent frameworks, 22 AI agents and 3 no-code agent builders. By listing rank, the top MCP server is Evalview MCP, the top agent skill is caveman-manage and the top AI agent is maka. 6 of the 91 are published by the vendor behind the product, and 61 were active in the last 30 days. 2 of the 23 MCP servers run only as remote servers, with nothing to install locally. Each entry below links to its listing page, with grades, install commands and sources.

Updated Oct 26, 2026 · Built from catalog data; how we rank and grade

What the job is

Tagged listings
91
By type
23 MCP servers, 27 agent skills, 6 plugins, 10 agent frameworks, 22 AI agents and 3 no-code agent builders
Published by the vendor
6 of 91
Active in the last 30 days
61 of 91
Open source
57 of 91
MCP servers that run only remotely
2 of 23
Hosts named by the MCP servers
Claude Code (23), Cursor (23), VS Code (23)

Building blocks by type

The top entries of each type, by listing rank. Open a listing for its install command, grades and sources.

MCP servers 23 tagged

  1. 1
    Evalview MCP

    hidai25

    Regression testing for AI agents.

    • Active in the last 30 days
    • Open source (Apache-2.0)

    ★ 140↓ 2.1K/wkLast active Oct 16, 2026Local (stdio)Claude Code, Codex, VS Code, Cursor +2

  2. 2
    Ouroboros

    Q00

    Pins an acceptance spec; the verify command and expected output never enter the success contract.

    • Most GitHub stars here (6,269)
    • Most weekly downloads here (30.4K)
    • Active in the last 30 days

    ★ 6.3K↓ 30.4K/wkLast active Oct 24, 2026Local (stdio)Claude Code, Codex, VS Code, Cursor +2

  3. 3
    Agenda Intelligence MD

    vassiliylakhonin

    Evidence-readiness MCP server: validate, audit, and score briefs, memos, and evidence packs.

    • Runs locally or as a remote server
    • Active in the last 30 days
    • Open source (MIT)

    ★ 8↓ 1K/wkLast active Oct 24, 2026Local (stdio), Remote (HTTP)Claude Code, VS Code, Codex, Cursor +2

  4. 4
    Cosmergon

    rkocosmergon

    Living economy for AI agents — Conway physics, energy currency, marketplace, reputation.

    • Active in the last 30 days
    • Open source (MIT)

    ★ 4↓ 3.4K/wkLast active Oct 23, 2026Local (stdio)Claude Code, Codex, VS Code, Cursor +2

  5. 5
    Rubber Duck

    nesquikm

    Bridge to multiple LLMs and CLI agents: councils, debates, voting, and more

    • Active in the last 30 days
    • Open source (MIT)

    ★ 180Last active Oct 13, 2026Local (stdio)Claude Code, Codex, VS Code, Cursor +2

Search MCP servers for LLM evaluation → All connectors

Agent skills 27 tagged

  1. 1
    caveman-manage

    JuliusBrussee

    Inspect Caveman Cloud's experiment lifecycle and block unsafe execution.

    • Largest repository here (114,786 stars)
    • Active in the last 30 days
    • Open source (Apache-2.0)

    ★ 114.8KLast active Oct 13, 2026Claude Code, Claude apps, Codex, Cursor +6

  2. 2
    agent-evals

    devops-skills

    Build automated evaluation suites for AI agents using golden datasets, rubrics, and regression gates.

    • Repository has 48.4K stars
    • Active in the last 30 days
    • Open source (MIT)

    ★ 48.4KLast active Oct 24, 2026Claude Code, Claude apps, Codex, Cursor +6

  3. 3
    prime-intellect

    PrimeIntellect-ai

    Work with Prime Intellect products via the prime CLI and Python SDKs - verifiers environments and the Environments Hub, evaluations (local and hosted), Hosted…

    • Repository has 22.3K stars
    • Active in the last 30 days

    ★ 22.3KLast active Oct 21, 2026Claude Code, Claude apps, Codex, Cursor +6

  4. 4
    autoresearch

    Companion-Inc

    Bounded research experiment loop that tries hypotheses, measures benchmark evidence, keeps what works, and records what fails.

    • Repository has 10K stars
    • Active in the last 30 days
    • Open source (MIT)

    ★ 10KLast active Oct 24, 2026Claude Code, Claude apps, Codex, Cursor +6

  5. 5
    archify-tuning

    tt-a1i

    Improve Archify diagram accuracy, visual quality and complete authoring speed with fewer repairs.

    • Repository has 88.7K stars
    • Active in the last 30 days
    • Open source (MIT)

    ★ 88.7KLast active Oct 19, 2026Claude Code, Claude apps, Codex, Cursor +6

Search agent skills for LLM evaluation → All skills

Plugins 6 tagged

  1. 1
    Inspect AI

    ukaisi

    Extension for the Inspect AI evaluation framework.

    • Active in the last 30 days
    • Open source (MIT)

    ★ 7Last active Oct 20, 2026VS Code, Cursor, Windsurf, VSCodium

  2. 2
    BenchClaw — P2PCLAW Agent Benchmark

    agnuxo1

    Benchmark any AI agent (Claude, GPT, Gemini, Qwen, Kimi, DeepSeek, Grok, Llama…) on the P2PCLAW network.

    • Open source (MIT)

    ★ 9Last active Jun 5, 2026VS Code, Cursor, Windsurf, VSCodium

  3. 3
    Copilot Arena

    copilot-arena

    Code with and evaluate the latest LLMs and Code Completion models

    • Most GitHub stars here (372)

    ★ 372Last active Dec 1, 2025VS Code, Cursor, Windsurf, VSCodium

  4. 4
    AGENTEVAL

    tanishbhandari24

    Evaluate AI agents while you build them — benchmarks, traces, failure autopsy, and regression compare.

    • Open source (MIT)

    ★ 1Last active Sep 23, 2026VS Code, Cursor, Windsurf, VSCodium

Search plugins for LLM evaluation → All plugins

Agent frameworks 10 tagged

  1. 1
    Google Agent Development Kit (ADK)

    Google

    Open-source toolkit for building, evaluating and deploying agents in Python, TypeScript, Go, Java and Kotlin, tuned for Gemini but model-agnostic.

    • Most GitHub stars here (21,990)
    • Most weekly downloads here (2.4M)
    • Active in the last 30 days

    ★ 22K↓ 2.4M/wkLast active Oct 24, 2026

  2. 2
    judgeval

    JudgmentLabs

    Python SDK for tracing agents and scoring their traces with prompt-based judges, to find failures in production.

    • 1.1K GitHub stars
    • Active in the last 30 days
    • Open source (Apache-2.0)

    ★ 1.1K↓ 77.7K/wkLast active Oct 23, 2026

  3. 3
    fast-agent

    evalstate

    CLI-first Python agent framework and coding agent with MCP, A2A and Agent Skills support, live streaming and human input.

    • 3.9K GitHub stars
    • Active in the last 30 days
    • Open source (Apache-2.0)

    ★ 3.9K↓ 19.5K/wkLast active Oct 22, 2026

  4. 4
    EvoAgentX

    ANative-Lab

    Open-source framework that builds, evaluates and self-improves multi-agent LLM workflows from a goal.

    • 3.4K GitHub stars
    • Active in the last 30 days
    • For Python

    ★ 3.4KLast active Oct 22, 2026

Search agent frameworks for LLM evaluation → All harnesses & sdks

AI agents 22 tagged

  1. 1
    maka

    apache

    Apache-licensed agent workspace that keeps an append-only event log of every model and tool action.

    • Most GitHub stars here (5,870)
    • Active in the last 30 days
    • Open source (Apache-2.0)

    ★ 5.9KLast active Oct 24, 2026

  2. 2
    Arbor

    RUC-NLPIR

    Autonomous research agent that forms hypotheses, edits code and runs experiments, keeping only gains that hold on held-out data

    • 1.1K GitHub stars
    • Open source (Apache-2.0)

    ★ 1.1KLast active Sep 8, 2026

  3. 3
    BotsRated (beta)

    BotsRated

    Signed checks of paid AI services, before your agent pays.

  4. 4
    Council of AI — Measurement Agent

    CSOAI Ltd

    Independent AI-governance MEASUREMENT body.

  5. 5
    The Crawler Zoo Arena

    The Crawler Zoo

    Games for AI agents, played over plain HTTP: negotiation, social deduction, racing, puzzles and more, against house bots and other agents.

Search AI agents for LLM evaluation → All agents

No-code builders 3 tagged

  1. 1
    ElevenLabs Agents Platform

    ElevenLabs

    Platform for configuring, deploying and monitoring voice, chat and email agents in more than 90 languages

  2. 2
    Kore.ai

    Kore.ai

    Enterprise agent platform for building, governing and optimizing AI agents for customer service and employee work.

  3. 3
    UiPath Agent Builder

    UiPath

    Low-code canvas or Python for building enterprise agents that run alongside robots, APIs and people, with evals and audit logs.

Search no-code builders for LLM evaluation → All no-code builders

Questions

What are the top tools for LLM evaluation?
By listing rank, the top MCP server is Evalview MCP, the top agent skill is caveman-manage and the top AI agent is maka. Rank weighs adoption, freshness, documentation and publisher identity; money never changes it.
How many tools does the catalog list for LLM evaluation?
91 listings are tagged for it: 23 MCP servers, 27 agent skills, 6 plugins, 10 agent frameworks, 22 AI agents and 3 no-code agent builders. Tags come from an automated enrichment pass over the top-ranked listings, so tools outside that pass are not counted.
Which AI apps do the MCP servers work with?
Of the 23 MCP servers, 23 list Claude Code, 23 list Cursor and 23 list VS Code as a supported host. Open a listing for its install command and the other hosts it names.
Which of these are published by the vendor itself?
6 of the 91 are published by the vendor behind the product, including aig-scanner, data-leakage-detection and direct-injection-detection.

91 listings are tagged for this use case; the lists above show the top entries.

Related