“LLM evaluation”

Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents…

Independent assessment without debate context — adversarial loop prevents gaming

Testing and benchmarking LLM agents including behavioral testing,

Open-source platform to trace, test, evaluate and govern every LLM call and AI agent in production

Use for search evaluation, a POC or bake-off, Parallel vs Exa/Tavily comparisons, SimpleQA/DeepSearchQA benchmarks, or diagnosing search evaluation results.

AGENTEVAL

tanishbhandari24

Evaluate AI agents while you build them — benchmarks, traces, failure autopsy, and regression compare.

Use after completing any non-trivial task.

Settle a "should we switch TTS engines / is X better than Edge TTS" question by measuring produced audio files, not by reading library docs or trusting…

Evaluate technology candidates for a single decision-area using a 6-dimension rubric.

Evaluate Anvia agents and retrieval — deterministic metrics, semantic similarity, LLM judges, RAG quality, and CLI eval runs.

Use after completing any non-trivial task

Use when comparing basic, enhanced, GraphRAG, or agentic RAG designs, evaluating domain-specific RAG quality, tuning retrieval components, or deciding whether…

Use when summarizing agent evaluations where autonomous, assisted, failed, timed-out, or invalid outcomes must remain distinct and comparable.

Toolkit for tracing and evaluating AI agents and LLM apps, built by Weights & Biases

Run evals on the requirement-to-story map — deterministic traceability gate plus an LLM-judge review of every requirement→story→eval link — and write a dated…

Standard metrics, acceptance thresholds, and report format for evaluating ML models.

Scores binary predictions (Brier, log loss, ECE, MCE, AUROC, accuracy, reliability bins, per-cohort) or saved categorical outputs joined to retained targets…

Evaluate real Pi and Codex sessions against centralized criteria for personal and repo-local skills, identify whether skills helped, were neutral, or made…

Evaluate AI and agent features for grounded quality, safety, tool authority, and fallback.

Evaluation rules for five-class ESI prediction.

Evaluate third-party vendors for engineering fit.

Benchmarks LLM inference and drives GPU kernel optimization with Magpie.

lm-eval-harness: benchmark LLMs (MMLU, GSM8K, etc.).

Design honest evaluation across language/strata with sample-size warnings and exact metric.