Judgment Labs

Continuous-improvement stack for agents: monitor behavior, detect failures and validate fixes

Observability tool by Judgment Labs

IndexedApache-2.0★ 1,070

Install

Pick your client. Read the repository before you give any tool access to your data.

pip install judgeval

Get the weekly diff

The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.

About

Judgeval is an open-source Python SDK that adds OpenTelemetry-based tracing through a decorator, plus agent judges that label traces for failures or signals. Judges can score live production traffic with no latency impact and send Slack alerts, or replay on historical traces to validate fixes before shipping.

Use cases: monitoring-observability, llm-evaluation

Reviews

Write a review

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Fact sheet

Category: Tracing
Open source: Yesgithub.com
Works with: langgraph, openlit, claude-agent-sdkraw.githubusercontent.com
Protocols: opentelemetryraw.githubusercontent.com

Facts read from the vendor's own pages by claude-haiku-5-5 on Oct 24, 2026; a fact is shown only with the page it came from.

Signals

Stars: 1,070
Forks: 104
Open issues: 17
Weekly downloads: 77,683
Last commit: Oct 23, 2026 (5 days ago)
Last release: Oct 21, 2026 (6 days ago)
Last package publish: Oct 21, 2026 (6 days ago)

From GitHub (fetched Oct 24, 2026) and the package registry.

Details

Packages: pypi judgeval
License: Apache-2.0
Repository: github.com/JudgmentLabs/judgeval
Website: www.judgmentlabs.ai
Docs: docs.judgmentlabs.ai
First seen: Oct 24, 2026
Last checked: Oct 24, 2026
Source: README Docs Website github-graphql

Cite this page

Judgment Labs. Agent Mart. https://agentmart.dev/observability/judgment-labs (last updated 2026-10-24)