OpenJudge

Open-source Python evaluation framework with ready-made and custom graders for scoring AI agent quality and producing reward signals.

Harness by agentscope-ai

IndexedApache-2.0★ 888

Write a review

Install

From the package registry. Read the repository before you give any tool access to your data.

pip install py-openjudge

Get the weekly diff

The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.

About

OpenJudge is an open-source evaluation framework for AI applications such as agents and chatbots. It provides ready-made graders, can generate scenario-specific rubrics as graders, runs evaluations at scale, and converts grading results into reward signals for fine-tuning. The README lists agent graders for tool selection, memory, plan feasibility and trajectory, and graders that evaluate agent skill packages.

Use cases: llm-evaluation

Reviews

Write a review

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Fact sheet

Languages: PythonREADME
Packages: py-openjudge (pypi)README
Maturity: Betagithub.com

Facts extracted by Claude Haiku 5.5 from the linked sources on Oct 24, 2026. Wrong fact? Tell us.

Alternatives

Frameworks for the same language with a similar way of orchestrating agents.

Signals

Stars: 888
Forks: 73
Open issues: 13
Weekly downloads: 231
Last commit: Oct 20, 2026 (8 days ago)
Last release: Feb 12, 2026 (8 months ago)
Last package publish: May 15, 2026 (5 months ago)

From GitHub (fetched Oct 24, 2026) and the package registry.

Details

Packages: pypi py-openjudge
License: Apache-2.0
Repository: github.com/agentscope-ai/OpenJudge
Website: openjudge.me/
First seen: Oct 24, 2026
Last checked: Oct 24, 2026
Source: README github-graphql