DeepEval

Open-source LLM evaluation framework, similar to Pytest, for unit testing LLM apps and agents

Observability tool by Confident AI

IndexedApache-2.0★ 19,081

Install

Pick your client. Read the repository before you give any tool access to your data.

pip install deepeval

Get the weekly diff

The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.

About

DeepEval evaluates LLM apps end to end, complete agent trajectories and individual agent steps such as LLM calls, tool use and retrieval. Metrics such as G-Eval, task completion, answer relevancy and hallucination use an LLM of your choice as judge or local NLP models. Confident AI adds production tracing and failure monitoring on top.

Use cases: llm-evaluation, testing

Reviews

Write a review

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Fact sheet

Category: Evals · open source
Self-host: Yesconfident-ai.com
Open source: Yesraw.githubusercontent.com
Compliance (vendor says): SOC 2, HIPAAconfident-ai.com
Regions: Canada, Australia, Japanconfident-ai.com
Works with: langchainraw.githubusercontent.com

Plans

Free	$0	forever; 2 user seats; 1 project; 5 test runs per week; 1 GB-month of trace spans
Starter	$200/month	unlimited user seats; 5 projects; 5 GB-months of trace spans then $1 per GB-month
Team	$2000/month	unlimited seats and projects; 75 GB-months of trace spans; SOC2; SSO; dedicated support channel
Enterprise	Contact sales	custom pricing; dedicated on-prem deployment; HIPAA; custom data residency; 24x7 support

From the vendor's pricing page, read Oct 24, 2026.

Facts read from the vendor's own pages by claude-haiku-5-5 on Oct 24, 2026; a fact is shown only with the page it came from.

Signals

Stars: 19,081
Forks: 2,131
Open issues: 332
Last commit: Oct 23, 2026 (5 days ago)
Last release: Sep 22, 2026 (1 month ago)

Details

Packages: pypi deepeval
License: Apache-2.0
Repository: github.com/confident-ai/deepeval
Website: deepeval.com
Docs: deepeval.com/docs/getting-started
First seen: Oct 24, 2026
Last checked: Oct 24, 2026
Source: README Website Docs Pricing page