What are evals?

An eval checks how well an LLM application behaves by running it on test inputs and grading what comes back. OpenAI's guide compares the practice to behavior-driven development: describe the expected behavior first, then test against it. It is most useful when a prompt changes or a newer model is adopted, because the same tests show whether quality went up or down.

There is no single formal specification for evals, and the cited page describes one vendor's version. In OpenAI's Evals API an eval has two main parts. A data source configuration is a JSON schema describing each test item, such as a support ticket's text and its human-labeled category. Testing criteria are graders that decide whether an output passes, and the guide's example uses a string check that demands an exact match with the label. The test items sit in a JSONL file. A run applies a prompt template to every row, generates a response for each, grades it, and reports how many items passed, failed or errored, with a link to a dashboard report.

The guide's example is small, with three items, and it says real projects usually need more criteria, prompts and datasets. As of 26 October 2026, the page said OpenAI was deprecating its Evals platform, with existing users moving to read-only on October 31, 2026 and the platform shutting down on November 30, 2026; check OpenAI's deprecations page for the current timeline.