“Testing”

Human-validated set of 500 real GitHub issues from Python repositories for testing coding agents.

Plain Markdown file at a repository root that tells coding agents how to build, test and work in the project.

Leaderboard that tests how accurately models call functions and tools, from single calls to multi-turn agentic use.

Benchmark and framework that tests models and agents on tasks using real-world MCP servers.

1,266 hard-to-find questions with short answers for testing how persistently agents can search the web.

Stress test for realistic MCP use across Notion, GitHub, filesystem, PostgreSQL and Playwright servers.

Self-hosted web environment with realistic sites (shopping, forums, code hosting, maps) for testing web agents.

75 Kaggle competitions turned into offline tasks for testing machine learning engineering agents.

Grid puzzles that are easy for people but hard for AI, testing abstraction and reasoning on new problems.

Harder, long-horizon software engineering tasks from public and proprietary codebases, with a held-out set.