Protocols & benchmarks

Revised version of OSWorld, a benchmark of open-ended tasks on real desktop operating systems for computer-use agents.

Human-validated set of 500 real GitHub issues from Python repositories for testing coding agents.

Simulated customer-service benchmark where an agent follows domain policies and uses tools while a simulated user also acts.

Benchmark of hard, realistic command-line tasks that agents must complete inside a terminal sandbox.

Benchmark and framework that tests models and agents on tasks using real-world MCP servers.

2,500 expert-written, closed-ended questions across dozens of academic subjects, including multimodal items.

Interactive benchmark of novel game-like environments that agents must explore and learn without instructions.

Benchmark of 450+ real-world questions for general AI assistants, split into three difficulty levels.

Leaderboard that tests how accurately models call functions and tools, from single calls to multi-turn agentic use.

Set of 108 long-horizon computer-use workflows covering everyday and professional desktop work.

Harder, long-horizon software engineering tasks from public and proprietary codebases, with a held-out set.

Real work tasks from professionals across many occupations, graded by experts against human deliverables.

1,266 hard-to-find questions with short answers for testing how persistently agents can search the web.

Stress test for realistic MCP use across Notion, GitHub, filesystem, PostgreSQL and Playwright servers.

Self-hosted web environment with realistic sites (shopping, forums, code hosting, maps) for testing web agents.

75 Kaggle competitions turned into offline tasks for testing machine learning engineering agents.

Continuously refreshed coding benchmark built from new contest problems to limit training-data contamination.

Grid puzzles that are easy for people but hard for AI, testing abstraction and reasoning on new problems.