“coding”

Continuously refreshed coding benchmark built from new contest problems to limit training-data contamination.

Human-validated set of 500 real GitHub issues from Python repositories for testing coding agents.

Harder, long-horizon software engineering tasks from public and proprietary codebases, with a held-out set.

Benchmark of hard, realistic command-line tasks that agents must complete inside a terminal sandbox.

Self-hosted web environment with realistic sites (shopping, forums, code hosting, maps) for testing web agents.

75 Kaggle competitions turned into offline tasks for testing machine learning engineering agents.