AI R&D evaluations across the three closed labs

OpenAI, Anthropic and Google DeepMind each publish, with every frontier release, a system or model card whose dangerous-capability section reports how far the model is from automating AI research. This site collects those AI R&D scores from the cards alone, with a source and a confidence flag on every number. Use the menu on the left to flip between the labs; each lab page opens with its graphs. How the data were collected and how to read the figures is on the Method and data page.

Benchmarks more than one lab reports

Benchmarks reported by at least two labs, scores as the labs print them. SWE-bench Verified is the one benchmark all three report, and all three take it to the mid-90s before dropping it for SWE-Bench Pro. Terminal-Bench is re-versioned every few months; each version is a separate segment. MLE-bench and RE-Bench use different task subsets at different labs, so compare position against the ceiling or the human line rather than lab against lab.

Internal research evaluations, all labs on one axis

Every bounded AI-research evaluation (internal task suites, MLE-bench, PaperBench, research Q&A, DeepMind’s GRB) as percent of its ceiling, coloured by lab. A black cross is the last value the lab reported before its later cards dropped the series. The shape the labs share: an internal research evaluation climbs toward its ceiling within one to two years and is retired or replaced.

Unbounded evaluations, all labs on one log axis

Every unbounded evaluation (speedups, human-normalized scores, time horizons, lower-is-better errors) as a multiple on a log axis, coloured by lab. Solid lines are divided by the card’s human reference, so 1x is the reference reached and the label says what the reference is; dashed lines have no reference and are indexed to their first card. Anthropic reports by far the most such metrics; its kernel-optimization and LLM-training speedups pass their expert references in 2025 and 2026, then Suite 1 is dropped. OpenAI’s only unbounded series is METR’s time horizon; DeepMind’s is RE-Bench.

Reading across the three

Each lab measures a different thing under the AI R&D heading. DeepMind reports a single human-normalized number on a public METR suite, then an internal research-engineering benchmark; OpenAI reports pass rates on internal replication tasks and Kaggle-style competitions; Anthropic reports fractions of a human baseline on internal tasks plus its own RE-Bench subset. SWE-bench Verified is the only benchmark all three report, and even there the subset and scaffold differ. The common thread is qualitative: every lab has moved from “well below human” statements in 2024 to reporting models at or above the human baseline on several-hour ML engineering tasks by 2026, while each still judges the model below its own threshold for autonomous AI R&D. Two things to watch when reading across cards: benchmark versions and subsets change (Terminal-Bench 1 to 2.0, 2.1 and 4.0; MLE-bench full to 30 to revised; RE-Bench 7 tasks to 5), and later cards re-run earlier models and print different numbers, so a single model’s score can differ between its own card and a successor’s comparison table.