Method and data

Sources and provenance

The long-form data with a source and a confidence flag on every row is data/ai_rd_benchmarks.csv. Provenance marks in the table cells: no mark means the number was read from the card itself (or from the METR report the card cites); * means it was read from the lab’s launch post because the card does not give it; † means it was read from a search-result excerpt of a write-up and not found in any primary document; ‡ means it was recalled from memory and not checked; ≈ means the card gives the result only as a chart and the number was read off the chart (the highest point on the model’s curve, when the chart plots score against budget). Chart-read points are drawn hollow in every figure. Almost every cell was verified against the card on 2026-09-13; the handful still marked † or ‡ are launch-blog figures (SWE-bench Verified for the Codex models, Terminal-Bench for OpenAI models) that no card contains. Where a later card re-runs an earlier model and gets a different number, the cell keeps the model’s own card value and the notes column records the re-run. The card-by-card verification record is data/notes/ai-rd-benchmarks.md.

Scores are as the lab reports them and are not comparable across labs: each lab uses its own scaffold, time budget, attempt count and problem subset, and internal suites are only defined inside one lab. Conditions that change the number are given in parentheses inside the cell.

Reading the figures

The figures are built for reading saturation. Every bounded metric is drawn on its full range with the ceiling as a solid line; a human reference point stated by the card (an expert solution, a threshold the card expresses in human-hours) is a red dotted line; a rule-out threshold not tied to human performance is a grey dashed line. A black cross sits on a series’ last reported value when the lab’s later cards no longer report it, whether because it was retired as saturated, replaced by a harder version, or dropped from the determination; the documented reason, where the card gives one, is in the series file. Card names run along the top of each lab’s overview at their dates.

Each lab page has, in order: one overview with all of its bounded evaluations on the same 0 to 100 percent axis, so the order in which benchmarks saturate and get replaced is visible; one overview of its unbounded metrics (speedups, human-normalized scores above 1, time horizons, lower-is-better errors) on a log axis of multiples, where a series with a card-stated human reference is divided by it, so 1x is the reference reached, and a series with no reference (METR’s time horizons, Anthropic’s staff-survey uplift) is indexed to its first card and drawn dashed; small multiples with one panel per evaluation; the benchmark-by-model table; and the list of cards. Ceilings, directions, reference values and retirement dates are in data/ai_rd_benchmark_series.csv.

Curves read from the cards

OpenAI’s GPT-5.6 and GPT-6 Astra cards report the research debugging eval, KernelGen 1P, NanoGPT and PostTrainBench Lite only as line charts of score against budget (API cost, simulated latency or output tokens), with no printed values. scripts/digitize_curves.py recovers the points: it finds each chart’s axes and tick marks, finds the filled markers by their legend colour after removing the thin lines, and maps marker centres to data values. The result is data/ai_rd_scaling_curves.csv, drawn on the OpenAI page as one panel per evaluation and budget measure. Where a card prints a value that also appears on a chart (Astra’s 78.05 percent on research debugging), the digitised value agrees to within a tenth of a point. The scalar entered in the benchmark table for such rows is the highest point on the model’s curve, marked ≈.

DeepMind’s Gemini 2.5 Pro, 2.5 Deep Think and 3 Pro cards draw RE-Bench per task as grouped bar charts without printed values. scripts/digitize_bars.py reads them: it finds the gridlines, removes error bars and star markers by erosion, measures each bar and names it by its position in the group, which follows the legend. The result is data/ai_rd_rebench_tasks.csv, drawn on the DeepMind page; a bar that appears in more than one card reads the same to within 0.02. The Anthropic cards print their numbers; the one figure-only result, CoBench in the Fable 5.1 card, carries its values as bar labels and those are entered as reported.

Which cards are shown

Only cards on the evaluation frontier are shown. A card that improves on at most a quarter of the evaluations it shares with earlier cards (given at least three shared) adds no frontier information and is left out of the figures and tables; each lab page names the cards this removes, and their rows stay in the CSV. Smaller siblings released alongside a frontier model (Sonnet 4, Haiku 4.5, the Codex addenda, Gemini Flash-Lite) are flagged in the CSV and also left out.

Files

path what
data/ai_rd_benchmarks.csv one row per score: lab, model, card, benchmark, conditions, score, source, confidence
data/ai_rd_benchmark_series.csv per-series metadata: ceiling, direction, human reference, threshold, category, retirement
data/ai_rd_rebench_tasks.csv RE-Bench normalised score per task and run, read off the bar charts in the DeepMind cards
data/ai_rd_scaling_curves.csv points read off the score-against-budget charts in the OpenAI cards (model, evaluation, budget measure, x, y, figure)
data/notes/ai-rd-benchmarks.md collection and verification notes, card by card
src/benchmarks.py loading, the frontier rule, tables and every figure

Source: github.com/tecunningham/ai-rnd-benchmarks.