Google DeepMind

DeepMind’s Frontier Safety Framework has a Machine Learning R&D critical capability level. Through Gemini 3.1 Pro it was evaluated on five of METR’s seven RE-Bench tasks, reported as a human-normalized average within a 32-hour budget; the Gemini 3.7 Flash FSF report switches to an internal research engineering benchmark (GRB, 74 tasks, 2-hour timeout) with a 90 percent pass@1 rule-out threshold. Flash cards defer the determination to the most capable Pro model of the time.

Bounded evaluations

Every bounded evaluation on one 0 to 100 percent axis. The internal GRB benchmark replaces RE-Bench from the Gemini 3.7 Flash report. A black cross marks the last reported value of a series later cards drop; card names run along the top.

Unbounded evaluations

Every unbounded evaluation as a multiple. RE-Bench’s human-normalized average is already a multiple of the best 8-hour human attempt; Optimize LLM Foundry is the human runtime divided by the model’s.

Each evaluation over time

Numeric scores over card date, one panel per benchmark series. Same marks as on the OpenAI page.

RE-Bench by task

RE-Bench task by task, read off the bar charts in the Gemini 2.5 Pro, 2.5 Deep Think and 3 Pro cards, which draw these values and never print them. Each hollow point is one Gemini run (model and time budget) in release order; the red dotted line is the best 8-hour human attempt as drawn in the card, and the grey point DeepMind’s own 30-minute run of Claude 3.5 Sonnet. Gemini 3 Pro passes the human bar on Optimize a Kernel and Optimize LLM Foundry and approaches it on Restricted Architecture MLM; the Scaling Law Experiment stays far below. The cards’ printed human-normalized averages use a 32-hour best-of-k aggregation that these per-task bars do not reproduce, so no average is derived from them.

Table

Table 1: Google DeepMind: AI R&D and agentic-coding scores by model card.
Benchmark Gemini 2.5 Pro Gemini 2.5 Deep Think Gemini 3 Pro Gemini 3.1 Pro Gemini 3.5 Flash Gemini 3.6 Flash Gemini 3.7 Flash Gemini 3.8 Flash
RE-Bench (METR): best run vs best human solution “The updated chart reflects Gemini 2.5 Pro GA’s best runs, which scored between 50% and 125% of the best human-written solutions. However, the model does not reach our alert …” (GA model (06-05); best run within a 32-hour budget (43 x 45-min or 16 x 2-hour runs); 5 of 7 tasks)
RE-Bench (METR): human-normalized average 0.73 (Gemini 2.5 Pro Experimental (03-25); recalculated with METR’s methodology (May 2025 correction); 32-hour budget; 5 of 7 tasks) “Gemini 2.5 Deep Think performs broadly similarly to Gemini 2.5 Pro on RE-bench, improving performance on some challenges (particularly the Restricted Architecture MLM task) but …” (at least 20 attempts per task, 2-hour runs, aggregated to a 32-hour budget; 5 of 7 tasks; METR modular scaffold; preliminary Deep Think implementation) 1.04 (16 attempts per task with a 2-hour limit each, aggregated to a 32-hour budget (bootstrap of 16 from 24 runs); 5 of 7 tasks (two internet-requiring tasks omitted); METR modular scaffold) 1.27 (Deep Think mode; protocol not restated in card (Gemini 3 Pro FSF protocol: 16 x 2-hour attempts, 32-hour budget, 5 of 7 tasks)) “To assess these capabilities, we use a single internal benchmark: GRB”
RE-Bench (METR): Optimize LLM Foundry 47 s vs human 94 s (start 300 s) (Deep Think mode)
GRB (GDM internal research engineering benchmark) ~16% (preliminary) (same GRB evaluation as Gemini 3.7 Flash; rough comparison) 27% (74 tasks (pre-/post-training lifecycle), 2-hour timeout per task, scoped internal agent scaffold; CCL rule-out threshold pass@1 90%)
MLE-Bench 42.6% 49.7% 63.9%
SWE-bench Verified 63.8% (Experimental (03-25); single attempt)
67.2% (GA (06-05); multiple attempts)
59.6% (GA (06-05); single attempt)		 76.2% (single attempt; Thinking (High)) 80.6% (single attempt; Thinking (High))
SWE-Bench Pro 43.3% (SWE-Bench Pro (Public); single attempt) 54.2% (SWE-Bench Pro (Public); single attempt) 55.1% (SWE-Bench Pro (Public); single attempt) 58.7% (SWE-Bench Pro (Public))
Terminal-Bench 54.2% (v2.0; Terminus-2 agent) 68.5% (v2.0; Terminus-2 harness)
73.8% (v2.1; Terminus-2 harness)		 76.2% (v2.1; Terminus-2 harness) 78.0% (v2.1; Terminus-2 harness) 85.8% (v2.1 (harness not stated in card))
14.9% (v3.0 (‘General agent capabilities’))		 89.4% (v2.1 (harness not stated in card))
19.1% (v4.0 (‘General agent capabilities’))
DeepSWE v1.1 12% 37% 49% 65.3% 73.7%
FrontierCode 1.1 Main 43.6%
ML R&D CCL determination “Machine Learning R&D Autonomy Level 1 CCL Definition: The model can fully automate the AI R&D pipeline at a competitive cost. The cost comparison is relative to humans augmented …” “Machine Learning R&D Autonomy Level 1… CCL reached? No. Gemini 2.5 Deep Think exhibits improved performance on some RE-bench tasks, but still lags human performance overall …” (Deep Think mode) “Gemini 3 Pro performs better than Gemini 2.5 models, especially on the Scaling Law Experiment and Optimize LLM Foundry tasks in RE-Bench (Wijk et al., 2024). However the aggregate …” “Following FSF protocols, we conducted a full evaluation of Gemini 3.1 Pro (focusing on Deep Think mode). We found that the model remains below alert thresholds for the CBRN …” (Deep Think mode (full FSF evaluation focused on Deep Think; cyber also tested without)) “We evaluated Gemini 3.1 Pro for Frontier Safety as it was the most generally capable model as of publication of this model card… while Gemini 3.5 Flash excels at agents and …” “References the April 2026 FSF … We evaluated Gemini 3.1 Pro for Frontier Safety as it was the most generally capable model as of publication of this model card, and it did not …” “Gemini 3.7 Flash can complete individual coding tasks but lacks the independence to chain them into an end-to-end research workflow without human intervention. The model does not …” “We evaluated Gemini 3.7 Flash as outlined in our latest Frontier Safety Framework (April-2026), and found that it did not reach any Tracked or Critical Capability Levels (T/CCLs) …”

Cards