OpenAI
OpenAI’s Preparedness Framework tracks “AI Self-improvement” (earlier “Model autonomy”) with a mix of public and internal evaluations: SWE-bench Verified on a 477-problem subset, SWE-Lancer, internal pull-request replication (OpenAI PRs), MLE-bench (full set with pass@10 in the o1 era, a 30-competition subset with pass@1 from mid-2025, a revised 72-problem set from GPT-5.6), PaperBench, research-engineer interview questions, OpenAI-Proof Q&A, an internal research debugging eval and Monorepo-Bench; the GPT-5.6 card revised the suite around the research debugging eval, KernelGen 1P, NanoGPT and PostTrainBench Lite, reported only as curves of score against budget. The High threshold is worded as “a performant mid-career research engineer”.
Bounded evaluations
Unbounded evaluations
Each evaluation over time
Score against budget
Table
| Benchmark | o1-preview | o3-mini | o3 / o4-mini | GPT-5 | GPT-5.1-Codex-Max | GPT-5.2 | GPT-5.3-Codex | GPT-5.4 Thinking | GPT-5.5 | GPT-5.6 Sol | GPT-6 Astra |
|---|---|---|---|---|---|---|---|---|---|---|---|
| KernelGen 1P | ≈18% (highest point on the API cost curve, Figure 39 of the GPT-5.6 card) (re-run reported in the GPT-5.6 system card)≈ | ≈29.3% (highest point on the API cost curve, Figure 39 of the GPT-5.6 card) (re-run reported in the GPT-5.6 system card)≈ | ≈61.1% (highest point on the API cost curve, Figure 39 of the GPT-5.6 system card) (kernel writing/optimisation for OpenAI first-party hardware; correctness + latency; plotted vs cost/latency/tokens)≈ | ≈66.7% (highest point on the output tokens curve, Figure 53 of the GPT-6 Astra system card) (kernel writing/optimisation for OpenAI first-party hardware)≈ | |||||||
| NanoGPT | ≈2.3% (highest point on the API cost curve, Figure 43 of the GPT-5.6 card) (re-run reported in the GPT-5.6 system card)≈ | ≈2.7% (highest point on the API cost curve, Figure 43 of the GPT-5.6 card) (re-run reported in the GPT-5.6 system card)≈ | ≈16.6% (highest point on the output tokens curve, Figure 54 of the GPT-6 Astra system card) (one H100; reward = clamp((t_baseline - t_trial)/t_baseline, 0, 1))≈ | ≈23.6% (highest point on the output tokens curve, Figure 54 of the GPT-6 Astra system card) (one H100; normalised training-time reward)≈ | |||||||
| PostTrainBench Lite | ≈39.9% (highest point on the simulated latency curve, Figure 45 of the GPT-5.6 card) (re-run reported in the GPT-5.6 system card)≈ | ≈38.8% (highest point on the simulated latency curve, Figure 45 of the GPT-5.6 card) (re-run reported in the GPT-5.6 system card)≈ | ≈56.7% (highest point on the simulated latency curve, Figure 45 of the GPT-5.6 system card) (12 base-model x benchmark combos; 5 h; one H100; reward normalised base=0, instruct=1; LLM judge zeroes cheating)≈ | ≈77% (highest point on the output tokens curve, Figure 55 of the GPT-6 Astra system card) (12 combos; 5 h; one H100)≈ | |||||||
| Internal Research Debugging Eval | 44.1% (41 real internal bugs + 6 alignment-auditing tasks; mean rubric score) | 49.4% (41 real internal bugs + 6 alignment-auditing tasks) | 50.5% (41 real internal bugs + 6 alignment-auditing tasks; chart shows mean rubric score with 95% CI) | ≈68.3% (highest point on the output tokens curve, Figure 52 of the GPT-6 Astra system card) (41 real internal bugs + 6 alignment-auditing tasks; plotted vs API cost, latency and output tokens)≈ | 78.05% (41 real internal bugs + 6 alignment-auditing tasks; highest token budget) | ||||||
| OpenAI-Proof Q&A | 2% (20 internal research/engineering bottlenecks; container with code and run artifacts; pass@1) | 8% (8% also counting refusals) (20 problems; pass@1) | 3% (GPT-5.2 card); 4.2% (as re-run in GPT-5.4 card) (20 problems; pass@1; no browsing) | 6% (card chart); 5.8% (GPT-5.4 / GPT-5.5 cards) (20 problems; pass@1) | 4.16% (20 problems; pass@1) | 1.7% (20 problems; pass@1) | |||||
| MLE-bench: MLE-bench (75 competitions, AIDE) | 37% pass@10 (16% pass@1) (AIDE agent; 24 h per attempt; pass@10; post-mitigation) | 11% pass@1 / 20% pass@10 (post); 16% / 25% (pre) (AIDE agent; 24 h) | |||||||||
| MLE-bench: MLE-bench-30 | 8% (30 competitions <50 GB and <10 h; pass@1; 24 h; no browsing) | 17% (30 competitions <50 GB, <10 h; pass@1; 24 h) | 16% (GPT-5.2 card); 12.2% (as re-run in GPT-5.4 card) (30 competitions <50 GB, <10 h; pass@1; no browsing) | “Due to infra issues we do not report scores on GPT-5.3-codex” | 23.33% (30 competitions <50 GB, <10 h; bronze or better; pass@1) | 36.67% (30 competitions <50 GB, <10 h; bronze or better; pass@1) | |||||
| MLE-bench: MLE-Bench Revised | 87.18% (max; from GPT-6 Astra card); 5.6 card gives cost/latency curves only (72 problems; up to 3 leaderboard submissions; scored by percentile rank vs test-time-compute reference distribution) | 93.80% (max) (72 problems; up to 3 leaderboard submissions; percentile vs reference distribution) | |||||||||
| PaperBench | 8% (no browsing; high reasoning effort) | o3 18%; o4-mini 24% (chart label 25%) (pass@1; high reasoning effort; no browsing; full-effort PaperBench (20 papers) per card text) | |||||||||
| PaperBench: PaperBench-10 | 24% (10-paper subset (<10 GB data each); pass@1; high reasoning effort; no browsing) | 40% (10-paper subset; pass@1; Extra High reasoning effort; no browsing) | 39% (10-paper subset; pass@1; high reasoning effort; no browsing) | ||||||||
| OpenAI PRs | 0% (pre- and post-mitigation) | o3 44%; o4-mini 39% (helpful-only 45% / 41%) (no browsing) | 45% (no browsing; launch candidate) | 53% (no browsing) | 55% (no browsing) | ||||||
| SWE-Lancer IC SWE Diamond | 14% (Jul 2025 dataset); 7% / $17,375 (GPT-4.5 card) (pass@1, high reasoning effort) | o3 54% / $86,100 (no browsing LC); o3 browsing helpful-only 55% / $76,250; o4-mini 52% (pass@1 avg of 3 runs; Jul 17, 2025 dataset update; $453,800 total possible) | 55% (ChatGPT agent 60%) (pass@1 avg of 3; high reasoning effort; no browsing; Jul 17, 2025 dataset) | 80% (pass@1 avg of 3; Jul 17, 2025 dataset) | 81.4% (aggregator value; not in the system card)† | ||||||
| OpenAI Research Engineer interviews: multiple choice | 76.8% (post-mitigation); 80.4% (pre) (97 MCQs; cons@32) | 80% (post); 77% (pre) (97 questions; cons@32) | o3 80%; o4-mini 83% (97 questions; cons@32; launch candidates (no browsing)) | ||||||||
| OpenAI Research Engineer interviews: coding | “o1-preview (pre-mitigation) demonstrating a 21 percentage point improvement over GPT-4o in multiple-choice and a 15 percentage point improvement in coding (pass@1 metric)” (18 problems; pass@1; pass@128 also plotted (all models ~100%)) | 92% (18 problems; pass@1; post-mitigation) | o3 97-98%; o4-mini 98-99% (18 problems; pass@1; ‘saturated’) | ||||||||
| Agentic tasks (autonomy suite) | 42% (average over agentic tasks and subtasks; terminal/Python harness) | 27% (post); 26% (pre) (average over agentic tasks and subtasks; one Ranger browser task removed vs o1 card) | |||||||||
| METR external evaluation | “not above that of the best existing public model (Claude 3.5 Sonnet) … could make non-trivial progress on 2 out of 7 AI R&D tasks … could not confidently upper-bound the …” (METR baseline and adapted (advice-giving) scaffolds; access Aug 26 - Sep 9, 2024) | ||||||||||
| METR external evaluation: 50% time horizon | o3 1h30m; o4-mini 1h15m (METR time-horizon methodology; updated task suite; reward hacks scored 0; earlier checkpoints; 15 days) | 2h17m (65m to 4h25m) in METR report; card says ~2h15m (65m to 4h30m) (METR time-horizon methodology; 3 weeks; reasoning-trace access; o3 1h30m for comparison) | 2h42m point estimate (75m to 350m) in card; METR report: about 2h40m (75m to 5h50m) (METR mainline scaffold; locked checkpoint; compaction scaffold tried without significant gain; 2 weeks) | about 6.6 h (95% CI 3h20m to 17h30m) (high (not xhigh) reasoning effort; METR expanded software-task suite (Time Horizon 1.1)) | about 11.3 h (95% CI 5 to 40 h) if cheating counted as failure; not considered robust (Time Horizon 1.1 suite; METR ReAct harness; cheating rate higher than any public model METR had evaluated) | ||||||
| SWE-bench Verified | 41.3% (Agentless 1.0 scaffold; 5 tries per instance; n=477; pre- and post-mitigation both 41.3%) | 61% (internal tool scaffold; avg of 4 tries; n=477; non-final checkpoint) | o3 68% (no browsing) / 69% (browsing); o4-mini 69% / 68%; o3 helpful-only 71% (n=477; internal tool scaffold; avg of 4 tries; launch candidates) | 74% (card); 74.9% (launch blog, medium verbosity) (n=477; internal scaffold (bash + apply_patch); avg of 4 tries; card run at maximum trained-in verbosity) | 77.9% (launch blog; not in the system card)† | 80.0% (launch blog; not in the system card)‡ | |||||
| SWE-bench Verified: Agentless | 39% (post-mitigation); 40% (pre) (Agentless 1.0; 5 tries; n=477; launch candidate) | ||||||||||
| SWE-Bench Pro | 56.8% (launch blog; not in the system card)† | 58.6% (launch blog; not in the system card)† | |||||||||
| Terminal-Bench | 58.1% (Terminal-Bench 2.0; launch blog)† | 77.3% (Terminal-Bench 2.0; launch blog; not in the system card)† | 82.7% (Terminal-Bench 2.0; launch blog; not in the system card)† | ||||||||
| Monorepo-Bench | 56.70% (GPT-5.4 card); 55% (GPT-5.3-Codex card) (pull-request-style tasks in a large internal repo; hidden tests) | 56% (pull-request-style tasks in a large internal repo; hidden tests; agentic rollout) | 59.33% (pull-request-style tasks in a large internal repo; hidden tests) | 60% (pull-request-style tasks in a large internal repo; hidden tests) | |||||||
| AI self-improvement (earlier: model autonomy) classification | “Score for both o1-preview and o1-mini: Low. Summary: Neither o1-preview nor o1-mini advance self-exfiltration, self-improvement, or resource acquisition capabilities sufficiently …” (Preparedness Framework v1; o1-preview and o1-mini) | “Score for OpenAI o3-mini: Medium … o3-mini’s performance on SWE-Bench Verified demonstrates its ability to competently execute well-specified coding tasks, which elevates it to …” (Preparedness Framework v1) | “determined that OpenAI o3 and o4-mini do not reach the High threshold in any of our three Tracked Categories … on evaluations designed to test more real-world or open-ended tasks …” (Preparedness Framework v2 (first card under it)) | “gpt-5-thinking showed modest improvement across all of our self-improvement evaluations, but did not meet our High thresholds … modest improvement across all of our …” (Preparedness Framework v2; gpt-5-thinking treated as High in bio/chem) | “It does not reach High capability on AI self-improvement … GPT-5.1-Codex-Max showed modest improvement across all of our self-improvement evaluations, but did not meet our High …” (treated as High in bio; below High in cyber) | “gpt-5.2-thinking performed at a similar capability level to gpt-5.1-codex-max and did not meet our High thresholds. The High capability threshold is defined to be equivalent to a …” (High threshold defined as ‘a performant mid-career research engineer’) | “It does not reach High capability on AI self-improvement … The High capability threshold is defined to be equivalent to a performant mid-career research engineer. Performance in …” (first launch treated as High capability in Cybersecurity (precautionary); treated as High in bio) | “GPT-5.4 Thinking did not meet our thresholds for High capability in AI Self-Improvement. The High capability threshold is defined to be equivalent to a performant mid-career …” (treated as High in cyber and bio/chem) | “GPT-5.5 did not meet our thresholds for High capability in AI Self-Improvement. The High capability threshold is defined to be equivalent to a performant mid-career research …” (treated as High in bio/chem and cyber (below Critical)) | “None of them reach our High threshold in AI Self-Improvement … updated and expanded … Existing measures were also becoming saturated (Monorepo-Bench) or upon review contained …” (Sol, Terra and Luna; revised self-improvement suite) | “designation not confirmed … we determined that Astra reaches the Critical level in Cybersecurity capability, and the High level in the Biological and Chemical category. In AI …” (same revised suite as GPT-5.6) |
Cards
Cards left out as dominated, that is no better than an earlier card on all or almost all of the evaluations they share: o1 (Dec 2024: better on 1 of 4 shared evaluations); GPT-4.5 (Feb 2025: better on 0 of 7 shared evaluations); GPT-5.1 (Nov 2025: better on 0 of 5 shared evaluations). Their rows stay in the CSV.