| RE-Bench (METR): 7 AI R&D tasks (pre-release RE-Bench) |
|
“optimize a kernel’, ’GPT-2 Chat RL … scaffolding for rust codecontests” (2 h per task; 5 runs per task; METR agent) |
|
|
|
|
|
|
|
|
|
|
| RE-Bench (METR): 4-task modified subset run by Anthropic |
|
|
0.2386, ‘well below human performance’ (triton_cumsum, rust_codecontests_inference, restricted_mlm, fix_embedding; internal infra, not comparable to METR’s scores) |
|
|
|
|
|
|
|
|
|
| RE-Bench (METR): METR 5-task subset with ground-truth scoring |
|
|
“impressive AI R&D capabilities” (32 h total time budget per task; AIDE scaffold; METR) |
|
|
|
|
|
|
|
|
|
| Internal AI Research Evaluation Suite 1 |
|
|
“notably far from thresholds” |
|
|
|
|
|
“Claude Mythos Preview clears the 4h and 8h thresholds on all tasks, and the 40h threshold on 2/3 of the tasks. … We take the suite’s saturation as the expected outcome for a model …” (standard scaffold; bounded [0-1] tasks no longer reported) |
|
|
“Recent models have crossed the highest human baselines for many of the automated task-based AI R&D evaluations described in Section 8.3 of the Claude Opus 4.6 System Card, and …” |
| Internal AI Research Evaluation Suite 1: Kernel optimization |
|
|
~16x (hard variant (as cited in the Claude 4 card)) |
72.65x (hard variant; thresholds 4x to 100x by variant) |
58.47x (Opus 4 72.65x); well below threshold (hard variant) |
108.64x (Opus 4.1 66.86x, Opus 4 48.92x); crossed hard-variant threshold for the first time (hard variant; thresholds 4x to 100x) |
252.42x; mean above 100x; first to cross 200x (~8 h) (hard variant; threshold 100x; 4x ~1 h, 200x ~8 h, 300x ~40 h human effort) |
190x standard scaffold; 427x experimental scaffold (300x ~40 expert-hours) (hard variant; threshold 100x; 4x ~1 h, 200x ~8 h, 300x ~40 h human effort) |
399.42x (Opus 4.6 190x standard / 427x experimental) (hard variant; standard scaffold; threshold 100x; 4x ~1 h, 200x ~8 h, 300x ~40 h human effort) |
430.93x (Mythos Preview 399.42x; Opus 4.7 371.75x) (hard variant; standard scaffold; threshold 100x; 4x ~1 h, 200x ~8 h, 300x ~40 h human effort) |
449.46x, new record (Mythos 5 430.93x) (hard variant; standard scaffold; threshold 100x; 4x ~1 h, 200x ~8 h, 300x ~40 h human effort) |
|
| Internal AI Research Evaluation Suite 1: LLM training |
|
|
“legitimate 7x speedup (human baseline is 4x)” (distribution of max speedup per trial (Fig 22)) |
2.993x (expert reference 4x) (threshold 4x (~4 to 8 h)) |
2.837x (Opus 4 2.993x; expert 4x) (threshold 4x (~4 to 8 h)) |
5.5x (Opus 4.1 2.5x, Opus 4 2.8x in the same run); first to cross 4x (threshold 4x (~4 to 8 h)) |
16.53x (threshold 4x ~4 to 8 h) (threshold 4x (~4 to 8 h)) |
34x (threshold 4x) (threshold 4x (~4 to 8 h)) |
51.91x as printed; corrected to 61.79x in the Opus 4.7 card (aggregation bug); Fable 5 card fixed-CPU re-run 60.81x (threshold 4x (~4 to 8 h)) |
69.61x (Mythos Preview re-run 60.81x); hard variant 8.36x (per Opus 5 card) (fixed-CPU configuration (June 2026 re-run of all recent models)) |
easy 68.54x (Mythos 5 69.61x); new hard variant 14.19x (Mythos 5 8.36x) (fixed-CPU; hard variant starts from the already-optimised reference) |
|
| Internal AI Research Evaluation Suite 1: Quadruped RL |
|
|
0.08 (human baseline 1.0) (continuous control task) |
1 run of 30 crossed threshold (1.25); generally well below (easier variant, no hyperparameters) |
1 of 30 trials above threshold (1.183; Opus 4 1.25) (easier variant) |
one run crossed threshold (1.302; Opus 4.1 1.005, Opus 4 0.769); median well below (easier variant) |
19.48 (no-hparam) and 19.16 (no-reward-function); threshold 12 ~4 h; median also above (threshold 12 (~4 h); normalised to 1 in plots) |
20.96 (no-hparam) and 21.99 (no-reward-function); threshold 12 ~4 h; median above (threshold 12 (~4 h); normalised to 1 in plots) |
30.87 (threshold 12 ~4 h) (threshold 12 (~4 h); normalised to 1 in plots) |
29.54 (Sonnet 5 and Opus 5 cards list 29.55); threshold 12 (threshold 12 (~4 h); normalised to 1 in plots) |
31.3, new record (Mythos 5 29.55); threshold 12 (threshold 12 (~4 h); normalised to 1 in plots) |
|
| Internal AI Research Evaluation Suite 1: Text-based RL |
|
|
|
0.625 (threshold 0.9) (threshold 0.9 (~8 h)) |
0.425 (Opus 4 0.625; threshold 0.9) (threshold 0.9 (~8 h)) |
0.850 (Opus 4.1 0.475, Opus 4 0.675; threshold 0.9) (threshold 0.9 (~8 h)) |
0.975 best (threshold 0.9 ~8 h); mean still below (threshold 0.9 (~8 h)) |
0.975 best (threshold 0.9 ~8 h); mean well above threshold for the first time (threshold 0.9 (~8 h)) |
|
|
|
|
| Internal AI Research Evaluation Suite 1: Time-series forecasting |
|
|
|
easy 6.02 (0.818); hard 6.15 (0.815); below threshold on hard (easy and hard variants) |
6.541 (Opus 4 6.15); above threshold (not met) (hard variant) |
easy 5.91; hard 5.30, ‘just meeting the threshold for the harder problem’ |
easy 5.66 (met; baseline 5.8 ~4 h); hard 5.71 (not met; baseline 5.3) (easy and hard variants) |
hard 5.86 (not met; baseline 5.3); medium 5.76 (met in one trial; baseline 5.8 ~8 h) (hard and medium variants) |
4.55 (threshold <5.3 ~40 h; met) (hard variant) |
4.51 (threshold <5.3 ~40 h; met) (hard variant) |
5.68 (threshold <5.3; not met; Mythos 5 4.51) (hard variant) |
|
| Internal AI Research Evaluation Suite 1: Novel compiler |
|
|
|
64.44% basic; 9.44% advanced (threshold 90%) |
74.4% basic; 6.81% advanced (Opus 4 64.44% / 9.44%) |
81.7% basic; 29.7% advanced (Opus 4.1 75.6% / 8.33%); below 90% threshold |
93.7% basic; 69.37% complex (threshold 90% ~40 h) (threshold 90% on complex tests (~40 h)) |
98.2% basic; 65.83% complex (threshold 90% ~40 h) (threshold 90% on complex tests (~40 h)) |
77.2% complex (threshold 90% ~40 h; not met) (threshold 90% on complex tests (~40 h)) |
85.3% complex (threshold 90%; not met, the only unsaturated task) (threshold 90% on complex tests (~40 h)) |
80.91% complex (Mythos 5 85.3%; threshold 90%; not met) (threshold 90% on complex tests (~40 h)) |
|
| Internal AI Research Evaluation Suite 2 |
|
|
0.440 (rule-out threshold 0.6) |
0.355 (Sonnet 3.7 0.440; rule-out threshold 0.6) (weighted average over tasks) |
“We did not run Internal AI Research Evaluation Suite 2 or the internal model use survey for this incremental release.” |
0.514 (Opus 4.1 0.507; Sonnet 4 0.403; threshold 0.6) |
0.604 (threshold 0.6, narrowly surpassed) (rule-out threshold 0.6) |
0.6124 (Opus 4.5 0.604; threshold 0.6) (rule-out threshold 0.6) |
0.65 (Opus 4.6 0.612; threshold 0.6) |
|
|
|
| Internal AI R&D productivity survey |
|
|
|
“some productivity gains” (4 members of technical staff; >2 h each on their own AI R&D tasks) |
“We did not run Internal AI Research Evaluation Suite 2 or the internal model use survey for this incremental release.” |
“0/7 researchers believed that the model could completely automate the work of a junior ML researcher. … One participant estimated an overall productivity boost of ~100%, and …” (7 staff, primarily from the top 30 internal Claude Code users; >2 h each; early snapshots) |
median 100%; mean 220%; 9 of 18 at 100% or more (18 staff, primarily from the top 30 internal Claude Code users; >2 h each) |
mean 152%; median 100%; range 30 to 700% (16 respondents, deliberately including general research and infrastructure roles rather than only power users) |
geometric mean about 4x; wide distribution (informal opt-in Slack poll, 130 responses; ‘how much did AI-powered systems accelerate your work output over the past week’; options 1x to >=20x) |
“These are drawn from a sample of 886 day-to-day uses of a nearly-final version of the model. … Cluster: states an unverified guess as fact (41/886) … Cluster: Claude reported work …” |
“If Claude Opus 5 represented a practical jump in AI R&D capability larger than its small margin over Mythos 5, we would expect to observe more signs of internal adoption (in our …” |
“[METR:] For instance, we expect that [Mythos 5.1] likely provides a higher productivity uplift than Mythos Preview.” |
| Internal survey: drop-in replacement for an entry-level (L4) researcher |
|
|
|
0 of 4 say it could completely automate a junior ML researcher (4 respondents) |
|
0 of 7 say it could completely automate a junior ML researcher (7 respondents) |
0 of 18 believe it could fully automate an entry-level remote-only research or engineering role (18 respondents) |
raw: 2 ‘already possible’, 3 ‘likely within 3 months of scaffolding’, 11 ‘unlikely’; after clarification 0 of 16 (16 respondents; question: drop-in L4 replacement with 3 months of scaffolding/tooling, >50% probability) |
1 of 18 ‘already here’; 4 of 18 give 50% odds within 3 months of scaffolding; consistent with not being a drop-in L4 (18 respondents; Google Forms; 10 dimensions plus ‘literally drop in replace L4s’; no clarifying dialogue this time) |
|
|
|
| Internal AI R&D acceleration measure |
|
|
|
|
|
|
|
“a significant number of real-world R&D tasks which the model failed to complete, but would have been straightforward for entry-level human researchers” (internal Claude Code usage and developer productivity; listed as a preliminary AI R&D-5 evaluation) |
“On the current pipeline, the slope ratio lands between 1.86× and 4.3× depending on the choice of breakpoint. … The slope measurement tells us that Anthropic’s capability …” (Anthropic ECI (fork of Epoch’s ECI) two-piece linear fit; three breakpoints) |
“meaningful acceleration … substantial in specific, well-scoped tasks” (internal measures combining productivity estimates with usage/output indicators (not published)) |
“no discontinuity coinciding with Opus 5” (internal measures only partially published) |
“internal usage of recent AI models has been a key factor in maintaining the current rate of progress” (cross-cutting metrics (many redacted in the public Risk Report); indicators lag) |
| Internal AI R&D acceleration measure: CoBench (internal root-cause diagnosis on historical Anthropic infrastructure) |
|
|
|
|
|
|
|
|
|
45.6% (re-run reported in the Fable 5.1 / Mythos 5.1 card; ‘computed from a slightly larger set of problems’ than the August 2026 Risk Report) |
59.6% (re-run reported in the Fable 5.1 / Mythos 5.1 card; ‘computed from a slightly larger set of problems’ than the August 2026 Risk Report) |
57.6% (model-graded rubric; snapshot of codebase, logs, messaging and docs at a past timestamp) |
| METR general autonomy time horizon |
|
|
about 55 human-minutes (METR GAC suite (97 tasks); METR agent) |
|
|
|
|
|
|
|
|
|
| METR external evaluation: External AI R&D testing (METR, Epoch AI and other partners) |
|
|
|
|
|
|
|
|
“Claude Mythos Preview rediscovered 4 of 5 key insights, while Claude Opus 4.6 discovered just 2 of 5 key insights. There was no direct baseline for discovering these insights.” (early snapshot; task well-scoped and easy to verify) |
|
|
|
| METR external evaluation: 38 hardest METR software tasks incl. R&D-centred tasks; one open-ended research task |
|
|
|
|
|
|
|
|
|
“likely unable to fully and reliably automate R&D for frontier projects spanning multiple weeks” (pre-release snapshot of Mythos 5; API access) |
|
|
| METR external evaluation: Budget NanoGPT Speedrun |
|
|
|
|
|
|
|
|
|
|
|
“particularly impressive … consistent with an above-trend increase in specific capabilities relevant for AI R&D” (Mythos 5.1; API access with visible reasoning over 10 business days; plus 212 benchmark scores and an interview provided by Anthropic) |
| METR external evaluation: Sunlight and LMCA |
|
|
|
|
|
|
|
|
|
|
|
“However, we still observed [Mythos 5.1] achieving subexpert performance in Sunlight and LMCA.” (Mythos 5.1) |
| SWE-bench Verified |
33.4% (Anthropic scaffold (bash + file-edit tools); reported retrospectively in the Oct 2024 addendum) |
49.0% (pass@1; bash + file-edit tool scaffold (per research post)) |
62.3% (full 500; pass@1; bash, editor and thinking tools)†
70.3% (n=489 subset; parallel sampling, regression-test filtering and internal scoring model)* |
72.5% (full 500; no extended thinking)
79.4% (parallel test-time compute: multiple attempts, regression-test filtering, internal scoring model) |
74.5% (full 500; no extended thinking)* |
77.2% (full 500; mean of 10 trials; 200K thinking budget; no test-time compute; minor prompt addition)
82.0% (parallel test-time compute)* |
80.9% (80.6% with 64k thinking) (full 500; no extended thinking; mean of 5 trials; 200k context) |
80.84% (81.4% with prompt modification) (full 500; mean of 25 trials; adaptive thinking, max effort) |
93.9% (full 500; mean of 5 trials; adaptive thinking, max effort) |
Fable 5 95.0%; Mythos 5 95.5% (full 500; mean of 5 trials; adaptive thinking, max effort; Fable 5 scores reflect production safeguards with fallback to Opus 4.8) |
96.0% (full 500; mean of 5 trials; adaptive thinking, max effort) |
“8.2 SWE-bench Pro, Multilingual, and Multimodal … We report three variants, where each score is an average over five trials: SWE-bench Pro … SWE-bench Multilingual … SWE-bench …” |
| SWE-bench Verified: hard subset (RSP checkpoint) |
|
|
9.65/42 (23%; threshold >50%) (42 tasks estimated >1 h; mean over 20 attempts) |
16.6/42 (39.5%; threshold >50%) (42 tasks; mean over 10 runs) |
18.4/42 (43.8%; threshold >50%) (pass@1) |
20.4/45 (45.3%; threshold 22.5/45) (45 tasks (42 in earlier cards)) |
21/45 (threshold 22.5/45); ‘just below saturation’ |
21.24/45 (threshold 22.5/45) |
|
|
|
|
| SWE-Bench Pro |
|
|
|
|
|
|
|
|
77.8% (mean of 5 trials) |
Fable 5 80.0%; Mythos 5 80.3% (mean of 5 trials) |
79.2% (mean of 5 trials) |
81.2% (Fable 5 80%; Opus 5 79.2%) (mean of 5 trials) |
| Terminal-Bench |
|
|
|
43.2% (launch-time Terminal-bench (May 2025); no extended thinking; harness not stated in post)* |
46.5% (Terminus 2 harness; version ambiguous (listed under ‘Terminal-bench 2.0’ in the Opus 4.5 card Table 2.3.A))
43.3% (launch-time Terminal-Bench; no extended thinking)‡ |
50.0% (Terminus 2 harness; launch-time Terminal-Bench (Sep 2025); listed under ‘Terminal-bench 2.0’ in the Opus 4.5 card table)
51.0% (v2.0; Terminus-2 in Harbor) |
59.27% +/- 1.34% (57.76% with 64k thinking) (v2.0; Terminus-2 in Harbor; 128k thinking budget; 1,335 trials) |
65.4% (max effort; 61.1% medium, 55.1% low) (v2.0; Terminus-2 in Harbor; 89 tasks x 15 = 1,335 trials) |
82% (92.1% with TB 2.1 fixes and 4 h timeouts) (v2.0; Terminus-2 in Harbor; 445 trials; max effort; 1M token budget) |
Fable 5 84.3% (20.9% of trials fell back to Opus 4.8); Mythos 5 88.0% (v2.1; mini-SWE-agent harness (switched from Terminus-2); high effort; 445 trials)
Fable 5 42.0% (public leaderboard 44.5%) (v4.0 (66 tasks); Claude Code –bare; max effort; 15 trials per task) |
“89.1%’ appears in the card only as a ’mean claim coverage” (v2.1)
52.3% (public leaderboard 51.8%) (v4.0; Claude Code –bare; max effort; 15 trials per task) |
Fable 5.1 55.8%; Mythos 5.1 60.9% (Opus 5 52.3%; Fable 5 42.0%) (v4.0 (66 tasks); Claude Code –bare; max thinking effort; 15 trials per task (10 for Mythos 5.1); SE +/-1.6-2) |
| Terminal-Bench-Science |
|
|
|
|
|
|
|
|
|
Fable 5 24.7% (public leaderboard 21.4%) (v0.1 (70 tasks); Claude Code –bare; max effort; 10 trials per task) |
29.0% (public leaderboard 30.0%) (v0.1; Claude Code –bare; max effort; 12 trials per task) |
52.6% (Opus 5 29.0%; Fable 5 24.7%) (v0.1 (70 tasks); Fable 5.1; Claude Code –bare; max effort; 10 trials per task; SE +/-3.5-4.5) |
| FrontierBench v0.1 (Terminal-Bench successor) |
|
|
|
|
|
|
|
|
|
|
44.4% at xhigh effort (summary table 43.3%; Fable 5 33.7%; Sonnet 5 17%; Opus 4.8 18.7%) (74 tasks; mini-SWE-agent on GKE; 5 attempts per task; 4% of trials fell back to Opus 4.8 via safety classifiers) |
|
| Internal agentic coding evaluation |
64% (Claude 3 Opus 38%) (agentic loop; sandboxed; hidden tests) |
|
|
|
|
|
|
|
|
|
|
|
| METR data deduplication (RSP checkpoint) |
|
|
4/30 trials above threshold; median F1 70.2% (rule-out threshold 20% of trials) |
15/46 trials (32.6%) above threshold; median F1 76.2%; threshold crossed |
|
|
|
|
|
|
|
|
| AI R&D threshold determination |
|
“as an indicative precursor … increased proficiency in autonomy-relevant software engineering tasks” |
“sufficiently far away from the ASL-3 capability thresholds … notably far from thresholds” |
“does not pose the autonomy risks specified in our threat model” |
“Claude Opus 4.1 demonstrates incremental improvements compared to Claude Opus 4, consistent with a model that does not cross the”notably more capable" threshold from our RSP …” |
“remained below the ASL-4 threshold of fully automating the work of an entry-level, remote-only researcher” |
“roughly reached the pre-defined thresholds for straightforward ASL-4 rule-out … plausible that models equipped with highly effective scaffolding may not be very far away” |
“rule-out case is more tenuous than for any previous model … gray zone’; ’plausible that models equipped with highly effective scaffolding may be close” |
“would include several past models … with less confidence than for any prior model” |
“remains well below the capability level of our human engineers … not close to being able to substitute for our Research Scientists and Research Engineers” |
“not close to substituting for Research Scientists and Engineers” |
“well below the capability of our human researchers and engineers … in line with current trends … provides a meaningful acceleration to AI R&D efforts in many circumstances” |
