Development share: what labs state versus what their public runs show
Two numbers per lab. The stated share is a lab’s own accounting of all compute, where one exists. The logged share is what this report can reconstruct run by run from public records, and is always a floor because small ablations and failed runs are missing. The gap between the two columns is the survivorship problem measured.
| 0 |
Ai2 Olmo 3 (whole pipeline) |
82% |
Morrison et al. 2026: 6.85M of 8.34M H100-hours |
reported |
| 1 |
Ai2 OLMo 2 era |
50% |
Morrison et al. 2025: development ~50% of trai... |
reported |
| 2 |
Meta LLaMA 1 |
70% |
2048 A100 x ~5 months vs 1.77M A100-hours rele... |
derived |
| 3 |
EleutherAI Pythia |
50% |
discarded v0 suite: 544,280 total vs 272,140 r... |
reported |
| 4 |
Marin (535B ladder only) |
1% |
issue #8435: ladder 'only costs 1% of total co... |
reported |
| 5 |
Ai2 model ladder (OLMo 2 targets) |
1% |
Bhagia et al. 2024: ladder ~1% of the two targ... |
reported |
| 6 |
Marin (logged runs) |
31% |
49 non-final rows with FLOPs summed |
floor |
| 7 |
Ai2 OLMo (logged runs) |
0% |
21 non-final rows with FLOPs summed |
floor |
| 8 |
Meta Llama (logged runs) |
3% |
11 non-final rows with FLOPs summed |
floor |
| 9 |
Prime Intellect (logged runs) |
10% |
14 non-final rows with FLOPs summed |
floor |
| 10 |
DeepSeek (disclosed ablations) |
7% |
32 non-final rows with FLOPs summed |
floor |
Whole-programme figures cluster far above anything reconstructable from public runs. Where a lab has accounted for everything (Ai2 twice, Meta once, EleutherAI once) the non-final share is 50 to 82 percent; where we can only sum public runs it is 0 to 15 percent. The two 1 percent rows are a different object: the cost of a predictive ladder alone, which is small by design and is not the development budget.
How well experiments predicted the final run
| 0 |
Cerebras-GPT 13B |
loss |
111M to 6.7B fit, 3.7x extrapolation |
within 0.5% (lab); see chapter for our refit r... |
reported |
| 1 |
Marin MoE 1e21 test |
loss |
1e18 to 1e20 IsoFLOP fit, 10x extrapolation |
-0.0027 log units (2.599 measured vs 2.606 pre... |
reported |
| 2 |
Marin Delphi 1e23 |
loss |
7 IsoFLOP optima 3e18 to 3e20, 300x extrapolation |
within 0.2% (pre-registered) |
snippet |
| 3 |
Ai2 OLMo 2 7B-4T and 13B-5T |
task accuracy |
16 ladder runs 190M to 1.3B, ~1% of target com... |
within ~2 points on 4 tasks, ~4 points average... |
reported |
| 4 |
Meta Llama 3 405B |
compute-optimal size |
40M to 16B at 6e18 to 1e22, 3,800x extrapolation |
predicted 402B / 16.55T; trained 405B / 15.6T |
reported |
| 5 |
DeepSeek LLM 7B and 67B |
loss |
~80 IsoFLOP runs at 1e17 to 3e20, ~1,000x extr... |
described as 'well-predicted'; no numerical er... |
reported |
| 6 |
Prime Intellect INTELLECT-2/3 |
RL reward |
1.5B, 4B, 7B proxies for 32B and 106B |
qualitative transfer only; no fitted relation |
reported |
Every pretraining lab that published a ladder also reports that it worked, at extrapolations from 4x to 3,800x. That is consistent with ladders being cheap and reliable for loss and for compute-optimal sizing, and it is also what a lab would choose to publish. Task-accuracy prediction (Ai2) is noticeably worse than loss prediction, and RL has no quantitative version yet.
Marginal return to compute at the final run
Elasticity of loss with respect to compute, d log L / d log C, evaluated on each lab’s own law at its final run. These are small numbers because the fitted irreducible terms are large, and they shrink with scale.
| 0 |
Cerebras-GPT law at 13B (2.3e22 reported FLOPs) |
-0.0501 |
lab law |
| 1 |
Marin MoE law at the 1e21 test |
-0.0363 |
lab law |
| 2 |
Marin MoE law at the 2.7e24 hero run |
-0.0217 |
lab law, far extrapolation |
| 3 |
Our refit of Cerebras-GPT at 13B |
-0.0479 |
our fit, n=6 |
Mapping back to the elasticity paper
| Experimental compute share S_E |
Stated whole-programme shares of 50 to 82 percent (Ai2, Meta, EleutherAI); public-run floors of 0 to 15 percent |
Four labs with a stated figure; more needed |
| Algorithmic efficiency growth g_A and scale bias |
Cross-release overlays exist for Llama 1 to 3 (MMLU vs compute); OLMo 2 and 3 final train losses are now in runs.csv but sit on different data mixes, so the loss overlay needs the c4_en eval loss (logged only for Olmo 3) or a common benchmark |
Needs a common held-out loss or benchmark |
| Marginal return to compute at the frontier |
Elasticities from lab laws, about -0.03 to -0.06 at the largest fitted points |
Two labs; DeepSeek’s law pending |
| Informativeness of experiments |
Residuals from 0.2 to 0.5 percent in loss at 4x to 300x extrapolation; 2 to 4 points on tasks |
Five labs |
What would move these numbers most
- Loss values for the experimental runs. The final-run losses for OLMo, Marin and Pythia are now pulled from W&B (
data/losses/), which gives Pythia a fitted curve; for OLMo and Marin the ladder and ablation rows still carry no loss, because the OLMo ladder project is private and Marin’s ablation sizes are not yet enumerated, so their panels show final points only.
- The Morrison et al. (2026) split of Olmo 3’s 6.85M development hours across pretraining, midtraining, post-training and synthetic data.
- Any second lab publishing a whole-programme accounting like Ai2’s; Marin is the obvious ask because its issue tracker already contains the runs.