8  Cross-lab synthesis

8.1 Development share: what labs state versus what their public runs show

Two numbers per lab. The stated share is a lab’s own accounting of all compute, where one exists. The logged share is what this report can reconstruct run by run from public records, and is always a floor because small ablations and failed runs are missing. The gap between the two columns is the survivorship problem measured.

lab / programme non-final share basis confidence
0 Ai2 Olmo 3 (whole pipeline) 82% Morrison et al. 2026: 6.85M of 8.34M H100-hours reported
1 Ai2 OLMo 2 era 50% Morrison et al. 2025: development ~50% of trai... reported
2 Meta LLaMA 1 70% 2048 A100 x ~5 months vs 1.77M A100-hours rele... derived
3 EleutherAI Pythia 50% discarded v0 suite: 544,280 total vs 272,140 r... reported
4 Marin (535B ladder only) 1% issue #8435: ladder 'only costs 1% of total co... reported
5 Ai2 model ladder (OLMo 2 targets) 1% Bhagia et al. 2024: ladder ~1% of the two targ... reported
6 Marin (logged runs) 31% 49 non-final rows with FLOPs summed floor
7 Ai2 OLMo (logged runs) 0% 21 non-final rows with FLOPs summed floor
8 Meta Llama (logged runs) 3% 11 non-final rows with FLOPs summed floor
9 Prime Intellect (logged runs) 10% 14 non-final rows with FLOPs summed floor
10 DeepSeek (disclosed ablations) 7% 32 non-final rows with FLOPs summed floor

Whole-programme figures cluster far above anything reconstructable from public runs. Where a lab has accounted for everything (Ai2 twice, Meta once, EleutherAI once) the non-final share is 50 to 82 percent; where we can only sum public runs it is 0 to 15 percent. The two 1 percent rows are a different object: the cost of a predictive ladder alone, which is small by design and is not the development budget.

8.2 How well experiments predicted the final run

final run predicted quantity experiments used outcome confidence
0 Cerebras-GPT 13B loss 111M to 6.7B fit, 3.7x extrapolation within 0.5% (lab); see chapter for our refit r... reported
1 Marin MoE 1e21 test loss 1e18 to 1e20 IsoFLOP fit, 10x extrapolation -0.0027 log units (2.599 measured vs 2.606 pre... reported
2 Marin Delphi 1e23 loss 7 IsoFLOP optima 3e18 to 3e20, 300x extrapolation within 0.2% (pre-registered) snippet
3 Ai2 OLMo 2 7B-4T and 13B-5T task accuracy 16 ladder runs 190M to 1.3B, ~1% of target com... within ~2 points on 4 tasks, ~4 points average... reported
4 Meta Llama 3 405B compute-optimal size 40M to 16B at 6e18 to 1e22, 3,800x extrapolation predicted 402B / 16.55T; trained 405B / 15.6T reported
5 DeepSeek LLM 7B and 67B loss ~80 IsoFLOP runs at 1e17 to 3e20, ~1,000x extr... described as 'well-predicted'; no numerical er... reported
6 Prime Intellect INTELLECT-2/3 RL reward 1.5B, 4B, 7B proxies for 32B and 106B qualitative transfer only; no fitted relation reported

Every pretraining lab that published a ladder also reports that it worked, at extrapolations from 4x to 3,800x. That is consistent with ladders being cheap and reliable for loss and for compute-optimal sizing, and it is also what a lab would choose to publish. Task-accuracy prediction (Ai2) is noticeably worse than loss prediction, and RL has no quantitative version yet.

8.3 Marginal return to compute at the final run

Elasticity of loss with respect to compute, d log L / d log C, evaluated on each lab’s own law at its final run. These are small numbers because the fitted irreducible terms are large, and they shrink with scale.

curve and point d log L / d log C source
0 Cerebras-GPT law at 13B (2.3e22 reported FLOPs) -0.0501 lab law
1 Marin MoE law at the 1e21 test -0.0363 lab law
2 Marin MoE law at the 2.7e24 hero run -0.0217 lab law, far extrapolation
3 Our refit of Cerebras-GPT at 13B -0.0479 our fit, n=6

8.4 Mapping back to the elasticity paper

Paper primitive What this report gives Status
Experimental compute share S_E Stated whole-programme shares of 50 to 82 percent (Ai2, Meta, EleutherAI); public-run floors of 0 to 15 percent Four labs with a stated figure; more needed
Algorithmic efficiency growth g_A and scale bias Cross-release overlays exist for Llama 1 to 3 (MMLU vs compute); OLMo 2 and 3 final train losses are now in runs.csv but sit on different data mixes, so the loss overlay needs the c4_en eval loss (logged only for Olmo 3) or a common benchmark Needs a common held-out loss or benchmark
Marginal return to compute at the frontier Elasticities from lab laws, about -0.03 to -0.06 at the largest fitted points Two labs; DeepSeek’s law pending
Informativeness of experiments Residuals from 0.2 to 0.5 percent in loss at 4x to 300x extrapolation; 2 to 4 points on tasks Five labs

8.5 What would move these numbers most

  1. Loss values for the experimental runs. The final-run losses for OLMo, Marin and Pythia are now pulled from W&B (data/losses/), which gives Pythia a fitted curve; for OLMo and Marin the ladder and ablation rows still carry no loss, because the OLMo ladder project is private and Marin’s ablation sizes are not yet enumerated, so their panels show final points only.
  2. The Morrison et al. (2026) split of Olmo 3’s 6.85M development hours across pretraining, midtraining, post-training and synthetic data.
  3. Any second lab publishing a whole-programme accounting like Ai2’s; Marin is the obvious ask because its issue tracker already contains the runs.