
3 Ai2 OLMo
Ai2 publishes data, code, recipes, W&B logs, and intermediate checkpoints for OLMo 2 and OLMo 3, plus a model-ladder paper on predicting task performance from small runs. Two accounting papers by Morrison et al. give the development vs. final-run split directly: roughly 50/50 for the OLMo 2 era (ICLR 2025) and 82/18 for the whole Olmo 3 pipeline (2026). Working notes with sources are in data/notes/olmo.md.
3.1 The number that matters: development vs. final-run compute
Morrison et al. (2026) account for the entire Olmo 3 pipeline (7B and 32B, Instruct and Think). Of 8.34M H100-hours, development (experimentation, failed runs, ablations, synthetic data) took 6.85M and the final runs 1.49M. That is an experiment-to-final ratio near 4.6 to 1, up from about 1 to 1 in their OLMo 2-era paper.
The rows below are the runs we can reconstruct individually. Final-run losses are pulled from Ai2’s public W&B projects (data/losses/ai2-olmo/, stage-1 runs stitched from 11 to 75 restart segments each). The loss column is the training cross-entropy at the end of each stage (median of the last 20 logged points), the only loss both generations log; Olmo 3 stage 1 also logs in-loop held-out losses, and those are the comparable numbers: c4_en CE 2.469 (7B, 5.93T) and 2.301 (32B, 5.70T), Pile 1.776 and 1.639. Training loss is on the training mix, so OLMo 2 (Dolma) and Olmo 3 (Dolma 3) stage-1 values, 2.23 / 2.11 and 1.85 / 1.59, are not one curve, and the midtraining and long-context stages read lower still because the anneal mixes are easier. The panel therefore draws points without a fitted law, and the ladder rows still carry no loss (the ai2-llm/olmo-ladder project is private); the run-size map shows the ladder and the released models on the parameters-vs-tokens plane.
3.2 Run sizes, compute split, timeline



Experimental share of logged FLOPs: 0% (a lower bound: unlogged and failed runs are missing).


The logged-compute split above counts only runs we have individually, by 6ND FLOPs: the 16 ladder runs are about 1% of the compute of the two OLMo 2 targets they predicted, and the ablation series have no sizes attached, so the per-run picture shows a final-run-dominated budget. The lab’s own accounting, which includes everything, says the opposite. The gap between those two bars is the survivorship problem in one picture.
3.3 Published scaling experiments
| project | date | stage | intervention | model_sizes | compute_range | metric | fitted_exponent | exponent_source | raw_data_available | source_url | notes | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | olmo-ladder | 2024-12 | pretraining | Model ladder: 4 sizes x 4 Chinchilla multiples... | 190M,370M,760M,1.3B (non-embedding) | ~1% of the compute of the targets (7B-4T, 13B-... | Task loss (per-task CE on correct continuation... | Per-task A,B,alpha,beta,E fitted (values in pa... | paper tables | logs | [src](https://arxiv.org/abs/2412.04403) | Prediction error on 7B-4T and 13B-5T: <2 abs. ... |
| 1 | olmo2 | 2024-11 | pretraining | Architecture/stability: RMSNorm with reordered... | 1B (ablation); 7B (stability comparison) | not stated | Loss spikes; L2 gradient-norm growth and spiki... | figure_only | [src](https://arxiv.org/abs/2501.00656) | Neither reordered norm nor QK-norm alone impro... | ||
| 2 | olmo2 | 2024-11 | midtraining | Microannealing: 19 anneal runs testing individ... | not stated (likely 7B) | 130B tokens total (< 3 x 50B full anneals) | OLMES downstream accuracy after anneal | numbers_in_text | [src](https://arxiv.org/abs/2501.00656) | Effects found to persist when sources mixed. | ||
| 3 | olmo2 | 2024-11 | midtraining | Anneal budget (50B/100B/300B) and model soupin... | 7B,13B,32B | 7B: 3x50B; 13B: 3x100B+1x300B; 32B: 2x100B+1x300B | Downstream evals of souped vs individual check... | numbers_in_text | [src](https://github.com/allenai/OLMo-core/blo... | Souping consistently beat individual checkpoin... | ||
| 4 | olmo2 | 2025-03 | pretraining | Compute-accounting of the OLMo 2 development p... | 20M to 13B | Models trained on up to 5.6T tokens; total 493... | Energy / carbon / water; development share of ... | numbers_in_text | [src](https://arxiv.org/abs/2503.05804) | Development (hyperparameter tuning, ablations ... | ||
| 5 | olmo3 | 2025-11 | midtraining | Microanneals: 5B target-source + 5B web (10B) ... | not stated | 10B tokens per microanneal; count not stated | Base-eval deltas vs web-only baseline | figure_only | [src](https://arxiv.org/abs/2512.13961) | Part of two-part framework: lightweight distri... | ||
| 6 | olmo3 | 2025-11 | midtraining | Integration tests: full 100B midtraining annea... | 7B (Olmo 3 7B base checkpoint) | 100B tokens per test | Base quality and post-trainability (SFT/RL) of... | figure_only | [src](https://arxiv.org/abs/2512.13961) | Final midtraining mixes: OLMo-midtraining-mix-... | ||
| 7 | olmo3 | 2025-11 | pretraining | Architecture: 8192 pretraining context (vs 409... | 7B,32B | ablation scale not stated | Throughput and downstream evals | figure_only | [src](https://arxiv.org/abs/2512.13961) | Ablation details not retrieved. | ||
| 8 | olmo3 | 2025 | pretraining | Run management: 7B pretraining schedule extend... | 7B,32B | ~196B 32B tokens (~0.4e22 FLOPs) trained past ... | n/a | logs | [src](https://github.com/allenai/OLMo-core/blo... | From official scripts OLMo-3-1025-7B-pretrain-... | ||
| 9 | olmo3 | 2026-05 | post-training/RL | Compute-accounting of full Olmo 3 pipeline (Mo... | 7B,32B (Instruct, Think) | 8.34M GPU-h total; 6.85M development (82.2%); ... | GPU-hours, energy, carbon, water per stage | numbers_in_text | [src](https://arxiv.org/abs/2605.01158) | Think post-training 17x Instruct energy (32B: ... | ||
| 10 | olmo-hybrid | 2026-04 | pretraining | Chinchilla-style scaling laws for transformer ... | 7 sizes (values not retrieved) x D/N in {10,20... | Compute-optimal projections evaluated 1e18-1e2... | Mean of 11 held-out validation CE losses (C4, ... | alpha, beta statistically indistinguishable ac... | paper text/tables (via search summaries) | numbers_in_text | [src](https://arxiv.org/abs/2604.03444) | Implied token savings 1.3-1.9x from 1B to 70B;... |
| 11 | olmo-hybrid | 2026-03 | pretraining | 1B-scale architecture sweep before 7B: hybrid ... | 1B | not stated | Bits-per-byte on evaluation slices | figure_only | [src](https://allenai.org/blog/olmohybrid) | Ordering: hybrid GDN > pure GDN > transformer ... |
3.4 What the lab says about itself
| project | claim | value | unit | source_url | quote | notes | |
|---|---|---|---|---|---|---|---|
| 0 | olmo3 | olmo3-all-development-aggregate: GPU-hours | 6.849711e6 | H100-hours | [src](https://arxiv.org/abs/2605.01158) | Total ... Dev 6,849,711 (Table 5) | Aggregate, not a single run. Dev GPU-h by stag... |
| 1 | olmo3 | olmo3-all-final-runs-aggregate: GPU-hours | 1.485329e6 | H100-hours | [src](https://arxiv.org/abs/2605.01158) | Total (final runs) 1,485,329 | Reported directly in https://arxiv.org/html/26... |
| 2 | olmo2 | olmo2-development-aggregate: GPU-hours | 6.8e5 | GPU-hours | [src](https://arxiv.org/abs/2503.05804) | Table 1: development total 680k GPU hours, 813... | Morrison et al. 2025 (OLMo 2-era accounting), ... |
3.5 Key numbers
| project | run_id | run_type | stage | params_total | tokens | flops | flops_method | gpu_hours | loss | benchmark_name | benchmark_value | confidence | source_url | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | olmo2 | olmo2-1B-stage1 | final | pretraining | 1.50e+09 | 4.00e+12 | 3.60e+22 | 6ND derived | NaN | NaN | NaN | reported | [src](https://github.com/allenai/OLMo/blob/mai... | |
| 1 | olmo2 | olmo2-1B-stage2 | final | midtraining | 1.50e+09 | 5.00e+10 | 4.50e+20 | 6ND derived | NaN | NaN | NaN | reported | [src](https://github.com/allenai/OLMo/blob/mai... | |
| 2 | olmo2 | olmo2-7B-stage1 | final | pretraining | 7.30e+09 | 3.90e+12 | 1.71e+23 | 6ND derived | NaN | 2.2297 | NaN | reported | [src](https://arxiv.org/abs/2501.00656) | |
| 3 | olmo2 | olmo2-7B-stage2-soup | final | midtraining | 7.30e+09 | 1.50e+11 | 6.57e+21 | 6ND derived (3 seeds x 50B) | NaN | 1.5966 | MMLU | 63.7 | reported | [src](https://arxiv.org/abs/2501.00656) |
| 4 | olmo2 | olmo2-13B-stage1 | final | pretraining | 1.37e+10 | 5.00e+12 | 4.11e+23 | 6ND derived | NaN | 2.1078 | NaN | reported | [src](https://arxiv.org/abs/2501.00656) | |
| 5 | olmo2 | olmo2-13B-stage2-soup | final | midtraining | 1.37e+10 | 6.00e+11 | 4.93e+22 | 6ND derived (3x100B + 1x300B) | NaN | 1.5110 | MMLU | 67.5 | reported | [src](https://arxiv.org/abs/2501.00656) |
| 6 | olmo2 | olmo2-32B-stage1 (peteish32) | final | pretraining | 3.22e+10 | 6.06e+12 | 1.17e+24 | 6ND derived | 930000.0 | NaN | NaN | derived | [src](https://github.com/allenai/OLMo-core/blo... | |
| 7 | olmo2 | olmo2-32B-stage2-soup | final | midtraining | 3.22e+10 | 5.00e+11 | 9.66e+22 | 6ND derived (100B seed1110 + 100B seed2662 + 3... | NaN | NaN | MMLU | 74.9 | reported | [src](https://github.com/allenai/OLMo-core/blo... |
| 8 | olmo3 | olmo3-7B-stage1 (internal: OLMo25) | final | pretraining | 7.30e+09 | 5.93e+12 | 2.60e+23 | 6ND derived | 234475.0 | 1.8528 | NaN | reported | [src](https://github.com/allenai/OLMo-core/blo... | |
| 9 | olmo3 | olmo3-7B-stage2-midtrain | final | midtraining | 7.30e+09 | 1.00e+11 | 4.38e+21 | 6ND derived | 4848.0 | 1.2268 | NaN | reported | [src](https://github.com/allenai/OLMo-core/blo... | |
| 10 | olmo3 | olmo3-7B-stage3-longcontext | final | midtraining | 7.30e+09 | 5.00e+10 | 2.19e+21 | 6ND derived | 4781.0 | 1.2955 | MMLU (base) | 66.9 | reported | [src](https://github.com/allenai/OLMo-core/blo... |
| 11 | olmo3 | olmo3-32B-stage1 (internal: stego32-highlr-fil... | final | pretraining | 3.22e+10 | 5.50e+12 | 1.06e+24 | 6ND derived (tokens actually consumed by branc... | 1050287.0 | 1.5918 | NaN | reported | [src](https://github.com/allenai/OLMo-core/blo... | |
| 12 | olmo3 | olmo3-32B-stage2-midtrain-ingredients-1+2 | final | midtraining | 3.22e+10 | 2.00e+11 | 3.86e+22 | 6ND derived (2 x 100B) | 29058.0 | 1.0916 | NaN | reported | [src](https://github.com/allenai/OLMo-core/blo... | |
| 13 | olmo3 | olmo3-7B-Think-posttrain (DPO+RLVR) | final | post-training/RL | 7.30e+09 | 10000.0 | NaN | NaN | reported | [src](https://arxiv.org/abs/2605.01158) | ||||
| 14 | olmo3 | olmo3-7B-Instruct-posttrain (SFT+DPO+RLVR) | final | post-training/RL | 7.30e+09 | 4200.0 | NaN | NaN | reported | [src](https://arxiv.org/abs/2605.01158) | ||||
| 15 | olmo3 | olmo3-32B-Instruct-posttrain | final | post-training/RL | 3.22e+10 | 7100.0 | NaN | NaN | reported | [src](https://arxiv.org/abs/2605.01158) | ||||
| 16 | olmo3 | olmo3.1-32B-Think-extended-RL | final | post-training/RL | 3.22e+10 | 110000.0 | NaN | AIME 2025 delta | 5.6 | derived | [src](https://huggingface.co/allenai/Olmo-3.1-... | |||
| 17 | olmo-hybrid | olmo-hybrid-7B-stage1 | final | pretraining | 7.43e+09 | 5.93e+12 | 2.64e+23 | 6ND derived | NaN | NaN | NaN | reported | [src](https://github.com/allenai/OLMo-core/blo... | |
| 18 | olmo-hybrid | olmo-hybrid-7B-stage2+3 | final | midtraining | 7.00e+09 | 3.00e+11 | 1.26e+22 | 6ND derived (2x100B midtrain souped + 100B lon... | NaN | NaN | NaN | reported | [src](https://github.com/allenai/OLMo-core/blo... | |
| 19 | olmo-ladder | ladder-190M-1xC | ladder | pretraining | 1.90e+08 | 3.81e+09 | 4.35e+18 | 6ND derived; D = 20*N*multiple (Chinchilla xC) | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2412.04403) | |
| 20 | olmo-ladder | ladder-190M-2xC | ladder | pretraining | 1.90e+08 | 7.61e+09 | 8.70e+18 | 6ND derived; D = 20*N*multiple (Chinchilla xC) | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2412.04403) | |
| 21 | olmo-ladder | ladder-190M-5xC | ladder | pretraining | 1.90e+08 | 1.90e+10 | 2.17e+19 | 6ND derived; D = 20*N*multiple (Chinchilla xC) | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2412.04403) | |
| 22 | olmo-ladder | ladder-190M-10xC | ladder | pretraining | 1.90e+08 | 3.81e+10 | 4.35e+19 | 6ND derived; D = 20*N*multiple (Chinchilla xC) | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2412.04403) | |
| 23 | olmo-ladder | ladder-370M-1xC | ladder | pretraining | 3.71e+08 | 7.43e+09 | 1.65e+19 | 6ND derived; D = 20*N*multiple (Chinchilla xC) | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2412.04403) | |
| 24 | olmo-ladder | ladder-370M-2xC | ladder | pretraining | 3.71e+08 | 1.48e+10 | 3.31e+19 | 6ND derived; D = 20*N*multiple (Chinchilla xC) | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2412.04403) | |
| 25 | olmo-ladder | ladder-370M-5xC | ladder | pretraining | 3.71e+08 | 3.71e+10 | 8.27e+19 | 6ND derived; D = 20*N*multiple (Chinchilla xC) | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2412.04403) | |
| 26 | olmo-ladder | ladder-370M-10xC | ladder | pretraining | 3.71e+08 | 7.43e+10 | 1.65e+20 | 6ND derived; D = 20*N*multiple (Chinchilla xC) | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2412.04403) | |
| 27 | olmo-ladder | ladder-760M-1xC | ladder | pretraining | 7.58e+08 | 1.52e+10 | 6.90e+19 | 6ND derived; D = 20*N*multiple (Chinchilla xC) | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2412.04403) | |
| 28 | olmo-ladder | ladder-760M-2xC | ladder | pretraining | 7.58e+08 | 3.03e+10 | 1.38e+20 | 6ND derived; D = 20*N*multiple (Chinchilla xC) | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2412.04403) | |
| 29 | olmo-ladder | ladder-760M-5xC | ladder | pretraining | 7.58e+08 | 7.58e+10 | 3.45e+20 | 6ND derived; D = 20*N*multiple (Chinchilla xC) | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2412.04403) | |
| 30 | olmo-ladder | ladder-760M-10xC | ladder | pretraining | 7.58e+08 | 1.52e+11 | 6.90e+20 | 6ND derived; D = 20*N*multiple (Chinchilla xC) | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2412.04403) | |
| 31 | olmo-ladder | ladder-1.3B-1xC | ladder | pretraining | 1.28e+09 | 2.56e+10 | 1.96e+20 | 6ND derived; D = 20*N*multiple (Chinchilla xC) | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2412.04403) | |
| 32 | olmo-ladder | ladder-1.3B-2xC | ladder | pretraining | 1.28e+09 | 5.12e+10 | 3.93e+20 | 6ND derived; D = 20*N*multiple (Chinchilla xC) | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2412.04403) | |
| 33 | olmo-ladder | ladder-1.3B-5xC | ladder | pretraining | 1.28e+09 | 1.28e+11 | 9.82e+20 | 6ND derived; D = 20*N*multiple (Chinchilla xC) | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2412.04403) | |
| 34 | olmo-ladder | ladder-1.3B-10xC | ladder | pretraining | 1.28e+09 | 2.56e+11 | 1.96e+21 | 6ND derived; D = 20*N*multiple (Chinchilla xC) | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2412.04403) | |
| 35 | olmo2 | olmo2-microanneals-series | ablation | midtraining | 7.30e+09 | 1.30e+11 | 5.69e+21 | 6ND | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2501.00656) | |
| 36 | olmo2 | olmo2-1B-architecture-ablations | ablation | pretraining | 1.00e+09 | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2501.00656) | ||||
| 37 | olmo3 | olmo3-microanneal-template | ablation | midtraining | 1.00e+10 | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2512.13961) | ||||
| 38 | olmo3 | olmo3-integration-test-template | ablation | midtraining | 1.00e+11 | NaN | NaN | NaN | reported | [src](https://arxiv.org/abs/2512.13961) | ||||
| 39 | olmo-hybrid | olmo-hybrid-1B-scale-experiments | ablation | pretraining | 1.00e+09 | 1.00e+11 | 6.00e+20 | 6ND | NaN | NaN | NaN | reported | [src](https://allenai.org/blog/olmohybrid) | |
| 40 | olmo3 | olmo3-32B-stage3-longcontext | final | midtraining | 3.22e+10 | 1.00e+11 | 1.93e+22 | 6ND derived | 23497.0 | 1.1216 | MMLU (base) | 76.2 | reported | [src](https://github.com/allenai/OLMo-core/blo... |
| 41 | olmo3 | olmo3-32B-Think-posttrain (DPO+RLVR) | final | post-training/RL | 3.22e+10 | 96561.0 | NaN | MATH; MMLU | 96.1 | reported | [src](https://arxiv.org/abs/2605.01158) |
3.6 Narrative
Process. The OLMo 2 development pattern: architecture and stability changes ablated at 1B, 19 “microanneal” runs totalling 130B tokens to test individual data sources, then final stage-1 runs (1 to 1.2 epochs) and several stage-2 anneals weight-averaged into the released model. Olmo 3 formalised this into per-source microanneals (5B target tokens plus 5B web) feeding “integration tests” (full 100B midtraining anneals on candidate mixes).
Ladder. Bhagia et al. (2024) trained 16 ladder models (190M, 370M, 760M, 1.3B, each at 1x, 2x, 5x, 10x Chinchilla tokens) and predicted OLMo 2 7B-4T and 13B-5T task accuracy within about 2 points on four tasks and about 4 points averaged over eight. The ladder cost about 1% of the targets’ compute.
Run management visible in the scripts. The official training scripts show decisions the papers do not: the Olmo 3 7B schedule was extended from 5T to 7T tokens mid-run and then hard-stopped at one epoch (5.93T); the 32B midtraining branched at 5.50T while pretraining ran on to 5.70T, so about 196B tokens of 32B pretraining were not used by the released model.
Not yet reconstructable. Per-task fitted coefficients from the ladder paper; the split of the 6.85M development hours across pretraining, midtraining, post-training and synthetic data; OLMo 2 7B and 13B GPU-hours; OLMo 2 32B losses (comet.ml, key-gated) and the ladder losses (private W&B project). arXiv was unreachable from this environment, so paper numbers came via secondary summaries and Ai2’s GitHub repositories; the notes file lists 13 items to verify against the primary sources.