4  Meta Llama

Meta is the closed-weights-process, open-weights-model case: the Llama 3 paper describes a scaling ladder, data-mix experiments and annealing ablations in detail but never counts them, and only Llama 1 discloses total development compute. Working notes with sources are in data/notes/llama.md; 39 verbatim quantitative statements are in the claims table.

4.1 The one disclosed development budget: Llama 1

The LLaMA 1 paper estimates the whole project at 2,048 A100s for about five months (2,638 MWh). At the paper’s own throughput that is roughly 6.0M A100-hours, against 1.77M for the four released models. Around 70 percent of Llama 1’s compute went to runs that did not ship. Llama 2, 3 and 4 say nothing comparable.

4.2 Llama 3’s ladder, as far as it is published

Models of 40M to 16B parameters at budgets from 6e18 to 1e22 FLOPs gave the compute-optimal relation N_opt(C) = A · C^α with α = 0.53 and A = 0.29, which predicted 402B parameters and 16.55T tokens at the 3.8e25 budget; Meta chose 405B and 15.6T. The run count is never stated. The two-stage benchmark prediction (normalised log-likelihood from compute, then a sigmoid to ARC Challenge accuracy) is figure-only.

4.3 Across releases: released models, one benchmark

With no experiment runs sized, the cross-release picture is the released models only. Five-shot MMLU on the pre-trained base against 6ND compute, Llama 1 to Llama 3.

4.4 Run sizes, compute split, timeline

4.5 Published scaling experiments

project date stage intervention model_sizes compute_range metric fitted_exponent exponent_source raw_data_available source_url notes
0 llama-3 2024-07 pretraining IsoFLOP scaling-law ladder: vary model size vs... 40M-16B params (subset of sizes per budget; co... 6e18-1e22 FLOPs negative log-likelihood on held-out validation... alpha=0.53 (A=0.29) for optimal training token... numbers_in_text (text: 0.53/0.29; figure legen... figure_only [src](https://arxiv.org/abs/2407.21783) Extrapolation to 3.8e25 FLOPs -> 402B params o...
1 llama-3 2024-07 pretraining Two-stage downstream benchmark prediction (ARC... ladder 40M-16B; Llama 2 7B-70B; target 405B ~1e19-1e22 (stage 1); Llama 2 models ~8e22-8.4... normalized NLL per character of correct answer... not reported (linear and sigmoid coefficients ... figure_only figure_only [src](https://arxiv.org/abs/2407.21783) 'extrapolates over four orders of magnitude .....
2 llama-3 2024-07 pretraining Data-mix selection via scaling-law experiments... not stated ('several small models', 'a larger ... not stated predicted large-model benchmark performance figure_only [src](https://arxiv.org/abs/2407.21783) No numbers at all are published for these runs...
3 llama-3 2024-07 midtraining Annealing as data-quality probe: take 50%-trai... 8B ~1.9e21 FLOPs per anneal run (derived 6*8e9*40... downstream benchmark change numbers_in_text [src](https://arxiv.org/abs/2407.21783) Only GSM8k/MATH deltas are given numerically (...
4 llama-3 2024-07 midtraining Annealing on GSM8k and MATH training sets (fol... 8B and 405B not stated GSM8k and MATH validation accuracy 8B: +24.0% GSM8k, +6.4% MATH; 405B: negligible numbers_in_text numbers_in_text [src](https://arxiv.org/abs/2407.21783) Interpreted by authors as evidence the flagshi...
5 llama-3 2024-07 pretraining Parallelism/scale configuration of 405B pre-tr... 405B 430 / 400 / 380 TFLOPs per GPU; BF16 MFU 43% /... BF16 Model FLOPs Utilization numbers_in_text [src](https://arxiv.org/abs/2407.21783) MFU drop 43%->41% attributed to lower per-DP-g...
6 llama-3 2024-07 pretraining Batch-size ramp and LR schedule of flagship (s... 405B within the 3.8e25 flagship run training stability (loss spikes) numbers_in_text [src](https://arxiv.org/abs/2407.21783) 'we observed few loss spikes and did not requi...
7 llama-2 2023-07 pretraining Model-size series 7B/13B/34B/70B at fixed 2.0T... 7B, 13B, 34B, 70B 8.4e22-8.4e23 FLOPs (derived 6ND) training loss (Figure 5) none reported figure_only [src](https://arxiv.org/abs/2307.09288) 'after pretraining on 2T Tokens, the models st...
8 llama-1 2023-02 pretraining Model-size series 7B/13B/33B/65B trained on 1.... 6.7B, 13.0B, 32.5B, 65.2B 4.0e22-5.5e23 FLOPs (derived 6ND) training loss vs tokens (Figure 1) none reported figure_only [src](https://arxiv.org/abs/2302.13971) 'the performance of a 7B model continues to im...
9 llama-4 2025-04 pretraining MetaP hyperparameter-transfer technique (per-l... Scout 109B/17B active; Maverick 400B/17B activ... Behemoth: 32K GPUs at 390 TFLOPs/GPU (FP8); to... not stated not reported (no MetaP values published) figure_only [src](https://ai.meta.com/blog/llama-4-multimo... Blog only; no technical report. No numbers for...

4.6 What the lab says about itself

project claim value unit source_url quote notes
0 llama-3 flagship pre-training compute 3.8e25 FLOPs [src](https://arxiv.org/abs/2407.21783) our flagship language model was pre-trained us... Section 1
1 llama-3 scaling-law compute range 6e18-1e22 FLOPs [src](https://arxiv.org/abs/2407.21783) we construct our scaling laws by pre-training ... Section 3.2.1
2 llama-3 scaling-law ladder hyperparameters 2000 warmup steps; LR 2e-4 to 4e-4; batch 250K-4M various [src](https://arxiv.org/abs/2407.21783) we use a cosine learning rate schedule with a ... Section 3.2.1
3 llama-3 fitted token-scaling exponent alpha=0.53, A=0.29 dimensionless [src](https://arxiv.org/abs/2407.21783) we assume a power-law relation between compute... Figure 3 legend (arXiv HTML v3, datacompute.sv...
4 llama-3 compute-optimal prediction 402B params / 16.55T tokens params / tokens [src](https://arxiv.org/abs/2407.21783) Extrapolation of the resulting scaling law to ... Section 3.2.1
5 llama-3 decision to use 405B 405B params [src](https://arxiv.org/abs/2407.21783) An important observation is that IsoFLOPs curv... Section 3.2.1
6 llama-3 two-stage benchmark prediction accuracy 4 orders of magnitude; slight underestimate qualitative [src](https://arxiv.org/abs/2407.21783) We find this two-step scaling law prediction, ... ARC Challenge; Figure 4
7 llama-3 data-mix scaling law experiments several small models per mix count (vague) [src](https://arxiv.org/abs/2407.21783) we perform scaling law experiments in which we... Section 3.1.2; no counts or sizes
8 llama-3 final data mix 50/25/17/8 % tokens [src](https://arxiv.org/abs/2407.21783) Our final data mix contains roughly 50% of tok... Section 3.1.2
9 llama-3 annealing effect on 8B +24.0% GSM8k; +6.4% MATH % (relative or absolute unclear) [src](https://arxiv.org/abs/2407.21783) annealing improved the performance of a pre-tr... Section 3.1.3
10 llama-3 annealing as data-quality probe 8B at 50% trained; 40B tokens; 30%/70% tokens / weight [src](https://arxiv.org/abs/2407.21783) We measure the value of such datasets by annea... Section 3.1.3
11 llama-3 flagship LR schedule 8e-5 peak; 8000 warmup; 1,200,000 steps; 8e-7 ... various [src](https://arxiv.org/abs/2407.21783) We pre-train Llama 3 405B using AdamW with a p... Section 3.4.1
12 llama-3 flagship batch-size ramp 4M -> 8M (after 252M tokens) -> 16M (after 2.8... tokens/batch [src](https://arxiv.org/abs/2407.21783) we use an initial batch size of 4M tokens and ... Section 3.4.1
13 llama-3 long-context stage tokens ~800B; six stages 8K->128K tokens [src](https://arxiv.org/abs/2407.21783) In Llama 3 405B pre-training, we increased con... Section 3.4.2. Sec 2.1 says the 15.6T-token ru...
14 llama-3 annealing stage 40M tokens; LR to 0; Polyak averaging tokens [src](https://arxiv.org/abs/2407.21783) During pre-training on the final 40M tokens, w... Section 3.4.3
15 llama-3 cluster size 16K of 24K GPUs H100 GPUs [src](https://arxiv.org/abs/2407.21783) Llama 3 405B is trained on up to 16K H100 GPUs... Section 3.3.1
16 llama-3 MFU 38-43% BF16 MFU [src](https://arxiv.org/abs/2407.21783) we achieve an overall BF16 Model FLOPs Utiliza... Table 4: 8,192 GPUs 430 TFLOPs/GPU 43%; 16,384...
17 llama-3 reliability during 54-day snapshot 466 interruptions (47 planned, 419 unexpected)... count / % [src](https://arxiv.org/abs/2407.21783) During a 54-day snapshot period of pre-trainin... Section 3.3.4; Table 5
18 llama-3 smaller models over-trained qualitative [src](https://arxiv.org/abs/2407.21783) While our scaling laws suggest our flagship mo... Section 1
19 llama-3 total GPU hours (herd) 39.3M H100-80GB GPU-hours [src](https://github.com/meta-llama/llama-mode... Training utilized a cumulative of 39.3M GPU ho... Llama 3.1 model card (Jul 23 2024); per-model ...
20 llama-3 total GPU hours (Llama 3 April release) 7.7M H100-80GB GPU-hours [src](https://github.com/meta-llama/llama3/blo... Pretraining utilized a cumulative 7.7M GPU hou... Llama 3 model card, Model Release Date April 1...
21 llama-3 post-training rounds 6 rounds [src](https://arxiv.org/abs/2407.21783) Following Llama 2, we apply the above methods ... Section 4.1.6
22 llama-3 code expert continued pre-training 1T tokens (>85% code) tokens [src](https://arxiv.org/abs/2407.21783) This is accomplished by branching the main pre... Section 4.3.1
23 llama-3 author count 559 authors [src](https://arxiv.org/abs/2407.21783) Contributors section of https://arxiv.org/html... Verified against arXiv HTML v3 (23 Nov 2024) o...
24 llama-3 derived training duration ~72 days days [src](https://x.com/EpochAIResearch/status/181... (Epoch AI) Llama 3.1 405B was trained using a ... Third-party derivation; snippet only
25 llama-3 clusters used for Llama 3 two 24K-GPU clusters GPUs [src](https://engineering.fb.com/2024/03/12/da... (search snippet) Meta announced two 24k GPU cl... Engineering blog 2024-03-12; page not directly...
26 llama-2 GPU hours 184320 / 368640 / 1038336 / 1720320; total 331... A100-80GB GPU-hours [src](https://arxiv.org/abs/2307.09288) A cumulative of 3.3M GPU hours of computation ... Table 2: 7B 184320 400W 31.22; 13B 368640 400W...
27 llama-2 training hardware A100; RSC InfiniBand 400W; production RoCE 350... [src](https://arxiv.org/abs/2307.09288) We pretrained our models on Meta's Research Su... Section 2.2.1
28 llama-2 no saturation at 2T tokens 2T tokens [src](https://arxiv.org/abs/2307.09288) Figure 5: Training Loss for Llama 2 models. We... Figure only; no numeric loss
29 llama-2 training dates Jan 2023 - Jul 2023 [src](https://github.com/meta-llama/llama/blob... Llama 2 was trained between January 2023 and J... Model card
30 llama-2 SFT and preference data 27,540 SFT; 1,418,091 comparisons annotations [src](https://arxiv.org/abs/2307.09288) We stopped annotating SFT after collecting a t... Sections 3.1, 3.2; Table 6
31 llama-1 GPU hours 82,432 / 135,168 / 530,432 / 1,022,362 A100-80GB GPU-hours [src](https://arxiv.org/abs/2302.13971) LLaMA-7B A100-80GB 400W 82,432 36 MWh 14; LLaM... Table 15 carbon footprint; PUE 1.1; 0.385 kg C...
32 llama-1 total development compute 2048 A100 x ~5 months; 2,638 MWh; 1,015 tCO2eq [src](https://arxiv.org/abs/2302.13971) we estimate that we used 2048 A100-80GB for a ... Section 6; implies ~6.0M GPU-h total vs 1.77M ...
33 llama-1 65B throughput and duration 380 tokens/sec/GPU; 21 days [src](https://arxiv.org/abs/2302.13971) When training a 65B-parameter model, our code ... Section 2.4
34 llama-1 over-training rationale 7B improves after 1T tokens tokens [src](https://arxiv.org/abs/2302.13971) although Hoffmann et al. (2022) recommends tra... Section 1
35 llama-4 GPU hours Scout 5.0M; Maverick 2.38M; total 7.38M H100-80GB GPU-hours [src](https://github.com/meta-llama/llama-mode... Model pre-training utilized a cumulative of 7.... Model card; release date April 5, 2025
36 llama-4 training tokens Scout ~40T; Maverick ~22T tokens [src](https://github.com/meta-llama/llama-mode... Llama 4 Scout was pretrained on ~40 trillion t... Model card
37 llama-4 Behemoth throughput 390 TFLOPs/GPU on 32K GPUs (FP8) TFLOPs/GPU [src](https://ai.meta.com/blog/llama-4-multimo... (blog, via search snippet and mirrors) while p... Blog not directly fetched; wording confirmed b...
38 llama-4 Behemoth size 288B active; 16 experts; ~2T total params [src](https://ai.meta.com/blog/llama-4-multimo... (blog, via mirrors) Llama 4 Behemoth ... 288B ...

4.7 Key numbers

project run_id run_type stage params_total tokens flops flops_method gpu_hours loss benchmark_name benchmark_value confidence source_url
0 llama-3 l3-405b-pretrain final pretraining 4.05e+11 1.56e+13 3.80e+25 reported in paper (Sec 1); 6*N*D check = 3.79e25 30840000.0 NaN MMLU (5-shot, pre-trained) 85.2 reported [src](https://arxiv.org/abs/2407.21783 ; https...
1 llama-3 l3-405b-longcontext midtrain midtraining 4.05e+11 8.00e+11 1.90e+24 derived 6*N*D with N=405e9, D=800e9 NaN NaN NaN derived [src](https://arxiv.org/abs/2407.21783)
2 llama-3 l3-405b-anneal midtrain midtraining 4.05e+11 4.00e+07 9.70e+19 derived 6*N*D with D=40e6 NaN NaN NaN reported [src](https://arxiv.org/abs/2407.21783)
3 llama-3 l3-405b-posttrain final post-training/RL 4.05e+11 NaN NaN MMLU (5-shot, post-trained) 87.3 reported [src](https://arxiv.org/abs/2407.21783)
4 llama-3 l3-70b-pretrain final pretraining 7.00e+10 1.50e+13 6.30e+24 derived 6*N*D with D=15e12 (model card says '1... 7000000.0 NaN MMLU (5-shot, pre-trained) 79.3 reported [src](https://arxiv.org/abs/2407.21783 ; https...
5 llama-3 l3-8b-pretrain final pretraining 8.00e+09 1.50e+13 7.20e+23 derived 6*N*D with D=15e12 1460000.0 NaN MMLU (5-shot, pre-trained) 66.7 reported [src](https://arxiv.org/abs/2407.21783 ; https...
6 llama-3 l3-code-expert-branch midtrain midtraining 1.00e+12 NaN NaN NaN reported [src](https://arxiv.org/abs/2407.21783)
7 llama-3 l3-multilingual-expert-branch midtrain midtraining NaN NaN NaN reported [src](https://arxiv.org/abs/2407.21783)
8 llama-3 l3-scaling-ladder ladder pretraining range 6e18 to 1e22 FLOPs per run (reported) NaN NaN NaN reported [src](https://arxiv.org/abs/2407.21783)
9 llama-3 l3-datamix-scaling-ablations ablation pretraining NaN NaN NaN reported [src](https://arxiv.org/abs/2407.21783)
10 llama-3 l3-anneal-dataquality-ablations-8b ablation midtraining 8.00e+09 4.00e+10 1.90e+21 derived 6*N*D per anneal run with N=8e9, D=40e9 NaN NaN NaN reported [src](https://arxiv.org/abs/2407.21783)
11 llama-3 l3-anneal-gsm8k-math-8b ablation midtraining 8.00e+09 NaN NaN GSM8k / MATH validation improvement from annea... NaN reported [src](https://arxiv.org/abs/2407.21783)
12 llama-3 l3-anneal-gsm8k-math-405b ablation midtraining 4.05e+11 NaN NaN GSM8k / MATH improvement from annealing NaN reported [src](https://arxiv.org/abs/2407.21783)
13 llama-3 l3-405b-mfu-configs unknown pretraining 4.05e+11 NaN NaN NaN reported [src](https://arxiv.org/abs/2407.21783)
14 llama-2 l2-7b-pretrain final pretraining 7.00e+09 2.00e+12 8.40e+22 derived 6*N*D with D=2.0e12 184320.0 NaN MMLU (5-shot, pre-trained) 45.3 reported [src](https://arxiv.org/abs/2307.09288 ; https...
15 llama-2 l2-13b-pretrain final pretraining 1.30e+10 2.00e+12 1.56e+23 derived 6*N*D with D=2.0e12 368640.0 NaN MMLU (5-shot, pre-trained) 54.8 reported [src](https://arxiv.org/abs/2307.09288 ; https...
16 llama-2 l2-34b-pretrain final pretraining 3.40e+10 2.00e+12 4.08e+23 derived 6*N*D with D=2.0e12 1038336.0 NaN MMLU (5-shot, pre-trained) 62.6 reported [src](https://arxiv.org/abs/2307.09288 ; https...
17 llama-2 l2-70b-pretrain final pretraining 7.00e+10 2.00e+12 8.40e+23 derived 6*N*D with D=2.0e12 1720320.0 NaN MMLU (5-shot, pre-trained) 68.9 reported [src](https://arxiv.org/abs/2307.09288 ; https...
18 llama-2 l2-chat-rlhf final post-training/RL 7.00e+09 NaN NaN NaN reported [src](https://arxiv.org/abs/2307.09288)
19 llama-1 l1-7b-pretrain final pretraining 6.70e+09 1.00e+12 4.02e+22 derived 6*N*D 82432.0 NaN MMLU (5-shot) 35.1 reported [src](https://arxiv.org/abs/2302.13971 ; https...
20 llama-1 l1-13b-pretrain final pretraining 1.30e+10 1.00e+12 7.80e+22 derived 6*N*D 135168.0 NaN MMLU (5-shot) 46.9 reported [src](https://arxiv.org/abs/2302.13971 ; https...
21 llama-1 l1-33b-pretrain final pretraining 3.25e+10 1.40e+12 2.73e+23 derived 6*N*D 530432.0 NaN MMLU (5-shot) 57.8 reported [src](https://arxiv.org/abs/2302.13971 ; https...
22 llama-1 l1-65b-pretrain final pretraining 6.52e+10 1.40e+12 5.48e+23 derived 6*N*D 1022362.0 NaN MMLU (5-shot) 63.4 reported [src](https://arxiv.org/abs/2302.13971 ; https...
24 llama-4 l4-scout-pretrain final pretraining 1.09e+11 4.00e+13 4.10e+24 derived 6*N_active*D with N=17e9, D=40e12 (MoE... 5000000.0 NaN NaN reported [src](https://github.com/meta-llama/llama-mode...
25 llama-4 l4-maverick-pretrain final pretraining 4.00e+11 2.20e+13 2.20e+24 derived 6*N_active*D with N=17e9, D=22e12 2380000.0 NaN NaN reported [src](https://github.com/meta-llama/llama-mode...
26 llama-4 l4-behemoth-pretrain unknown pretraining 2.00e+12 3.00e+13 5.20e+25 derived 6*N_active*D with N=288e9, D=30e12 (lo... NaN NaN NaN derived [src](https://ai.meta.com/blog/llama-4-multimo...

4.8 Narrative

Llama 3 (2024). 405B on 15.6T tokens, 3.8e25 FLOPs, 30.84M H100-hours for the flagship and 39.3M for the herd (from the model card, not the paper). MFU 38 to 43 percent; a 54-day snapshot logged 466 interruptions with over 90 percent effective training time. Development work the paper describes without counting: the IsoFLOP ladder, “several small models” per candidate data mix, annealing 40B tokens on a half-trained 8B as a data-quality probe, and code and multilingual expert branches continued from the main run. The ladder’s share of the flagship budget is a guess in the data (about 0.2 percent) and is flagged as such.

Llama 2 (2023). Four sizes on 2T tokens, 3.31M A100-hours including the unreleased 34B, trained January to July 2023. No ladder or pretraining ablation is described.

Llama 4 (2025). Scout and Maverick (17B active) at 5.0M and 2.38M H100-hours; Behemoth (288B active, about 2T total) on 32K GPUs at 390 TFLOPs per GPU in FP8, unreleased.

Not reconstructable. Any experiment run count or compute for Llama 2 to 4; the two-stage prediction’s coefficients; the ladder’s per-run sizes (read off a figure). Primary sources (arXiv, Meta AI, Hugging Face) were blocked from this environment; text was checked against GitHub-hosted conversions of the papers, and post-trained benchmark tables were omitted where the conversion garbled them.