1  Reference scaling suites: Pythia and Cerebras-GPT

These are designed scaling suites, not labs with an experiments-vs-final split. They give clean reference curves on a public dataset (the Pile) against which the lab chapters can be read. Working notes with sources are in data/notes/reference.md.

1.1 Cerebras-GPT: a published law and its out-of-sample point

Seven sizes from 111M to 13B, each trained on about 20 tokens per parameter. Cerebras fitted a loss-versus-compute law on the six smaller models and reports that the 13B landed within 0.5 percent of it. We treat the 13B as the out-of-sample “final run” and draw both Cerebras’s own law (grey, from the paper) and our refit on the same six points.

Our refit on 111M to 6.7B: L = 0.46 + 36.3·C^-0.067 (n=6). Cerebras’s law: L = 0.51 + 47.7·C^-0.074 (n=0).

run_id flops loss predicted residual log_residual elasticity_at_final
17 cerebras-gpt-13b 2.300000e+22 1.575 1.5955 -0.0205 -0.0129 -0.0479

Two things to carry into the lab chapters. First, Cerebras counts training FLOPs about 1.7 times higher than the 6ND convention, so any cross-suite overlay needs one convention. Second, the elasticity of loss with respect to compute at the 13B point is small in absolute terms because the fitted irreducible term is large; the table above gives the number.

1.2 Pythia: sixteen models, one token stream

70M to 12B, standard and deduplicated Pile, the same 300B tokens in the same order, 154 checkpoints per model. EleutherAI published no numerical loss values, but the W&B project is public, and the Pile validation loss at the final step of each model’s training run is now in runs.csv for 13 of the 16 models (data/losses/eleutherai/; the run-to-model mapping is by matching layer count, width, batch size and data path in the run configs, since the repository’s own mapping is “rough and partial”, and no finished deduplicated run matching 1B, 6.9B or 12B was found). The compute axis is 6ND (total parameters, including untied embeddings).

Fit on the standard-Pile models: L = 1.60 + 5.17e+08·C^-0.428 (n=8). The deduplicated models sit above the standard ones on this axis because their validation set is the deduplicated Pile, a harder target; the two series are not one curve.

The benchmark view below uses the released zero-shot evaluation files.

Reported A100-hours for the sixteen released models: 272,140. The paper’s total including the discarded first version of the suite is 544,280, so about three quarters of the suite’s compute went into the released models. That discarded suite is a rare published example of experiment compute that would otherwise be invisible.

1.3 Both suites on one compute axis

The two suites are close but not on one curve: Cerebras reports Pile test loss and counts FLOPs about 1.7x above 6ND; Pythia’s number is the trainer’s Pile validation loss at the last step, on 6ND compute. Both use the GPT-NeoX tokenizer, so the nats-per-token scale is shared.

1.4 Published fits and key numbers

project date stage intervention model_sizes compute_range metric fitted_exponent exponent_source raw_data_available source_url notes
0 cerebras-gpt 2023-04-06 pretraining compute-optimal loss-vs-compute scaling law (f... 111M;256M;590M;1.3B;2.7B;6.7B (fit); 13B (held... 2.6e18-6.3e21 FLOPs (fit); 2.3e22 (13B validat... Pile test loss (cross-entropy, nats/token) -0.0737 reported: paper Sec. 3.1 eq. L(f) = (f / 5.984... logs [src](https://arxiv.org/abs/2304.03208; https:... Paper claims 13B trained to within 0.5% of pro...
1 cerebras-gpt 2023-04-06 pretraining muP + muTransfer (HPs tuned on 40M proxy, tran... 111M;256M;590M;1.3B;2.7B (muP) vs same-shape S... 2.6e18-1.1e21 FLOPs Pile test loss; zero-shot downstream average (... no exponent reported for muP series; paper rep... logs [src](https://arxiv.org/abs/2304.03208; https:... Reported results: muP models average 0.43% low...
project date stage intervention model_sizes compute_range metric fitted_exponent exponent_source raw_data_available source_url notes
0 pythia 2023-04-03 pretraining model-size ladder at fixed 300B tokens (token-... 70M;160M;410M;1B;1.4B;2.8B;6.9B;12B (x2 data c... 1.27e20-2.13e22 FLOPs (6*N_total*D, derived); ... downstream accuracy (LAMBADA, PIQA, WSC, WinoG... none: paper fits no scaling law; no loss-vs-si... logs [src](https://arxiv.org/abs/2304.01373; https:... Paper finding: deduplication gave no clear ben...
1 pythia 2023-04-03 pretraining batch-size ablations at fixed 300B tokens (0.2... 70M (0.25M BS); 160M, 410M, 1B (0.5M BS); 1.4B... 1.27e20-2.5e21 FLOPs (6ND, derived) downstream accuracy (lm-eval-harness) at token... none reported logs [src](https://github.com/EleutherAI/pythia/tre... Paper reports no convergence issues using 2M-t...
project run_id run_type stage params_total tokens flops flops_method gpu_hours loss benchmark_name benchmark_value confidence source_url
0 pythia pythia-70m ladder pretraining 7.04e+07 3.00e+11 1.27e+20 6*N*D with N=total params incl. untied embed/u... 510.0 2.8878 lambada_openai_acc_0shot 0.1846 reported [src](https://arxiv.org/abs/2304.01373; https:...
1 pythia pythia-70m ladder pretraining 7.04e+07 3.00e+11 1.27e+20 6*N*D with N=total params incl. untied embed/u... 510.0 2.8878 piqa_acc_0shot 0.5947 reported [src](https://arxiv.org/abs/2304.01373; https:...
2 pythia pythia-70m-deduped ladder pretraining 7.04e+07 3.00e+11 1.27e+20 6*N*D with N=total params incl. untied embed/u... 510.0 3.0127 lambada_openai_acc_0shot 0.1915 reported [src](https://arxiv.org/abs/2304.01373; https:...
3 pythia pythia-70m-deduped ladder pretraining 7.04e+07 3.00e+11 1.27e+20 6*N*D with N=total params incl. untied embed/u... 510.0 3.0127 piqa_acc_0shot 0.5979 reported [src](https://arxiv.org/abs/2304.01373; https:...
4 pythia pythia-160m ladder pretraining 1.62e+08 3.00e+11 2.92e+20 6*N*D with N=total params incl. untied embed/u... 1030.0 2.5123 lambada_openai_acc_0shot 0.3284 reported [src](https://arxiv.org/abs/2304.01373; https:...
5 pythia pythia-160m ladder pretraining 1.62e+08 3.00e+11 2.92e+20 6*N*D with N=total params incl. untied embed/u... 1030.0 2.5123 piqa_acc_0shot 0.6273 reported [src](https://arxiv.org/abs/2304.01373; https:...
6 pythia pythia-160m-deduped ladder pretraining 1.62e+08 3.00e+11 2.92e+20 6*N*D with N=total params incl. untied embed/u... 1030.0 2.5839 lambada_openai_acc_0shot 0.3417 reported [src](https://arxiv.org/abs/2304.01373; https:...
7 pythia pythia-160m-deduped ladder pretraining 1.62e+08 3.00e+11 2.92e+20 6*N*D with N=total params incl. untied embed/u... 1030.0 2.5839 piqa_acc_0shot 0.6181 reported [src](https://arxiv.org/abs/2304.01373; https:...
8 pythia pythia-410m ladder pretraining 4.05e+08 3.00e+11 7.29e+20 6*N*D with N=total params incl. untied embed/u... 2540.0 2.1637 lambada_openai_acc_0shot 0.5162 reported [src](https://arxiv.org/abs/2304.01373; https:...
9 pythia pythia-410m ladder pretraining 4.05e+08 3.00e+11 7.29e+20 6*N*D with N=total params incl. untied embed/u... 2540.0 2.1637 piqa_acc_0shot 0.6676 reported [src](https://arxiv.org/abs/2304.01373; https:...
10 pythia pythia-410m-deduped ladder pretraining 4.05e+08 3.00e+11 7.29e+20 6*N*D with N=total params incl. untied embed/u... 2540.0 2.2675 lambada_openai_acc_0shot 0.5244 reported [src](https://arxiv.org/abs/2304.01373; https:...
11 pythia pythia-410m-deduped ladder pretraining 4.05e+08 3.00e+11 7.29e+20 6*N*D with N=total params incl. untied embed/u... 2540.0 2.2675 piqa_acc_0shot 0.6746 reported [src](https://arxiv.org/abs/2304.01373; https:...
12 pythia pythia-1b ladder pretraining 1.01e+09 3.00e+11 1.82e+21 6*N*D with N=total params incl. untied embed/u... 4830.0 2.0445 lambada_openai_acc_0shot 0.5620 reported [src](https://arxiv.org/abs/2304.01373; https:...
13 pythia pythia-1b ladder pretraining 1.01e+09 3.00e+11 1.82e+21 6*N*D with N=total params incl. untied embed/u... 4830.0 2.0445 piqa_acc_0shot 0.7073 reported [src](https://arxiv.org/abs/2304.01373; https:...
14 pythia pythia-1b-deduped ladder pretraining 1.01e+09 3.00e+11 1.82e+21 6*N*D with N=total params incl. untied embed/u... 4830.0 NaN lambada_openai_acc_0shot 0.5804 reported [src](https://arxiv.org/abs/2304.01373; https:...
15 pythia pythia-1b-deduped ladder pretraining 1.01e+09 3.00e+11 1.82e+21 6*N*D with N=total params incl. untied embed/u... 4830.0 NaN piqa_acc_0shot 0.7002 reported [src](https://arxiv.org/abs/2304.01373; https:...
16 pythia pythia-1.4b ladder pretraining 1.41e+09 3.00e+11 2.54e+21 6*N*D with N=total params incl. untied embed/u... 7120.0 1.9605 lambada_openai_acc_0shot 0.6161 reported [src](https://arxiv.org/abs/2304.01373; https:...
17 pythia pythia-1.4b ladder pretraining 1.41e+09 3.00e+11 2.54e+21 6*N*D with N=total params incl. untied embed/u... 7120.0 1.9605 piqa_acc_0shot 0.7089 reported [src](https://arxiv.org/abs/2304.01373; https:...
18 pythia pythia-1.4b-deduped ladder pretraining 1.41e+09 3.00e+11 2.54e+21 6*N*D with N=total params incl. untied embed/u... 7120.0 2.0295 lambada_openai_acc_0shot 0.6187 reported [src](https://arxiv.org/abs/2304.01373; https:...
19 pythia pythia-1.4b-deduped ladder pretraining 1.41e+09 3.00e+11 2.54e+21 6*N*D with N=total params incl. untied embed/u... 7120.0 2.0295 piqa_acc_0shot 0.7203 reported [src](https://arxiv.org/abs/2304.01373; https:...
20 pythia pythia-2.8b ladder pretraining 2.78e+09 3.00e+11 4.99e+21 6*N*D with N=total params incl. untied embed/u... 14240.0 1.8714 lambada_openai_acc_0shot 0.6472 reported [src](https://arxiv.org/abs/2304.01373; https:...
21 pythia pythia-2.8b ladder pretraining 2.78e+09 3.00e+11 4.99e+21 6*N*D with N=total params incl. untied embed/u... 14240.0 1.8714 piqa_acc_0shot 0.7388 reported [src](https://arxiv.org/abs/2304.01373; https:...
22 pythia pythia-2.8b-deduped ladder pretraining 2.78e+09 3.00e+11 4.99e+21 6*N*D with N=total params incl. untied embed/u... 14240.0 1.9281 lambada_openai_acc_0shot 0.6515 reported [src](https://arxiv.org/abs/2304.01373; https:...
23 pythia pythia-2.8b-deduped ladder pretraining 2.78e+09 3.00e+11 4.99e+21 6*N*D with N=total params incl. untied embed/u... 14240.0 1.9281 piqa_acc_0shot 0.7410 reported [src](https://arxiv.org/abs/2304.01373; https:...
24 pythia pythia-6.9b ladder pretraining 6.86e+09 3.00e+11 1.23e+22 6*N*D with N=total params incl. untied embed/u... 33500.0 1.7768 lambada_openai_acc_0shot 0.6728 reported [src](https://arxiv.org/abs/2304.01373; https:...
25 pythia pythia-6.9b ladder pretraining 6.86e+09 3.00e+11 1.23e+22 6*N*D with N=total params incl. untied embed/u... 33500.0 1.7768 piqa_acc_0shot 0.7524 reported [src](https://arxiv.org/abs/2304.01373; https:...
26 pythia pythia-6.9b-deduped ladder pretraining 6.86e+09 3.00e+11 1.23e+22 6*N*D with N=total params incl. untied embed/u... 33500.0 NaN lambada_openai_acc_0shot 0.6889 reported [src](https://arxiv.org/abs/2304.01373; https:...
27 pythia pythia-6.9b-deduped ladder pretraining 6.86e+09 3.00e+11 1.23e+22 6*N*D with N=total params incl. untied embed/u... 33500.0 NaN piqa_acc_0shot 0.7601 reported [src](https://arxiv.org/abs/2304.01373; https:...
28 pythia pythia-12b ladder pretraining 1.18e+10 3.00e+11 2.13e+22 6*N*D with N=total params incl. untied embed/u... 72300.0 1.7412 lambada_openai_acc_0shot 0.7046 reported [src](https://arxiv.org/abs/2304.01373; https:...
29 pythia pythia-12b ladder pretraining 1.18e+10 3.00e+11 2.13e+22 6*N*D with N=total params incl. untied embed/u... 72300.0 1.7412 piqa_acc_0shot 0.7601 reported [src](https://arxiv.org/abs/2304.01373; https:...
30 pythia pythia-12b-deduped ladder pretraining 1.18e+10 3.00e+11 2.13e+22 6*N*D with N=total params incl. untied embed/u... 72300.0 NaN lambada_openai_acc_0shot 0.7097 reported [src](https://arxiv.org/abs/2304.01373; https:...
31 pythia pythia-12b-deduped ladder pretraining 1.18e+10 3.00e+11 2.13e+22 6*N*D with N=total params incl. untied embed/u... 72300.0 NaN piqa_acc_0shot 0.7628 reported [src](https://arxiv.org/abs/2304.01373; https:...
0 cerebras-gpt cerebras-gpt-111m ladder pretraining 1.11e+08 2.22e+09 2.60e+18 reported in HF model card / paper (Cerebras FL... NaN 2.5660 lambada_acc_0shot 0.1940 reported [src](https://arxiv.org/abs/2304.03208; https:...
3 cerebras-gpt cerebras-gpt-256m ladder pretraining 2.56e+08 5.12e+09 1.30e+19 reported in HF model card / paper (Cerebras FL... NaN 2.2990 lambada_acc_0shot 0.2930 reported [src](https://arxiv.org/abs/2304.03208; https:...
6 cerebras-gpt cerebras-gpt-590m ladder pretraining 5.90e+08 1.18e+10 6.10e+19 reported in HF model card / paper (Cerebras FL... NaN 2.1840 lambada_acc_0shot 0.3660 reported [src](https://arxiv.org/abs/2304.03208; https:...
9 cerebras-gpt cerebras-gpt-1.3b ladder pretraining 1.32e+09 2.63e+10 2.80e+20 reported in HF model card / paper (Cerebras FL... NaN 1.9960 lambada_acc_0shot 0.4620 reported [src](https://arxiv.org/abs/2304.03208; https:...
12 cerebras-gpt cerebras-gpt-2.7b ladder pretraining 2.65e+09 5.30e+10 1.10e+21 reported in HF model card / paper (Cerebras FL... NaN 1.8340 lambada_acc_0shot 0.5670 reported [src](https://arxiv.org/abs/2304.03208; https:...
15 cerebras-gpt cerebras-gpt-6.7b ladder pretraining 6.66e+09 1.33e+11 6.30e+21 reported in HF model card / paper (Cerebras FL... NaN 1.7040 lambada_acc_0shot 0.6360 reported [src](https://arxiv.org/abs/2304.03208; https:...
17 cerebras-gpt cerebras-gpt-13b final pretraining 1.29e+10 2.57e+11 2.30e+22 reported in HF model card / paper (Cerebras FL... NaN 1.5750 lambada_acc_0shot 0.6960 reported [src](https://arxiv.org/abs/2304.03208; https:...