5  DeepSeek

DeepSeek is the lab with the most-quoted training cost and the most explicit statement of what that cost leaves out. The V3 report’s 2.788M H800-hours ($5.576M at $2 per hour) covers “only the official training of DeepSeek-V3, excluding the costs associated with prior research and ablation experiments on architectures, algorithms, or data.” This chapter asks how much of the excluded part the reports themselves let us reconstruct. Working notes with sources are in data/notes/deepseek.md; 51 verbatim statements are in the claims table.

5.1 What the reports disclose of the excluded compute

The V3 report describes its ablations by size and tokens: multi-token prediction and auxiliary-loss-free balancing each tested on a 15.7B mixture-of-experts model at 1.33T tokens and a 228.7B model at 540 to 578B tokens, and FP8 against BF16 on models “similar to DeepSeek-V2-Lite and DeepSeek-V2” for about 1T tokens each. Summed at 6ND with both runs of each pair counted, those disclosed ablations come to roughly one sixth of the compute of the 3.3e24 final run (the bar below gives the exact share). Nothing about the many runs behind the architecture and data choices is counted, so this is a floor, and a low one.

Disclosed non-final share for V3: 16% of logged FLOPs.

Two contrasts with the labs that count everything. Ai2 reports 82 percent development for Olmo 3 and Meta about 70 percent for LLaMA 1. If DeepSeek’s true ratio were similar, the widely quoted $5.6M would be a fifth to a third of the programme’s compute cost.

5.2 The one real scaling-law programme: DeepSeek LLM (2024)

The first report is the only one with a full ladder: eight IsoFLOP budgets from 1e17 to 3e20 FLOPs with about ten allocations each (roughly 80 runs, derived), fitted to M_opt = 0.1715 · C^0.5243 and D_opt = 5.8316 · C^0.4757 (non-embedding FLOPs per token and tokens), plus hyperparameter laws for learning rate and batch size. The 7B and 67B are overlaid on the loss-versus-compute curve and described as “well-predicted” at about 1,000 times the largest ladder run, but no numerical error is printed. Later reports (V2, V3) ablate at two fixed scales instead of fitting laws, and R1, V3.2 and V4 mention small-scale preparation only qualitatively. Disclosure of the experimental process has fallen with each release even as the cost figure became more precise.

Both released models sit well above the fitted optimum in tokens: they were deliberately over-trained for inference efficiency, as most released open models are.

5.3 Across releases: released models, one benchmark

5.4 Post-training cost, stated

R1’s revised report gives the RL programme’s cost directly: 512 H800s for 198 hours (R1-Zero, 101K GPU-hours) plus 80 hours (R1, 41K) plus 5K for SFT data, 147K GPU-hours or $294K, about 5 percent of the V3 pretraining run. A 30B preparatory model was trained on A100s first with no figures given. V3.2 states that post-training now exceeds 10 percent of pretraining cost.

5.5 Run sizes and timeline

5.6 Published scaling experiments

project date stage intervention model_sizes compute_range metric fitted_exponent exponent_source raw_data_available source_url notes
0 deepseek-llm 2024-01-05 pretraining batch size and learning rate vs compute budget... 177M FLOPs/token model at C=1e17 (grid); multi... 1e17-2e19 (fit); 1e20 (validation) generalization error (bits-per-byte on validat... B_opt = 0.2920*C^0.3271; eta_opt = 0.3118*C^-0... paper Eq. (1) / Sec 3.1 figure_only [src](https://arxiv.org/abs/2401.02954) near-optimal defined as within 0.25% of min ge...
1 deepseek-llm 2024-01-05 pretraining IsoFLOP optimal model/data allocation (compute... ~80 models: 8 compute budgets x ~10 model/data... 1e17-3e20 bits-per-byte on in-house validation set M_opt = 0.1715*C^0.5243; D_opt = 5.8316*C^0.47... paper Eq. (3), Fig. 4 figure_only [src](https://arxiv.org/abs/2401.02954) uses M=72*n_layer*d_model^2+12*n_layer*d_model...
2 deepseek-llm 2024-01-05 pretraining loss-vs-compute extrapolation to final 7B and ... small models 1e17-3e20 (grey circles) vs DeepS... 1e17-1e24 bits-per-byte on validation set power law of optimal generalization error vs C... paper Fig. 5 figure_only [src](https://arxiv.org/abs/2401.02954) 'using small-scale experiments can accurately ...
3 deepseek-llm 2024-01-05 pretraining data quality effect on scaling exponents (thre... IsoFLOP suites on early in-house data, current... not stated (presumably 1e17-3e20) bits-per-byte a/b: early data 0.450/0.550; current data 0.52... paper Table 4 numbers_in_text [src](https://arxiv.org/abs/2401.02954) 'as data quality improves, the model scaling e...
4 deepseek-v2 2024-05-07 pretraining attention variant: MHA vs GQA(8) vs MQA in 7B ... 7.1B / 6.9B / 6.9B dense, 1.33T tokens each ~5.5e22 each (6ND) BBH, MMLU, C-Eval, CMMLU numbers_in_text [src](https://arxiv.org/abs/2405.04434) Appendix D.1, Table 8; MHA best (MMLU 45.2 vs ...
5 deepseek-v2 2024-05-07 pretraining MLA vs MHA in MoE at two scales ~16B total (2.4-2.5B active) on 1.33T tokens; ... ~2e22 (small) to ~6e22 (large) (6ND active) BBH, MMLU, C-Eval, CMMLU; KV cache per token numbers_in_text [src](https://arxiv.org/abs/2405.04434) Appendix D.2, Table 9; MLA slightly better wit...
6 deepseek-v3 2024-12-27 pretraining multi-token prediction (MTP) objective 15.7B MoE on 1.33T tokens; 228.7B MoE on 540B ... ~2e22 to ~7e22 (6ND active; large active param... Pile BPB, BBH, MMLU, DROP, TriviaQA, NQ, Human... numbers_in_text [src](https://arxiv.org/abs/2412.19437) Table 4; large: HumanEval 44.5->53.7, GSM8K 72...
7 deepseek-v3 2024-12-27 pretraining auxiliary-loss-free load balancing vs auxiliar... 15.7B MoE on 1.33T tokens; 228.7B MoE on 578B ... ~2e22 to ~7e22 Pile BPB, BBH, MMLU, HumanEval, MBPP, GSM8K, MATH numbers_in_text [src](https://arxiv.org/abs/2412.19437) Table 5; large HumanEval 40.2->46.3
8 deepseek-v3 2024-12-27 pretraining FP8 mixed-precision vs BF16 training ~16B (V2-Lite-like) and ~230B (V2-like) MoE, ~... ~1.4e22 and ~1.3e23 per run training loss curve; relative loss error relative loss error < 0.25% Appendix B.1, Fig. 10 figure_only [src](https://arxiv.org/abs/2412.19437) 'the relative loss error of our FP8-training m...
9 deepseek-v3 2024-12-27 pretraining batch-wise vs sequence-wise load balancing 1B MoE (3 runs, validation loss 2.258/2.253/2.... not stated validation loss; expert load on Pile test numbers_in_text [src](https://arxiv.org/abs/2412.19437) Sec 4.5.3, Fig. 9
10 deepseek-r1 2025-01-22 post-training/RL RL preparation on a smaller model before scaling 30B model on A100 GPUs not stated not stated ('results ... have been promising') numbers_in_text [src](https://arxiv.org/abs/2501.12948v2) arXiv v2 Appendix B.4.4 only mentions the expe...
11 deepseek-r1 2025-01-22 post-training/RL RL training dynamics of R1-Zero (AIME accuracy... 671B/37B (DeepSeek-V3-Base) 101K H800 GPU-hours total AIME 2024 pass@1 (avg of 16 samples) and cons@... figure_only [src](https://arxiv.org/abs/2501.12948) Fig. 2/3; 15.6% -> 71.0% over 10,400 steps (st...
12 deepseek-r1 2025-01-22 post-training/RL large-scale RL vs distillation on Qwen-32B Qwen-32B-Base RL >10K steps vs Distill-Qwen-32... not stated AIME 2024, MATH-500, GPQA, LiveCodeBench numbers_in_text [src](https://arxiv.org/abs/2501.12948) Table 6; distillation wins (AIME 72.6 vs RL mo...
13 deepseek-r1 2025-01-22 post-training/RL unsuccessful attempts: process reward model, MCTS not stated not stated qualitative numbers_in_text [src](https://arxiv.org/abs/2501.12948) Sec 4.2 describes failures (reward hacking, se...
14 deepseek-v3.2 2025-12-02 midtraining DeepSeek Sparse Attention via continued pretra... 671B/37B; 2.1B dense warm-up + 943.7B sparse t... ~2e23 (6ND active, rough) benchmark parity (MMLU-Pro 85.0, AIME 2025 89.... numbers_in_text [src](https://github.com/deepseek-ai/DeepSeek-... training configurations aligned with V3.1-Term...

5.7 What the lab says about itself

project claim value unit source_url quote notes
0 deepseek-v3 official training cost excludes research/ablat... 5.576e6 USD [src](https://arxiv.org/abs/2412.19437) Note that the aforementioned costs include onl... Table 1 context; verbatim via GitHub-mirrored ...
1 deepseek-v3 cost assumption 2 USD per H800 GPU-hour [src](https://arxiv.org/abs/2412.19437) Assuming the rental price of the H800 GPU is $...
2 deepseek-v3 GPU-hour breakdown 2664K / 119K / 5K / 2788K H800 GPU-hours (pretraining / context extensio... [src](https://arxiv.org/abs/2412.19437) Pre-Training 2664K ($5.328M); Context Extensio... Table 1
3 deepseek-v3 pretraining efficiency 180000 H800 GPU-hours per trillion tokens [src](https://arxiv.org/abs/2412.19437) Consequently, our pre-training stage is comple... wording reassembled from two adjacent sentence...
4 deepseek-v3 cluster size 2048 H800 GPUs [src](https://arxiv.org/abs/2412.19437) DeepSeek-V3 is trained on a cluster equipped w...
5 deepseek-v3 training stability 0 irrecoverable loss spikes / rollbacks [src](https://arxiv.org/abs/2412.19437) Throughout the entire training process, we did...
6 deepseek-v3 FP8 ablation scale ~16B on 1.33T tokens; ~230B on ~0.9T tokens params; tokens [src](https://arxiv.org/abs/2412.19437) approximately 16B total parameters on 1.33T to... Appendix B.1, verified in arXiv HTML v2 on 202...
7 deepseek-v3 FP8 relative loss error <0.25 percent [src](https://arxiv.org/abs/2412.19437) the relative loss error of our FP8-training mo... Appendix B.1 / Fig. 10
8 deepseek-v3 MTP ablation scales 15.7B on 1.33T; 228.7B on 540B params; tokens [src](https://arxiv.org/abs/2412.19437) small MoE model, comprising 15.7B total parame... Sec 4.5.1 (paraphrase of table-adjacent text; ...
9 deepseek-v3 aux-loss-free ablation scales 15.7B on 1.33T; 228.7B on 578B params; tokens [src](https://arxiv.org/abs/2412.19437) Table 5: small MoE 15.7B total params, 1.33T t... Sec 4.5.2
10 deepseek-v3 load-balance ablation validation losses 2.258 / 2.253 / 2.253 validation loss (1B MoE: sequence-wise aux los... [src](https://arxiv.org/abs/2412.19437) On 1B MoE models, the validation losses are: 2... Sec 4.5.3
11 deepseek-v3 expert-load analysis scale 16B params [src](https://arxiv.org/abs/2412.19437) we record and analyze the expert load of a 16B... Sec 4.5.3 / Fig. 9
12 deepseek-v3 Pile-test BPB (base) 0.548 bits-per-byte [src](https://github.com/deepseek-ai/DeepSeek-V3) Pile-test (BPB): DeepSeek-V2 Base 0.606; Qwen2... README benchmark table
13 deepseek-v3 MTP acceptance rate 85-90 percent of second-token predictions accepted [src](https://arxiv.org/abs/2412.19437) the acceptance rate of the second token predic... Sec 5.4.3 / conclusion; wording via mirror
14 deepseek-v3 author count 199 named authors (Appendix A, v1 and v2; arXiv li... [src](https://arxiv.org/abs/2412.19437) Appendix A counted 2026-09-13: Research & Engi... REFUTES the earlier 139 (Interconnects, 2025-0...
15 deepseek-v3 total experimentation multiplier (external est... 2-4 x final-run cost [src](https://www.interconnects.ai/p/deepseek-... Lambert estimates the $5.5M reported figure re... paraphrase from GitHub-mirrored article text; ...
16 deepseek-v3 GPU fleet estimate (external) ~50,000 Hopper GPUs (mix of H800/H100/H20) [src](https://semianalysis.com/2025/01/31/deep... DeepSeek has access to roughly 50,000 Hopper G... from search-result snippet; semianalysis.com b...
17 deepseek-v2 training cost per trillion tokens 300.6K (67B) vs 172.8K (V2) H800 GPU-hours per trillion tokens [src](https://arxiv.org/abs/2405.04434) During our practical training on the H800 clus... Sec 3.2.3; verbatim from PDF text
18 deepseek-v2 pretraining tokens 8.1e12 tokens [src](https://arxiv.org/abs/2405.04434) Our tokenized pre-training corpus contains 8.1...
19 deepseek-v2 MLA ablation scales ~16B on 1.33T; ~250B on 420B params; tokens [src](https://arxiv.org/abs/2405.04434) Two small MoE models comprise about 16B total ... Appendix D.2
20 deepseek-v2 attention ablation scale 7B dense x3 on 1.33T params; tokens [src](https://arxiv.org/abs/2405.04434) We show the evaluation results for 7B dense mo... Appendix D.1
21 deepseek-v2 context extension 1000 steps x 576 seqs x 32K steps [src](https://arxiv.org/abs/2405.04434) We additionally train the model for 1000 steps...
22 deepseek-v2 V2-Lite scale 15.7B total / 2.4B active; 5.7T tokens params; tokens [src](https://arxiv.org/abs/2405.04434) DeepSeek-V2-Lite comprises 15.7B total paramet... Appendix B
23 deepseek-llm IsoFLOP experiment design 8 budgets x ~10 allocations; 1e17-3e20 compute budgets (FLOPs) [src](https://arxiv.org/abs/2401.02954) We selected 8 different compute budgets rangin... Sec 3.2
24 deepseek-llm optimal allocation fit M_opt = 0.1715*C^0.5243; D_opt = 5.8316*C^0.4757 power-law fit [src](https://arxiv.org/abs/2401.02954) M_opt = M_base * C^a, M_base = 0.1715, a = 0.5... Eq. (3)
25 deepseek-llm hyperparameter scaling fit eta_opt = 0.3118*C^-0.1250; B_opt = 0.2920*C^0... power-law fit [src](https://arxiv.org/abs/2401.02954) eta_opt = 0.3118 * C^-0.1250, B_opt = 0.2920 *... Eq. (1); fit over 1e17-2e19, validated at 1e20
26 deepseek-llm hyperparameter grid compute 1e17 (grid, 177M FLOPs/token); 1e20 (validatio... FLOPs [src](https://arxiv.org/abs/2401.02954) We initially conducted a grid search for batch... Sec 3.1
27 deepseek-llm prediction accuracy (qualitative) 1000 x compute extrapolation [src](https://arxiv.org/abs/2401.02954) The results indicate that using small-scale ex... no numeric error reported
28 deepseek-llm 7B/67B well-predicted (Fig. 5 caption) bits-per-byte [src](https://arxiv.org/abs/2401.02954) Performance scaling curve. The metric is bits-... Fig. 5 caption; figure with small runs and fin...
29 deepseek-llm near-optimal criterion 0.25 percent above minimum generalization error [src](https://arxiv.org/abs/2401.02954) We regarded the parameters used by models whos...
30 deepseek-llm data-quality dependence of exponents a/b = 0.450/0.550, 0.524/0.476, 0.578/0.422 exponents (early data, current data, OpenWebTe... [src](https://arxiv.org/abs/2401.02954) as data quality improves, the model scaling ex... Table 4
31 deepseek-llm author count 86 authors [src](https://arxiv.org/abs/2401.02954) (author block counted: Xiao Bi ... Yuheng Zou ... counted from GitHub-mirrored transcription
32 deepseek-r1 R1 training cost table 101K / 5K / 41K / 147K H800 GPU-hours (R1-Zero / SFT data creation / ... [src](https://arxiv.org/abs/2501.12948v2) Table 7 | Training costs of DeepSeek-R1, assum... arXiv v2 (4 Jan 2026) Appendix B.4.4; verbatim...
33 deepseek-r1 RL hardware and duration 64*8 H800; 198 h (R1-Zero); ~80 h (R1) GPUs; hours [src](https://arxiv.org/abs/2501.12948v2) For the training of DeepSeek-R1-Zero, we emplo...
34 deepseek-r1 preparatory small-model experiments 30B on A100 params [src](https://arxiv.org/abs/2501.12948v2) Regarding our research on DeepSeek-R1, we util...
35 deepseek-r1 R1-Zero RL steps 10400 steps (1.6 epochs; batch 512) [src](https://arxiv.org/abs/2501.12948v2) for a total of 10,400 steps, corresponding to ...
36 deepseek-r1 R1 second RL stage steps 1700 steps [src](https://arxiv.org/abs/2501.12948v2) The stage comprises a total of 1,700 training ...
37 deepseek-r1 SFT data ~600k reasoning + ~200k non-reasoning = ~800k samples [src](https://arxiv.org/abs/2501.12948) In total, we collect about 600k reasoning rela...
38 deepseek-r1 R1-Zero AIME progression 15.6 -> 71.0 (86.7 cons@64) percent [src](https://arxiv.org/abs/2501.12948) the pass@1 score on AIME 2024 increases from 1...
39 deepseek-r1 RL vs distillation ablation over 10K steps RL steps on Qwen-32B-Base [src](https://arxiv.org/abs/2501.12948) we conduct large-scale RL training on Qwen-32B...
40 deepseek-r1 author count 195 authors (v1 appendix, incl. departed members) [src](https://arxiv.org/abs/2501.12948) (Appendix A Contributions and Acknowledgments:...
41 deepseek-v2 author count 158 authors [src](https://arxiv.org/abs/2405.04434) (Appendix A Contributions and Acknowledgments ...
42 deepseek-v3.2 post-training budget vs pretraining 10 percent of pre-training cost (lower bound) [src](https://arxiv.org/abs/2512.02556) this framework allocates a post-training compu... via GitHub-mirrored transcription
43 deepseek-v3.2 DSA continued pretraining tokens 2.1e9 (dense warm-up) + 943.7e9 (sparse) tokens [src](https://github.com/deepseek-ai/DeepSeek-... We train the indexer for only 1000 steps, with... V3.2-Exp report PDF in repo
44 deepseek-v3.2 knowledge gap attributed to FLOPs [src](https://arxiv.org/abs/2512.02556) due to fewer total training FLOPs, the breadth...
45 deepseek-v3.2 agentic RL environments 1800 / 85000 environments / prompts [src](https://arxiv.org/abs/2512.02556) over 1,800 distinct environments and 85,000 co...
46 deepseek-v3.1 continued pretraining tokens 839e9 (630e9 at 32K + 209e9 at 128K) tokens [src](https://huggingface.co/deepseek-ai/DeepS... The 32K extension phase has been increased 10-... DeepSeek-V3.1-Base model card README, fetched ...
47 deepseek-v4 pretraining tokens 33T (Pro) / 32T (Flash) tokens [src](https://arxiv.org/abs/2606.19348) pre-train both models on more than 32T diverse... Confirmed in report Sec 4 (https://arxiv.org/h...
48 deepseek-v4 efficiency vs V3.2 27% FLOPs / 10% KV cache relative single-token inference cost at 1M con... [src](https://arxiv.org/abs/2606.19348) DeepSeek-V4-Pro requires only 27% of single-to...
49 lab-level Fire-Flyer 2 cluster 10000 PCIe A100 GPUs (built 2021) [src](https://arxiv.org/abs/2408.14158) Fire-Flyer 2 with 10,000 PCIe A100 GPUs, achie... SC24 paper; abstract via mirrors
50 lab-level headcount (press) ~160 employees (Dec 2024) [src](https://www.tomshardware.com/tech-indust... Deepseek has 'about 160' employees (sources sp... from search snippet; page blocked
51 deepseek-v3.2 author count 263 named authors (212 R&E + 27 Data Annotation + ... [src](https://arxiv.org/abs/2512.02556) (Appendix 'Author List' of https://arxiv.org/h... Replaces the earlier 342 in the notes, which m...
52 deepseek-v4 author count 318 named authors (270 R&E + 48 Business & Complia... [src](https://arxiv.org/abs/2606.19348) (Appendix A.1 of https://arxiv.org/html/2606.1...

5.8 Key numbers

project run_id run_type stage params_total tokens flops flops_method gpu_hours loss benchmark_name benchmark_value confidence source_url
0 deepseek-llm dsllm-7b-final final pretraining 6.90e+09 2.00e+12 8.50e+22 M*D with M=72*n_layer*d_model^2+12*n_layer*d_m... NaN NaN MMLU (5-shot) 48.2 reported [src](https://arxiv.org/abs/2401.02954)
1 deepseek-llm dsllm-67b-final final pretraining 6.70e+10 2.00e+12 9.90e+23 M*D; 95 layers, d=8192, GQA 8 kv heads, l_seq=... 601000.0 NaN MMLU (5-shot) 71.3 derived [src](https://arxiv.org/abs/2405.04434)
2 deepseek-llm dsllm-7b-chat final post-training/RL 6.90e+09 NaN NaN GSM8K (chat) 63.0 reported [src](https://arxiv.org/abs/2401.02954)
3 deepseek-llm dsllm-67b-chat final post-training/RL 6.70e+10 NaN NaN GSM8K (chat) 84.1 reported [src](https://arxiv.org/abs/2401.02954)
4 deepseek-llm dsllm-hparam-grid-1e17 ladder pretraining 1.00e+17 paper's compute budget C = M*D (non-embedding ... NaN NaN NaN reported [src](https://arxiv.org/abs/2401.02954)
5 deepseek-llm dsllm-hparam-sweep-1e17-2e19 ladder pretraining 2.00e+19 compute budgets ranging from 1e17 to 2e19 (val... NaN NaN NaN reported [src](https://arxiv.org/abs/2401.02954)
6 deepseek-llm dsllm-hparam-validation-1e20 ladder pretraining 1.00e+20 paper compute budget NaN NaN NaN reported [src](https://arxiv.org/abs/2401.02954)
7 deepseek-llm dsllm-isoflop-budget-min-1e17 ladder pretraining 2.30e+07 7.10e+08 1.00e+17 paper compute budget C=M*D; params derived as ... NaN NaN NaN derived [src](https://arxiv.org/abs/2401.02954)
8 deepseek-llm dsllm-isoflop-budget-max-3e20 ladder pretraining 1.60e+09 3.20e+10 3.00e+20 paper compute budget C=M*D; params derived as ... NaN NaN NaN derived [src](https://arxiv.org/abs/2401.02954)
9 deepseek-llm dsllm-scaling-data-quality-suite ladder pretraining NaN NaN NaN reported [src](https://arxiv.org/abs/2401.02954)
10 deepseek-v2 dsv2-final-pretrain final pretraining 2.36e+11 8.10e+12 1.02e+24 6*N_active*D = 6*2.1e10*8.1e12 1400000.0 0.606 MMLU (5-shot) 78.5 derived [src](https://arxiv.org/abs/2405.04434)
11 deepseek-v2 dsv2-context-ext-128k midtrain midtraining 2.36e+11 1.90e+10 2.40e+21 6*N_active*D; tokens derived as 1000 steps x 5... NaN NaN NaN derived [src](https://arxiv.org/abs/2405.04434)
12 deepseek-v2 dsv2-chat-sft-rl final post-training/RL 2.36e+11 NaN NaN GSM8K (chat RL) 92.2 reported [src](https://arxiv.org/abs/2405.04434)
13 deepseek-v2 dsv2-lite-final final pretraining 1.57e+10 5.70e+12 8.20e+22 6*N_active*D NaN NaN MMLU 58.3 reported [src](https://arxiv.org/abs/2405.04434)
14 deepseek-v2 dsv2-abl-attn-7b-mqa ablation pretraining 7.10e+09 1.33e+12 5.67e+22 6*N*D NaN NaN MMLU (5-shot) 37.9 reported [src](https://arxiv.org/abs/2405.04434)
15 deepseek-v2 dsv2-abl-attn-7b-gqa8 ablation pretraining 6.90e+09 1.33e+12 5.51e+22 6*N*D NaN NaN MMLU (5-shot) 41.2 reported [src](https://arxiv.org/abs/2405.04434)
16 deepseek-v2 dsv2-abl-attn-7b-mha ablation pretraining 6.90e+09 1.33e+12 5.51e+22 6*N*D NaN NaN MMLU (5-shot) 45.2 reported [src](https://arxiv.org/abs/2405.04434)
17 deepseek-v2 dsv2-abl-mla-small-mha ablation pretraining 1.58e+10 1.33e+12 2.00e+22 6*N_active*D NaN NaN MMLU (5-shot) 48.7 reported [src](https://arxiv.org/abs/2405.04434)
18 deepseek-v2 dsv2-abl-mla-small-mla ablation pretraining 1.57e+10 1.33e+12 1.90e+22 6*N_active*D NaN NaN MMLU (5-shot) 50.0 reported [src](https://arxiv.org/abs/2405.04434)
19 deepseek-v2 dsv2-abl-mla-large-mha ablation pretraining 2.51e+11 4.20e+11 6.30e+22 6*N_active*D NaN NaN MMLU (5-shot) 57.5 reported [src](https://arxiv.org/abs/2405.04434)
20 deepseek-v2 dsv2-abl-mla-large-mla ablation pretraining 2.47e+11 4.20e+11 5.40e+22 6*N_active*D NaN NaN MMLU (5-shot) 59.0 reported [src](https://arxiv.org/abs/2405.04434)
21 deepseek-v3 dsv3-final-pretrain final pretraining 6.71e+11 1.48e+13 3.30e+24 6*N_active*D = 6*3.7e10*1.48e13 (excludes MTP ... 2664000.0 0.548 MMLU (5-shot EM) 87.1 reported [src](https://arxiv.org/abs/2412.19437)
22 deepseek-v3 dsv3-context-ext-32k-128k midtrain midtraining 6.71e+11 1.26e+11 2.80e+22 6*N_active*D; tokens derived: 1000 steps x 192... 119000.0 NaN NaN derived [src](https://arxiv.org/abs/2412.19437)
23 deepseek-v3 dsv3-post-training final post-training/RL 6.71e+11 5000.0 NaN MATH-500 (chat EM) 90.2 reported [src](https://arxiv.org/abs/2412.19437)
25 deepseek-v3 dsv3-abl-mtp-small-baseline ablation pretraining 1.57e+10 1.33e+12 1.90e+22 6*N_active*D (2.4B activated params, V3 report... NaN NaN NaN reported [src](https://arxiv.org/abs/2412.19437)
26 deepseek-v3 dsv3-abl-mtp-small-mtp ablation pretraining 1.57e+10 1.33e+12 1.90e+22 6*N_active*D (2.4B activated params, V3 report... NaN NaN NaN reported [src](https://arxiv.org/abs/2412.19437)
27 deepseek-v3 dsv3-abl-mtp-large-baseline ablation pretraining 2.29e+11 5.40e+11 6.77e+22 6ND (active params) NaN NaN HumanEval (Pass@1) 44.5 reported [src](https://arxiv.org/abs/2412.19437)
28 deepseek-v3 dsv3-abl-mtp-large-mtp ablation pretraining 2.29e+11 5.40e+11 6.77e+22 6ND (active params) NaN NaN HumanEval (Pass@1) 53.7 reported [src](https://arxiv.org/abs/2412.19437)
29 deepseek-v3 dsv3-abl-auxfree-small-pair ablation pretraining 1.57e+10 1.33e+12 3.83e+22 6ND x 2 (baseline + variant) NaN NaN NaN reported [src](https://arxiv.org/abs/2412.19437)
30 deepseek-v3 dsv3-abl-auxfree-large-pair ablation pretraining 2.29e+11 5.78e+11 1.45e+23 6ND x 2 (baseline + variant) NaN NaN HumanEval (Pass@1) aux-loss -> aux-free NaN reported [src](https://arxiv.org/abs/2412.19437)
31 deepseek-v3 dsv3-abl-fp8-16b-pair ablation pretraining 1.60e+10 1.33e+12 3.83e+22 6ND x 2 (baseline + variant) NaN NaN NaN reported [src](https://arxiv.org/abs/2412.19437)
32 deepseek-v3 dsv3-abl-fp8-230b-pair ablation pretraining 2.30e+11 9.00e+11 2.27e+23 6ND x 2 (baseline + variant) NaN NaN NaN reported [src](https://arxiv.org/abs/2412.19437)
33 deepseek-v3 dsv3-abl-balance-1b-seqwise ablation pretraining 1.00e+09 NaN 2.258 NaN reported [src](https://arxiv.org/abs/2412.19437)
34 deepseek-v3 dsv3-abl-balance-1b-auxfree ablation pretraining 1.00e+09 NaN 2.253 NaN reported [src](https://arxiv.org/abs/2412.19437)
35 deepseek-v3 dsv3-abl-balance-1b-batchwise ablation pretraining 1.00e+09 NaN 2.253 NaN reported [src](https://arxiv.org/abs/2412.19437)
36 deepseek-v3 dsv3-abl-expertload-16b-pair ablation pretraining 1.60e+10 NaN NaN NaN reported [src](https://arxiv.org/abs/2412.19437)
37 deepseek-r1 r1-prep-30b-a100 ablation post-training/RL 3.00e+10 NaN NaN NaN reported [src](https://arxiv.org/abs/2501.12948v2)
38 deepseek-r1 r1-zero final post-training/RL 6.71e+11 101000.0 NaN AIME 2024 pass@1 71.0 reported [src](https://arxiv.org/abs/2501.12948v2)
39 deepseek-r1 r1-sft-data-creation final post-training/RL 5000.0 NaN NaN reported [src](https://arxiv.org/abs/2501.12948v2)
40 deepseek-r1 r1-final final post-training/RL 6.71e+11 41000.0 NaN AIME 2024 pass@1 79.8 reported [src](https://arxiv.org/abs/2501.12948v2)
41 deepseek-r1 r1-zero-qwen-32b ablation post-training/RL 3.20e+10 NaN NaN AIME 2024 pass@1 47.0 reported [src](https://arxiv.org/abs/2501.12948)
42 deepseek-r1 r1-distill-series final post-training/RL NaN NaN AIME 2024 pass@1 (Distill-Qwen-32B) 72.6 reported [src](https://arxiv.org/abs/2501.12948)
43 deepseek-v3.1 dsv31-continued-pretrain-840b midtrain midtraining 6.71e+11 8.39e+11 1.90e+23 6*N_active*D 150000.0 NaN NaN derived [src](https://huggingface.co/deepseek-ai/DeepS...
44 deepseek-v3.2 dsv32exp-dsa-dense-warmup midtrain midtraining 6.71e+11 2.10e+09 4.66e+20 6ND NaN NaN NaN reported [src](https://github.com/deepseek-ai/DeepSeek-...
45 deepseek-v3.2 dsv32exp-dsa-sparse-stage midtrain midtraining 6.71e+11 9.44e+11 2.10e+23 6*N_active*D (attention now sparse; rough) NaN NaN NaN reported [src](https://github.com/deepseek-ai/DeepSeek-...
46 deepseek-v3.2 dsv32-final-post-training final post-training/RL 6.71e+11 300000.0 NaN AIME 2025 93.1 derived [src](https://arxiv.org/abs/2512.02556)
47 deepseek-v3.2 dsv32-speciale final post-training/RL 6.71e+11 NaN NaN AIME 2025 96.0 reported [src](https://arxiv.org/abs/2512.02556)
48 deepseek-v4 dsv4-pro-pretrain final pretraining 1.60e+12 3.30e+13 9.70e+24 6*N_active*D = 6*4.9e10*3.3e13 NaN NaN NaN derived [src](https://arxiv.org/abs/2606.19348)
49 deepseek-v4 dsv4-flash-pretrain final pretraining 2.84e+11 3.20e+13 2.50e+24 6*N_active*D NaN NaN NaN derived [src](https://arxiv.org/abs/2606.19348)

5.9 Narrative

Hardware and accounting. V3 trained on 2,048 H800s; the report gives per-stage GPU-hours (2.664M pretraining, 119K context extension, 5K post-training). V2’s report gives per-trillion-token rates (300.6K and 172.8K GPU-hours per T tokens for the dense 67B and the MoE V2), which this chapter uses to derive GPU-hours for the 67B and V2 finals; those rows are flagged derived.

What is not reconstructable. The compute behind architecture and data decisions for every release after the first; any prediction error for the DeepSeek LLM law; active-parameter counts for two V3 ablation models (flagged as guesses); GPU-hours for V4 (33T and 32T tokens, no hours disclosed). Primary hosts were blocked from this environment; text came from DeepSeek’s GitHub-hosted PDFs and GitHub transcriptions of the arXiv papers, marked as such in the claims.