2  Marin (Stanford CRFM)

Marin runs every experiment as a public GitHub issue with W&B links, which makes it the only lab whose whole experiment record can be browsed: 446 issues carry the experiment label as of this writing, and the 8B and 32B retrospectives name every phase, restart and discarded cooldown. Final runs so far: Marin 8B (May 2025), Marin 32B (Oct 2025), Snowball 67B-A2B (2026), and a 535B-A23B hero run in progress. Working notes with sources are in data/notes/marin.md.

2.1 Marin at a glance

Two figures before the detail. First, the released and evaluated checkpoints over time, scored on Marin’s own 19-task base-model average and on MMLU, with circle area proportional to the cumulative pretraining compute behind each checkpoint. Post-training compute would be drawn as a ring around the disk, but the 8B instruct model’s SFT used 2.6e20 FLOPs, four ten-thousandths of the base’s 6.1e23, so the ring would be indistinguishable from the disk and is omitted; the 32B has no post-trained release. The 2026 runs are missing from this figure because none has a published benchmark table yet: Delphi’s held-out models (March 2026), Snowball 67B-A2B (pretraining finished August 2026) and the 535B-A23B hero run (in progress) have public loss curves, which the next figure uses, but no evaluations.

Two released dense models in five months is a short series, and the 32B’s 19-task average is below the 8B’s despite twice the compute: the 19-task suite saturates on easy tasks and the 32B’s gains were on MMLU (+7), MMLU-Pro (+9) and BBH. The discarded Bison cooldown sits two points under Mantis on the same tokens.

Second, every Marin pretraining run with a public loss curve on one axis, three generations of recipe: the five-width December 2024 ladder and the 2025 dense runs (the 8B with all its phases stitched together, the three abandoned Tootsie trials, the 32B) in the hand-tuned Llama-style recipe; Delphi’s three held-out compute-optimal models from March 2026 in the AdamH recipe; and the 2026 mixture-of-experts runs, Snowball 67B-A2B and the 535B-A23B hero run as far as it has got. The second chart puts them on cumulative compute (6ND, active parameters for the MoEs), which is what makes sizes and recipes comparable. The first 20B tokens of each run are cut. The MoE curves are the Paloma c4_en loss under dropless routing (no capacity-factor token dropping); the hero run’s first segment logged a capacity-dropped evaluation instead, which sits about 0.13 nats higher at the same step and is not drawn.

Three readings from the compute axis.

Size, within one recipe. Where the 13B trial stopped (1.8e23 FLOPs) it was at 2.50 against the 8B’s 2.60 at the same compute; the 24B at 3.5e23 was 2.45 against 2.51. The 70B stopped at 6.5e23, just past the finished 8B’s 6.1e23, at 2.42 against the 8B’s final 2.35: on matched compute the smaller model won, because it had a cooldown and the 70B never did. That is the same lesson as the trajectory figure below, from the other side: without a cooldown a trial’s loss understates the model. The ladder’s small models, trained to 90 to 900 tokens per parameter, sit well above everything else at equal compute, which is what over-training past the compute-optimal point costs.

Recipe, at equal compute. Delphi’s 1e23 model reached 2.34 on 9.4e22 FLOPs. The 32B, on the 2025 recipe, was at 2.42 when it had spent the same compute, and the 8B at 2.55; the finished 32B needed 1.2e24, thirteen times as much, to get to 2.20. Delphi’s 1e22 model likewise beats the 8B at matched compute by 0.12. Some of that is compute-optimal allocation rather than recipe, since the 2025 runs were far past optimal tokens per parameter at any given point, but it is the cleanest cross-generation comparison Marin’s record allows.

Architecture. Snowball, at 1.2e23 FLOPs of active-parameter compute, ended at 2.27, below Delphi’s 25B dense model at similar compute and below where the 32B was at that compute by 0.12. The hero run at 4.6T tokens (6.3e23 FLOPs so far) is at 2.29, level with where the 32B was at the same compute, and still at full learning rate with three quarters of its tokens to go. The 6ND-on-active-parameters convention flatters MoEs, since it ignores the memory and communication cost of the inactive experts, and the two MoE curves should be read against each other and against the Marin MoE law rather than as a like-for-like efficiency gain.

Bookkeeping: the 13B trial’s curve starts at 1.7T tokens because only its longest restart segment was fetched; the trials’ token axes use the average tokens per step from W&B, since their batch ramps are not public; Snowball’s active parameter count (about 2B) is from the model name, not a stated figure.

2.2 Returns to post-training compute

Marin has published no RL curve, and its only released post-trained model is the SFT-only 8B instruct. But one 2026 experiment (issue 3956, April to May 2026) is exactly a returns-to-SFT sweep: the Marin 8B base fine-tuned on OpenThoughts-4 math subsets of 256, 512, 1,024, 2k, 5k, 10k and 50k examples, each with reasoning traces written by one of eight teacher models, three training repeats per cell; the 32B base on 10k and 50k with four teachers. Every fine-tune was scored with evalchemy on AIME 24 and 25, AMC23, HMMT and MATH500, averaged over 3 to 10 sampling seeds, and the results are public W&B runs. data/posttraining/marin_sft_scaling.csv holds the 1,014 averaged evaluations; src/fetch_evalchemy.py rebuilds it. SFT compute is not logged, so the x-axis is 6ND at an assumed 10,000 tokens per example (OpenThoughts traces run to the 32k cap); the assumption shifts every point by the same factor.

Four readings.

The shape depends on the benchmark. MATH500 rises close to log-linearly in SFT compute, from 13 to 28 percent at 256 examples to 66 to 83 percent at 50k on the 8B, roughly seven points per doubling of data. AIME24 is flat within noise, under 4 percent, through 10k examples and then jumps to 18 to 36 percent at 50k. A competition benchmark reads as a threshold; a broad one reads as a smooth return.

The base model carries through. At the same data and teacher, the 32B lands 10 to 20 AIME24 points above the 8B (47 against 36 percent with Qwen3-30B-A3B traces at 50k). Pretraining compute and post-training compute are not substitutes here; the larger base gets more from every example.

The teacher matters as much as the data. At 50k examples on the 8B, traces from Qwen3-30B-A3B-Thinking give 36 percent on AIME24 and traces from Qwen3-4B give 18, at identical SFT compute. What is being bought is distilled capability from a stronger model, so this is a return to distillation, not to post-training compute alone, and it cannot be read across to RL.

The compute is tiny. The 50k-example run is about 2.4e19 FLOPs for the 8B under the stated assumption, four hundred-thousandths of the 6.1e23 pretraining run; the whole sweep, some 130 fine-tunes, is under 1e21. Whatever the right functional form, the marginal return per FLOP in this regime is orders of magnitude above the pretraining curve’s, which is the standard argument for why labs moved spend into post-training. Marin’s own move is the Snowball post-training pipeline, which had no published numbers as of September 2026.

2.3 Compute and cost

Marin has never published a dollar figure, a GPU-hour total or a budget. What exists: the compute was donated (Google’s TPU Research Cloud for the 2024 and 2025 runs and for Delphi; the Jen-Hsun and Lori Huang Foundation’s CoreWeave GB200 allocation for the 535B run, out of a total foundation purchase of about $108M of CoreWeave compute for AI research that is not broken down by project), development funding came from the Siegel Family Endowment and Schmidt Sciences, and the team grew from one to about ten FTE between 2025 and 2026. The table converts the 6ND compute of every sized Marin run into the book’s common yardstick, H100-hours at 40 percent utilisation and $2 per hour; these are what the runs would have cost at cloud list price on GPUs, not what Marin paid, which was nothing in cash for the accelerators.

project run_type flops H100-hours (40% MFU) USD at $2/H100-h
0 delphi ladder 1.19e+23 82,603 $0.17M
1 marin-13b aborted 1.77e+23 122,917 $0.25M
2 marin-24b aborted 3.50e+23 243,056 $0.49M
3 marin-32b final 1.03e+24 715,278 $1.43M
4 marin-32b midtrain 2.10e+23 145,833 $0.29M
5 marin-32b aborted 2.12e+23 147,167 $0.29M
6 marin-535b-a23b ladder 6.72e+21 4,665 $0.01M
7 marin-535b-a23b final 2.70e+24 1,875,000 $3.75M
8 marin-70b aborted 6.47e+23 449,306 $0.90M
9 marin-8b ladder 6.58e+21 4,573 $0.01M
10 marin-8b ablation 4.80e+21 3,333 $0.01M
11 marin-8b final 4.96e+23 344,622 $0.69M
12 marin-8b midtrain 1.13e+23 78,472 $0.16M
13 marin-ablations ablation 4.80e+20 333 $0.00M
14 marin-moe ladder 1.11e+23 76,875 $0.15M
15 marin-snowball-67b-a2b final 1.20e+23 83,333 $0.17M

All sized Marin runs together: 6.3e+24 FLOPs, about 4.4M H100-hours or $9M at list price; the 535B hero run alone is 43% of it. Rows typed final include the released 8B and 32B and the Snowball and 535B runs; aborted is the discarded 32B cooldown, rescues and the 13B/24B/70B trials; ladder is Delphi, the MoE sweeps and the 2024 width ladder.

The share that did not ship (aborted plus ladder plus ablation rows) is the experiment share this book is after, and for Marin it is a floor: the 446-issue long tail of small ablations has no compute attached, and the 2026 MoE screening cells (about 80 runs at d512 to d1280) are one row. People are the other missing half. At ten FTE and a fully loaded US research-engineer cost of $300k to $500k a year, Marin’s labour bill is a few million dollars a year, the same order as the list-price value of a year of its compute; but neither number is Marin’s, and the lab has not published either.

2.4 Algorithmic efficiency across recipe generations

The cleanest measure of algorithmic progress a lab’s own record allows is the compute its finished runs needed to reach a given loss, generation by generation. Finished matters: under a warmup-stable-decay schedule the loss drops at the cooldown wherever it happens to fall, so the point along a curve where it first crosses a threshold mostly measures where the cooldown was, not how good the recipe is. The figure below therefore uses only endpoints: the five December 2024 ladder runs, the two 2025 dense releases, Delphi’s three 2026 held-out models, and Snowball. The 13B, 24B and 70B trials and the running hero are excluded because they never cooled down. A power law is drawn through each generation’s endpoints; dashed where extrapolated.

At c4_en 2.35 (the finished 8B’s loss): 2025 recipe 6.0e+23 FLOPs, Delphi 8.2e+22, a 7.3x reduction in about ten months. At Snowball’s 2.268: the 2025 line needs 8.9e+23, Delphi’s line (extrapolated 2.6x past its largest run) 2.4e+23, Snowball used 1.2e+23: 7.4x over 2025 and 2.0x over Delphi, on active-parameter compute.

Three cautions on reading this as a rate of algorithmic progress. The generations differ in data as well as recipe (DCLM and Nemotron-CC for the 2025 runs, a Nemotron, StarCoder and ProofPile mix for Delphi), and c4_en is a web-text loss, so part of the gap is data mix. The 2025 runs were trained far past compute-optimal tokens per parameter, so part of the gap is allocation rather than optimizer or architecture; Delphi’s own contribution was to make the allocation principled. And MoE compute on active parameters ignores the memory and communication cost of the inactive experts, which is why Marin quotes its MoE gain as 6.7x theoretical but 3.6x realised on TPU v4. Marin’s own stated multipliers, on its Paloma macro-loss metric, are: the May 2026 recipe law against the April MoE law, which at equal loss implies about 2.1x less compute in seven weeks (the coefficient fell from 95.18 to 88.90 at a shared exponent of 0.0941); the MoE recipe against Delphi, 3.6x realised; and the 2025 optimizer benchmark, at most 1.4x over AdamW at 130M shrinking to about 1.1x at 1.2B. Taken together with the endpoint frontier, Marin’s record is consistent with something like a 5 to 10x fall in compute-to-loss between the 2025 and 2026 recipes, most of it from compute-optimal allocation and the AdamH recipe, with MoE adding a further two-fold on the active-parameter accounting.

2.5 All scaling experiments over time

Every Marin experiment with a date, on one time axis. The six families that produced a scaling curve are drawn as insets on shared axes (compute 1e17 to 1e25 FLOPs on x, loss 2.0 to 3.8 on y), each above its date, so their shapes and ranges can be compared directly; every other dated experiment in experiments.csv is a tick on the timeline, coloured by stage. The insets do not share a metric: the ladder and Tootsie panels are Paloma c4_en, Delphi is its Nemotron-mix eval loss, and the two MoE laws are Paloma macro loss, so compare shapes across panels, not levels.

The picture is of a lab that ran one width ladder, then trained by hand for a year, then built the apparatus: within five months of March 2026 it had an IsoFLOP suite validated at 300x, two successive MoE laws on the same functional form, a learning-rate law, and a hero-shape ladder. The 2025 panel is the only one whose curves are actual training trajectories rather than fitted laws or ladder endpoints; everything before it is five points and everything after it is a formula plus a validation star.

2.6 What is unusual here

Marin’s flagship 2025 runs were not preceded by a scaling ladder. The 8B was chosen “mostly out of pragmatism” (reserved capacity) and started “with what we had”; the 32B was “largely a scale-up of the 8B recipe” and needed two rescues and a discarded cooldown. In 2026 the lab built the machinery it had lacked: Delphi (an 81-run IsoFLOP suite whose pre-registered forecast hit the 1e23 run within 0.2 percent at a 300x extrapolation), a mixture-of-experts law validated at 1e21, a learning-rate law that removed per-size sweeps, and a four-rung ladder costing about 1 percent of the 535B run. So this chapter shows both regimes in one lab: scale-up by hand, then scale-up by law.

2.7 Compute that did not ship

Where OLMo’s development share comes from an accounting paper, Marin’s is visible run by run. The 32B programme discarded a full 1.07T-token cooldown (about 2.1e23 FLOPs) after GSM8K contamination and a data-permutation artefact were traced, and abandoned an optimizer-state rebuild and a Muon swap. Three larger Tootsie runs (13B, 24B, 70B) were opened and closed with empty results. The bar below counts only runs with a size attached, so the 446-issue long tail of small ablations is missing from it; it is a floor.

Non-final share of logged FLOPs: 31%; aborted alone: 22%.

2.8 The final runs as trajectories

runs.csv gives each phase of the 8B and 32B runs its own tokens and FLOPs. That is right for the compute split and wrong for the loss: the loss at the end of Phoenix reflects every token since step 0, not the 6.3T on that row. The figure below puts the W&B curves on the honest axis, cumulative training tokens (the trainer’s global step times the batch schedule, which reproduces the model card’s phase totals to within 0.03T), and draws each phase, cooldown and abandoned side branch as one piece of one trajectory. The metric is the Paloma c4_en eval loss throughout; train loss moves with the data mix and is not comparable across phases. The first 0.3T tokens are cut so the warm-up does not set the scale.

Start and end of every drawn segment (start = first eval point after the cut):

phase run_type tokens_start_T tokens_end_T loss_start loss_end delta
0 Kestrel final 0.315 2.768 2.616 2.546 -0.071
1 Ocelot final 2.776 3.745 2.585 2.600 0.015
2 Jellyfish (cooldown 1) midtrain 3.757 4.776 2.598 2.605 0.007
3 Phoenix (rewarm) final 4.789 11.085 2.567 2.505 -0.062
4 Starling (cooldown 2) midtrain 11.085 12.410 2.489 2.352 -0.137
5 Deeper Starling final 12.410 12.746 2.352 2.346 -0.006
6 dessert, Raccoon, Spoonbill (off Jellyfish) ablation 4.776 4.990 2.605 2.609 0.004
7 exp898 Raccoon (deep cooldown 1.7e-4->1.7e-5 f... ablation 4.776 4.902 2.605 2.612 0.007
8 exp916 Spoonbill (cooldown to 3e-5 + Tulu 0.3%... ablation 4.839 4.902 2.596 2.589 -0.008
9 Phase 1 (Llama-3-style) final 0.201 2.651 2.623 2.292 -0.331
10 Phase 3 (QK-Norm) final 2.718 5.671 2.286 2.265 -0.021
11 Mantis cooldown midtrain 5.402 6.442 2.266 2.199 -0.067
12 exp1390_32b_necro (optimizer-state rebuild res... aborted 2.718 2.785 2.290 2.291 0.001
13 Muon and necro rescues (discarded) aborted 2.617 2.886 2.261 2.306 0.045
14 Bison (discarded) aborted 5.402 6.409 2.266 2.221 -0.045

Four things are visible here that no per-row number shows.

Stable phases are nearly flat on this metric. Kestrel’s constant-learning-rate stretch from 2T to 2.77T moved the loss by under 0.01. Phoenix spent 5.5T tokens, 43 percent of the whole run, between 2.52 and 2.50. Phase 3 of the 32B bought 0.02 over 3T tokens. Under a warmup-stable-decay schedule the stable phase is banking capability that only shows up in the decay, so most of the tokens in both runs look wasted on a loss-versus-tokens plot and are not.

The cooldowns do the visible work. Starling, the second 8B cooldown, is 1.3T tokens or 10 percent of the run, and took the loss from 2.49 to 2.35: a larger drop than the entire 9T between 2T and 11T (0.06). The 32B’s Mantis cooldown bought 0.07 in 1.07T tokens against phase 3’s 0.02 in 3T. A final-run loss is therefore as much a measurement of cooldown depth as of scale, which matters when the cross-lab chapter compares final losses across labs.

The first cooldown bought nothing on c4_en. Jellyfish ended at 2.605, above the 2.585 where Ocelot started, while its benchmark gains were real (MMLU 65.3, the reason it was kept). The 70/30 high-quality mix moves the model away from web text, and c4_en is web text. The three short cooldown branches off Jellyfish (dessert, Raccoon, Spoonbill) all land within 0.02 of it. Their value was diagnostic: Raccoon and Spoonbill isolated the lm_head norm explosion that led to z-loss in Starling.

Restarts and rescues leave marks. Every phase boundary shows a transient: the batch and learning-rate change into Ocelot, the rewarm into Phoenix, a spike at 5.4T. The 32B’s phase 1 spikes are visible even in the sparse eval curve, and its two rescue attempts sit on top of the trunk and go nowhere (the Muon swap ends 0.045 above where it began). Bison and Mantis ran the same 1.07T tokens; Mantis sits 0.02 below Bison at matched tokens, so the permutation fix and MegaMath moved the loss, not only the contaminated GSM8K score.

The trajectory also corrects a comparison made in earlier drafts of this chapter, which read the abandoned 13B, 24B and 70B trials’ final losses against the finished 8B’s 2.35. Those trials never had a cooldown, so that reading is unfair to them. At matched tokens, with no cooldown on either side, the size advantage was real and grew with size:

run_id params tokens_T loss 8B trunk loss at same tokens gap
12 exp860 Tootsie 13B 13B 2.27 2.498 2.548 -0.050
13 exp861 Tootsie 24B 24B 2.43 2.448 2.546 -0.098
14 exp750 Tootsie 70B 70B 1.54 2.425 2.555 -0.130

What Marin gave up in April 2025 was that gap; the losses on the rows understate it.

2.9 The mixture-of-experts law and its held-out point

Marin fitted loss = 1.6 + 95.18 · C^-0.0941 on a 1e18 to 1e20 IsoFLOP sweep and then trained a 1e21 test: predicted 2.606, measured 2.599. That is the experiments-predict-the-final-run test in miniature, and the residual is the number the cross-lab chapter wants.

Log residual at 1e21: -0.0027. Elasticity of loss to compute at 1e21 from the law: -0.036; at the 2.7e24 hero-run scale it falls to -0.022 because the pinned irreducible term dominates.

2.10 Run sizes, compute split, timeline

Losses come from the public marin-community/marin W&B project (data/losses/marin/; the metric is the Paloma c4_en eval loss, the one the retrospectives plot, in nats per token with the Llama 3 tokenizer). Eighteen runs are covered: every 8B phase and cooldown, the 13B, 24B and 70B trials, and the 32B phases, rescues and cooldowns. The standard panel below draws them without a fitted law, for the reason the trajectory figure makes visible: most rows with a loss are continuation phases whose flops is that phase’s compute, not the cumulative compute the loss reflects, and the cooldown losses are on a different learning-rate footing from the pretraining phases. Read the abandoned trials against the 8B at matched tokens (table above), not against the finished stars. Delphi’s 81-run grid is one row here; its per-run sizes are in the repository’s hard-coded suite and can be expanded later.

One clean experiment series does exist: a five-width ladder from December 2024 (tootsie-scaling-*, hidden 512 to 2048 at 16 layers, each 210B tokens of DCLM-baseline, so 90 to 900 tokens per parameter rather than compute-optimal). Fitting a two-parameter power law to those five points and extrapolating to the released 8B and 32B is the report’s basic move, shown here on Marin’s own data; the residuals are in the table.

run_id flops loss predicted log_residual
0 marin-8b-base-deeper-starling (exp600_tootsie,... 6.100000e+23 2.3460 1.9455 0.1872
15 marin-32b-base Mantis (aggregate final artifact) 1.240000e+24 2.1993 1.8554 0.1700

The ladder is not a compute-optimal sweep, the law has no irreducible term (so it keeps falling where a real curve flattens), and the finals sit at the end of long cooldowns on a different data mix, so the residual mixes all three effects; a 200x to 400x extrapolation from five points is a demonstration of the method, not a test of Marin’s forecasting. It is still the only Marin curve that runs from experiments to finals on one metric.

Experimental share of logged FLOPs: 24% (a lower bound: unlogged and failed runs are missing).

2.11 Published scaling experiments

project date stage intervention model_sizes compute_range metric fitted_exponent exponent_source raw_data_available source_url notes
0 delphi 2026-05-11 pretraining Scaling suite / scaling law: IsoFLOP sweeps (Q... d512-d4096 grid (~1e8-1e10 params); optima 550... 3e18-3e20 fit; 1e21/1e22/1e23 held out eval loss (Paloma/Nemotron); D*~A*C^alpha from... not accessible (alpha for D* vs C not in reach... figure_only logs [src](https://openathena.ai/blog/delphi/ ; htt... Forecast of 1e23 run within 0.2% (300x extrapo...
1 delphi 2026-04-01 pretraining Scaling-law extrapolation check ('at least 100... as above ~1e21 ladder cost vs 1e23 target loss figure_only figure_only [src](https://x.com/WilliamBarrHeld/status/203... HF commit 'blog update: 100x -> 300x'.
2 marin-moe 2026-04-06 pretraining MoE v16 AdamH IsoFLOP sweep -> compute-optimal... d512-d2560 (MoE, 64 experts); recommendation 1... 1e18-1e20 fit; 1e21 validation Paloma macro loss; loss(C)=1.6+95.18*C^-0.0941 0.0941 (L_inf pinned 1.6) numbers_in_text logs [src](https://github.com/marin-community/marin... Predicted 2.606 vs measured 2.599 at 1e21; unc...
3 marin-moe 2026-05-30 pretraining May Recipe (MuonH, QB router, half-RoPE, XSA, ... d512-d1792 1e18-3e19 (1e18 dropped from fit) Paloma macro; loss=1.6+88.90*C^-0.0941 (README... 0.0941 (pinned to v16) numbers_in_text logs [src](https://github.com/marin-community/marin... 2.12-2.22x equal-throughput compute-equivalent...
4 marin-moe 2026-05-26 pretraining MuonH learning-rate scaling law sweep (~150 runs) d512-d1280 4-240 tokens/param (+TPP 5000) loss-optimal LR; LR=26.9*tokens^-0.395*dim^-0.... -0.395 (tokens), -0.121 (dim), +0.5 (batch); R... numbers_in_text logs [src](https://github.com/marin-community/marin... Adopted in heuristic as adam_lr=0.06602*tokens...
5 marin-moe 2026-03-31 to 2026-08-20 pretraining Agent MoE digest: 80 small-scale A/B experimen... d512-d1280 MoE cells ~1e17-1e20 per cell model-FLOPs speedup via scaling-law inversion ... numbers_in_text logs [src](https://github.com/marin-community/marin... 16 worked / 9 promising / 13 mixed / 32 did no...
6 marin-535b-a23b 2026-08 pretraining 4-rung hero-shape scaling ladder to forecast a... 1.6B-A61M (48B tok) ... 27.7B-A1.2B (926B tok) ~1.8e19 to ~6.7e21 per rung; ladder ~1% of 2.7... loss, grad norm, token dropping, eval trajecto... figure_only figure_only [src](https://x.com/percyliang/status/20909180... Forecast loss value not in accessible text (W&...
7 marin-moe 2026-03-17 pretraining MoE architecture + quantile balancing at 1e21 ... 14B-A2B (75B tok); 35B-A5B (326B tok) 1e21-1e22 paloma/macro_loss; predicted 2.3887 at 1e22 numbers_in_text logs [src](https://github.com/marin-community/marin... Blog: zero loss spikes, no aux loss.
8 marin-8b 2024-12 to 2025-06 pretraining Scaling laws to predict Tootsie 8B performance... not stated not stated c4_en bpb; lm_eval acc_norm; forced-choice bpb figure_only figure_only [src](https://github.com/marin-community/marin... Results section empty; two W&B reports (blocked).
9 marin-8b 2025-02 pretraining Scaling-law experiment plan: ladders of ~5 mod... up to 1.4B, projected to 8B-22B not stated loss/bpb figure_only figure_only [src](https://github.com/marin-community/marin... Closed 2025-08-24; unclear how much executed.
10 marin-ablations 2025-01-18 pretraining AdamW hyperparameter sweep across 3 (N,D) sett... 0.1B, 0.5B 10B-50B tokens loss; optimal hparams stable numbers_in_text logs [src](https://github.com/marin-community/marin... lr*width=1.6e-2*512 rule.
11 marin-ablations 2025-05-17 pretraining Optimizer benchmark: 10 optimizers, 4 sizes, 1... 130M-1.2B 1x-16x Chinchilla speedup over AdamW: <=1.4x (130M) shrinking to... numbers_in_text logs [src](https://github.com/marin-community/marin... Muon best at 1-4x; Soap/Kron at >=8x.
12 marin-ablations 2025-01-30 pretraining LR schedule (cosine, linear, WSD 20/40% const/... 0.5B 4.8e20 per run x7 loss numbers_in_text logs [src](https://github.com/marin-community/marin... Cosine best; high LR important; WSD 'isn't ter...
13 marin-ablations 2025-04 pretraining z-loss vs none from scratch 1.4B loss, lm_head norm numbers_in_text logs [src](https://github.com/marin-community/marin... Adopted z-loss 1e-4 as default.
14 marin-ablations 2025-05-06 pretraining MoE (8x1B) vs dense 1B, matched tokens 1B dense; 8x1B MoE train/eval loss, MFU numbers_in_text logs [src](https://github.com/marin-community/marin... MoE wins on tokens; dense better MFU; crossove...
15 marin-ablations 2024-11 pretraining Tokenizer choice (Llama3/Llama2/NeoX) small bpb numbers_in_text logs [src](https://github.com/marin-community/marin... Llama 3 tokenizer best.
16 marin-ablations 2025-01 pretraining High-quality many epochs vs lower-quality few ... loss/evals numbers_in_text logs [src](https://github.com/marin-community/marin... 'There's no data like more data'.
17 marin-ablations 2024 pretraining HTML->text extraction methods Paloma loss numbers_in_text logs [src](https://github.com/marin-community/marin... Format preservation helps.
18 marin-8b 2025-02 to 2025-05 midtraining Microannealing (70/30 anneals) to score cooldo... 8B checkpoint task accuracy + HQ eval loss numbers_in_text logs [src](https://github.com/marin-community/marin... 70% PT/15% FLAN/15% HQ best; HQ-only hurt task...
19 marin-8b 2025-03 midtraining Cooldown ablations of markdownified Wikipedia/... 8B checkpoint loss/evals numbers_in_text logs [src](https://github.com/marin-community/marin... No major improvement vs control.
20 marin-8b 2025-03 to 2025-04 midtraining Deep cooldown depth (1.7e-4 vs 3e-5 vs 1.7e-5)... 8B training loss drift, lm_head norm, SFT evals numbers_in_text logs [src](https://github.com/marin-community/marin... z-loss 1e-4 cured loss creep; adopted in Starl...
21 marin-32b 2025-06 pretraining Loss-spike mitigations at 32B: grad-norm clip ... 32B ~5e23 phase loss spikes numbers_in_text logs [src](https://github.com/marin-community/marin... Only QK-Norm (Qwen3 backbone) removed spikes.
22 marin-32b 2025-10 midtraining Data shuffle: LCG/affine permutation vs Feiste... 32B train-loss phase shift; Paloma losses numbers_in_text logs [src](https://github.com/marin-community/marin... Feistel removed phase shift and improved all P...
23 marin-ablations 2025-08 pretraining Batch-size scaling of Muon vs AdamW gap loss figure_only logs [src](https://github.com/marin-community/marin...
24 marin-ablations 2024-12 pretraining MuP vs heuristic hyperparameter scaling loss numbers_in_text logs [src](https://github.com/marin-community/marin... 'not worth it compared to our heuristic version'.
25 marin-ablations 2026-04 to 2026-05 post-training/RL SFT data scaling (exp3956): Marin 8B base fine... 8B, 32B (Marin bases) about 1e18 to 1e20 FLOPs of SFT at ~1e4 tokens... AIME24, AIME25, AMC23, HMMT, MATH500 accuracy ... none published logs [src](https://github.com/marin-community/marin... Per-run evalchemy results are public W&B runs ...

2.12 What the lab says about itself

project claim value unit source_url quote notes
0 marin-8b Experiment issues in marin-community/marin (la... 446 issues [src](https://github.com/search?q=repo%3Amarin... as of 2026-09-13
1 marin-8b Marin 8B 19-task average vs Llama 3.1 8B 66.6 vs 65.3 accuracy [src](https://github.com/marin-community/marin... beats Llama 3.1 8B on 14/19
2 marin-8b Compute source nearly all from TRC [src](https://github.com/marin-community/marin... "Nearly all of the compute used for Marin come... Google TPU Research Cloud; no dollar figures p...
3 marin-32b Discarded 32B cooldown (Bison) rerun as Mantis 1.074e12 tokens [src](https://github.com/marin-community/marin... GSM8K contamination via cached Dolmino test.js...
4 marin-moe MoE IsoFLOP law (v16 AdamH, 1e18 to 1e20) loss = 1.6 + 95.18 * C^-0.0941 [src](https://github.com/marin-community/marin... irreducible term pinned at 1.6
5 marin-moe MoE law validation at 1e21 FLOPs: predicted vs... 2.606 vs 2.599 loss [src](https://github.com/marin-community/marin...
6 delphi Delphi pre-registered forecast error at 1e23 F... 0.2 percent [src](https://openathena.ai/blog/delphi/) Delphi attempt 2 forecasts 1e23 to within 0.2% Confirmed in the blog text 2026-09-13: +0.2% a...
7 delphi Delphi IsoFLOP grid size 81 runs at budgets 3e18, 9e18, 1.8e19, 3e19, 9... runs [src](https://huggingface.co/datasets/marin-co... README: 'isoflop (81)' 88 checkpoints on the HF org incl. held-out 1e...
8 marin-535b-a23b 535B hero-shape ladder cost as share of the he... 1 percent [src](https://github.com/marin-community/marin... "only costs 1% of total compute" 4 rungs, 1.6B-A61M to 27.7B-A1.2B
9 marin-535b-a23b 535B-A23B hero run compute 2.7e24 FLOPs [src](https://github.com/marin-community/marin... Issue plan: 18.75T tokens, 11 x GB200 NVL72, ~...
10 marin-moe May 2026 1e23 MoE hero run vs preregistered loss within 1 percent [src](https://openathena.ai/blog/marin-535b-la... The 129B-parameter, 16B-active run landed with... Launch note, read 2026-09-13
11 lab-level Marin headcount growth under Open Athena 1 FTE -> 10 FTE [src](https://openathena.ai/blog/marin-535b-la... Since joining Open Athena, Marin has grown fro... Launch note, read 2026-09-13

2.13 Key numbers

project run_id run_type stage params_total tokens flops flops_method gpu_hours loss benchmark_name benchmark_value confidence source_url
0 marin-8b marin-8b-base-deeper-starling (exp600_tootsie,... final pretraining 8.03e+09 1.27e+13 6.10e+23 6*N*D with N=8.03e9 (Llama 3.1 8B dims: 32 lay... NaN 2.3460 MMLU 5-shot (LM Eval Harness) 67.6 derived [src](https://github.com/marin-community/marin...
1 marin-8b exp600 phase1 Kestrel (DCLM, WSD-S) final pretraining 8.03e+09 2.70e+12 1.30e+23 6*N*D NaN 2.5455 NaN derived [src](https://github.com/marin-community/marin...
2 marin-8b exp600 phase2 Ocelot (DCLM, WSD+EMA) final pretraining 8.03e+09 1.08e+12 5.20e+22 6*N*D on 2.7T->3.78T NaN 2.5998 NaN derived [src](https://github.com/marin-community/marin...
3 marin-8b exp600 phase3 Jellyfish (first cooldown, 'monu... midtrain midtraining 8.03e+09 1.00e+12 4.80e+22 6*N*D on 3.78T->4.78T NaN 2.6049 MMLU 5-shot 65.3 reported [src](https://github.com/marin-community/marin...
4 marin-8b exp600 dessert runs (math + FLAN patches off J... ablation midtraining 8.03e+09 NaN 2.6091 GSM8K; MATH; MMLU 5-shot NaN reported [src](https://github.com/marin-community/marin...
5 marin-8b exp600 phase4 Phoenix (rewarm, Nemotron-CC) final pretraining 8.03e+09 6.32e+12 3.00e+23 6*N*D on 4.78T->11.1T NaN 2.5046 NaN derived [src](https://github.com/marin-community/marin...
6 marin-8b exp898 Raccoon (deep cooldown 1.7e-4->1.7e-5 f... ablation midtraining 8.03e+09 1.00e+11 4.80e+21 6*N*D on ~100B tokens NaN 2.6115 NaN reported [src](https://github.com/marin-community/marin...
7 marin-8b exp916 Spoonbill (cooldown to 3e-5 + Tulu 0.3%... ablation midtraining 8.03e+09 NaN 2.5887 NaN reported [src](https://github.com/marin-community/marin...
8 marin-8b exp977 phase5 Starling (second cooldown, 'sens... midtrain midtraining 8.03e+09 1.34e+12 6.50e+22 6*N*D on 1.34T tokens (80,000 steps x 16Mi) NaN 2.3523 NaN reported [src](https://github.com/marin-community/marin...
9 marin-8b exp600 Deeper Starling dessert (final released... final midtraining 8.03e+09 3.00e+11 1.40e+22 6*N*D on 12.4T->12.7T NaN 2.3460 19-task avg 66.6 reported [src](https://github.com/marin-community/marin...
10 marin-8b marin-8b-instruct (deeper-starling-05-15 SFT) final post-training/RL 8.03e+09 5.30e+09 2.55e+20 6*N*D on 5Gi tokens (10,227 steps x 512Ki) NaN NaN 10-task instruct avg (IFEval, GSM8K-CoT, BBH, ... 43.8 reported [src](https://github.com/marin-community/marin...
11 marin-8b exp784/exp820 microannealing series (cooldown ... ablation midtraining 8.03e+09 NaN NaN NaN reported [src](https://github.com/marin-community/marin...
12 marin-13b exp860 Tootsie 13B aborted pretraining 1.30e+10 2.27e+12 1.77e+23 6ND derived (tokens from W&B) NaN 2.4983 NaN reported [src](https://github.com/marin-community/marin...
13 marin-24b exp861 Tootsie 24B aborted pretraining 2.40e+10 2.43e+12 3.50e+23 6ND derived (tokens from W&B) NaN 2.4480 NaN reported [src](https://github.com/marin-community/marin...
14 marin-70b exp750 Tootsie 70B aborted pretraining 7.00e+10 1.54e+12 6.47e+23 6ND derived (tokens from W&B) NaN 2.4247 NaN reported [src](https://github.com/marin-community/marin...
15 marin-32b marin-32b-base Mantis (aggregate final artifact) final pretraining 3.20e+10 6.44e+12 1.24e+24 6*N*D with N=32e9 (OLMo-2-32B dims) and D=6.437T NaN 2.1993 19-task avg 65.2 derived [src](https://github.com/marin-community/marin...
16 marin-32b exp1295_32b phase1 (Llama-3-style 32B, spiky) final pretraining 3.20e+10 2.68e+12 5.10e+23 6*N*D NaN 2.2924 NaN reported [src](https://github.com/marin-community/marin...
17 marin-32b exp1390_32b_necro (optimizer-state rebuild res... aborted pretraining 3.20e+10 1.00e+10 1.92e+21 6ND NaN 2.2905 NaN guess [src](https://github.com/marin-community/marin...
18 marin-32b exp1380_muon32b (Muon optimizer swap at 80k) aborted pretraining 3.20e+10 NaN 2.3064 NaN reported [src](https://github.com/marin-community/marin...
19 marin-32b exp1395_qwen3_32b phase3 (QK-Norm switch, warm... final pretraining 3.20e+10 2.68e+12 5.20e+23 6*N*D NaN 2.2649 NaN reported [src](https://github.com/marin-community/marin...
20 marin-32b exp1529_32b_bison_cooldown (first 32B cooldown) aborted midtraining 3.20e+10 1.07e+12 2.10e+23 6*N*D NaN 2.2213 19-task avg 63.0 reported [src](https://github.com/marin-community/marin...
21 marin-32b exp1529_32b_mantis_cooldown (released cooldown) midtrain midtraining 3.20e+10 1.07e+12 2.10e+23 6*N*D NaN 2.1993 19-task avg 65.2 reported [src](https://github.com/marin-community/marin...
22 delphi Delphi IsoFLOP sweep adamh_scaling_v6 (78 runs... ladder pretraining 8.63e+21 Sum of per-run budgets listed in reference_sca... NaN NaN NaN derived [src](https://github.com/marin-community/marin...
23 delphi delphi-2e19-550Mparams-5.8Btokens ladder pretraining 5.50e+08 5.80e+09 1.90e+19 6*N*D (nominal budget 2e19) NaN NaN NaN reported [src](https://huggingface.co/marin-community/d...
24 delphi Delphi optimal 1e21 (3 seeds: 0, 42, 62746) ladder pretraining 3.38e+09 4.63e+10 1.00e+21 reported budget NaN 2.7326 NaN reported [src](https://github.com/marin-community/marin...
25 delphi delphi-1e22-9.7Bparams-160Btokens (3 seeds) ladder pretraining 9.70e+09 1.60e+11 9.30e+21 6*N*D (nominal 1e22) NaN 2.5182 NaN reported [src](https://huggingface.co/marin-community/d...
26 delphi Delphi optimal 1e23 (25B params, 600B tokens) ladder pretraining 2.50e+10 6.28e+11 1.00e+23 6*N*D (nominal 1e23) NaN 2.3435 NaN reported [src](https://openathena.ai/blog/delphi/ ; htt...
27 marin-moe MoE v16 isoflop sweep (issue #4447) 1e18-1e20 ladder pretraining reported budgets NaN NaN NaN reported [src](https://github.com/marin-community/marin...
28 marin-moe MoE 1e21 test (14B-A2B, 75B tokens) #3800 ladder pretraining 1.40e+10 7.50e+10 9.00e+20 6*N_active*D (nominal 1e21) NaN NaN NaN reported [src](https://github.com/marin-community/marin...
29 marin-moe MoE 1e22 test (35B-A5B / '32B-A5B', 326B token... ladder pretraining 3.50e+10 3.26e+11 9.80e+21 6*N_active*D (nominal 1e22) NaN NaN NaN reported [src](https://github.com/marin-community/marin...
30 marin-moe May 2026 1e23 MoE 'hero run' (~120-131B total ... ladder pretraining 1.29e+11 1.02e+12 1.00e+23 reported budget (6*16e9*1.02e12=9.8e22 check) NaN NaN NaN derived [src](https://github.com/marin-community/marin...
31 marin-moe May Recipe MuonH compute-optimal baselines (4 ... ladder pretraining reported per-cell budgets NaN 3.5422 NaN reported [src](https://github.com/marin-community/marin...
32 marin-moe Agent MoE experiment digest (80 sub-issues of ... ablation pretraining NaN NaN NaN reported [src](https://github.com/marin-community/marin...
33 marin-moe MuonH LR scaling sweep #5951 (~150 runs) ablation pretraining NaN NaN NaN reported [src](https://github.com/marin-community/marin...
34 marin-snowball-67b-a2b Snowball 67B-A2B pretrain+midtrain on 10T toke... final pretraining 6.71e+10 1.05e+13 1.20e+23 6*N_active*D with N_active=2e9, D=1e13 NaN 2.2676 NaN derived [src](https://github.com/marin-community/marin...
35 marin-snowball-67b-a2b Snowball 2.7T-token checkpoint math RL + agent... ablation post-training/RL 6.70e+10 NaN NaN NaN reported [src](https://storage.googleapis.com/marin-pub...
36 marin-535b-a23b [Hero Run] 535B-A23B on 18.75T tokens (#8435) final pretraining 5.36e+11 1.88e+13 2.70e+24 reported (6*23e9*18.75e12=2.6e24 check) 1520000.0 2.2910 NaN reported [src](https://github.com/marin-community/marin...
37 marin-535b-a23b 535B hero-shape ladder rung 1: 1.6B-A61M, 48B ... ladder pretraining 1.60e+09 4.80e+10 1.80e+19 6*N_active*D NaN NaN NaN reported [src](https://x.com/percyliang/status/20909180...
38 marin-535b-a23b 535B hero-shape ladder rung 4: 27.7B-A1.2B, 92... ladder pretraining 2.77e+10 9.26e+11 6.70e+21 6*N_active*D NaN NaN NaN reported [src](https://x.com/percyliang/status/20909180...
39 marin-ablations exp143 DCLM 7B baseline replication ablation pretraining 7.00e+09 NaN NaN NaN reported [src](https://github.com/marin-community/marin...
40 marin-ablations exp524 tokenizer comparison (Llama3 vs Llama2 ... ablation pretraining NaN NaN NaN reported [src](https://github.com/marin-community/marin...
41 marin-ablations exp764 LR schedule comparison (7 runs, 0.5B, 1... ablation pretraining 5.00e+08 1.60e+11 4.80e+20 6*N*D per run NaN NaN NaN derived [src](https://github.com/marin-community/marin...
42 marin-ablations exp725 AdamW hyperparameter sweep (0.1B/10B, 0... ablation pretraining NaN NaN NaN reported [src](https://github.com/marin-community/marin...
43 marin-ablations exp935 z-loss vs none at 1.4B ablation pretraining 1.40e+09 NaN NaN NaN reported [src](https://github.com/marin-community/marin...
44 marin-ablations exp1183 OLMoE replication: 8x1B MoE vs 1B dense ablation pretraining NaN NaN NaN reported [src](https://github.com/marin-community/marin...
45 marin-ablations exp1290 Fantastic Pretraining Optimizers (10 o... ablation pretraining NaN NaN NaN reported [src](https://github.com/marin-community/marin...
46 marin-ablations exp636 HQ many epochs vs LQ few epochs ablation pretraining NaN NaN NaN reported [src](https://github.com/marin-community/marin...
47 marin-ablations exp246 HTML->text extraction comparison ablation pretraining NaN NaN NaN reported [src](https://github.com/marin-community/marin...
48 marin-ablations exp845/846/847 cooldown ablations for markdown... ablation midtraining 8.00e+09 NaN NaN NaN reported [src](https://github.com/marin-community/marin...
49 marin-ablations exp620 INT8 training in Levanter ablation pretraining NaN NaN NaN reported [src](https://github.com/marin-community/marin...
50 marin-ablations exp621 MuP for scaling laws ablation pretraining NaN NaN NaN reported [src](https://github.com/marin-community/marin...
51 marin-ablations exp1565 batch size widens Muon vs AdamW gap ablation pretraining NaN NaN NaN reported [src](https://github.com/marin-community/marin...
52 marin-ablations exp1572 Qwen3 (QK-Norm) speedruns incl. Muon ablation pretraining NaN NaN NaN reported [src](https://github.com/marin-community/marin...
53 marin-ablations exp633 Compel compression-ratio quality filter... ablation pretraining 1.40e+09 NaN NaN NaN reported [src](https://github.com/marin-community/marin...
54 marin-ablations exp3124 OLMoE-M quantile-balancing sweep (40B ... ablation pretraining 4.00e+10 NaN NaN NaN reported [src](https://github.com/marin-community/marin...
55 marin-8b tootsie-scaling-512 (Dec 2024 ladder, 16L, 210... ladder pretraining 2.49e+08 2.10e+11 3.13e+20 6ND derived NaN 3.2518 NaN derived [src](https://wandb.ai/marin-community/marin/r...
56 marin-8b tootsie-scaling-768 (Dec 2024 ladder, 16L, 210... ladder pretraining 4.61e+08 2.10e+11 5.80e+20 6ND derived NaN 3.0843 NaN derived [src](https://wandb.ai/marin-community/marin/r...
57 marin-8b tootsie-scaling-1024 (Dec 2024 ladder, 16L, 21... ladder pretraining 7.32e+08 2.10e+11 9.22e+20 6ND derived NaN 2.9819 NaN derived [src](https://wandb.ai/marin-community/marin/r...
58 marin-8b tootsie-scaling-1536 (Dec 2024 ladder, 16L, 21... ladder pretraining 1.45e+09 2.10e+11 1.83e+21 6ND derived NaN 2.8618 NaN derived [src](https://wandb.ai/marin-community/marin/r...
59 marin-8b tootsie-scaling-2048 (Dec 2024 ladder, 16L, 21... ladder pretraining 2.34e+09 2.10e+11 2.94e+21 6ND derived NaN 2.7973 NaN derived [src](https://wandb.ai/marin-community/marin/r...

2.14 Narrative

8B “Tootsie”. Llama-3.1-8B shape, 12.7T tokens, on two TPU v5e-256 slices and then a reserved v4-2048. Phases were named after the fact: Kestrel, Ocelot, Jellyfish (first cooldown), Phoenix (rewarm and data switch), Starling and Deeper Starling (second cooldown). Side runs that fed decisions: microanneals that settled the cooldown mix at 70/15/15, “dessert” patch runs, and two alternative cooldowns. Final 19-task average 66.6 against Llama 3.1 8B’s 65.3.

32B “Mantis”. Phase 1 (Llama-3-style, 80k steps) had persistent loss spikes that clipping softened but did not fix. Two rescues failed and were discarded. Phase 3 warm-started a Qwen3-32B backbone with QK-Norm from the 80k weights and the spikes disappeared. The first cooldown was discarded and rerun on the same tokens after the contamination and permutation bugs were found.

Scale-up by law (2026). Delphi, the MoE IsoFLOP law, the MuonH learning-rate law (R² 0.996 on a 150-run sweep), and the 535B ladder. Marin’s recipe document now codifies the sequence: tune at about 130M and 2.5B tokens, define scaling rules, run an IsoFLOP sweep, run a ladder, promote to a daily canary. Ideas for the MoE are screened at d512 to d1280 against the fitted law, with “effective speedup” computed by inverting the law and multiplying by measured throughput.

Not reconstructable from here. Delphi’s fitted exponent and the 535B ladder forecast (both in blocked pages or reports), the compute of the long tail of small ablations, any dollar figure (compute was donated by Google TRC and the Huang Foundation), and the reasons the 13B, 24B and 70B trials were abandoned (their W&B runs simply stop in April 2025 at 2.3T, 2.4T and 1.5T tokens).