Is AI progress accelerating? It depends on the shape of each benchmark’s curve

Epoch's Capabilities Index assumes every benchmark score follows one S-shaped curve of capability: sigmoid(slope × (capability − difficulty)). That is a choice. Below, seven different curves are fitted to each benchmark, from ECI's logistic to curves of any monotone shape, and compared on scores they never saw. Then each is fitted jointly with capability, to see how it changes the measured speed and acceleration of frontier progress. The curves agree on the rate up to a year ago (about 13.5 ECI points a year). The curves that predict newer models best measure faster recent growth than ECI, but how much faster is not pinned down by the data.

Is progress accelerating? It depends on the curve

Here each curve is fitted jointly with capability: curves and model positions alternate until they settle, with Claude 3.5 Sonnet = 130 and GPT-5 = 150 fixing the scale. Every option gives its own frontier (best capability to date, from GPT-4 on). The question is how fast it rose in the last 12 months () versus the 12 months before ().

Which benchmarks drive it?

Two views of where the logistic understates recent progress. Left: refit both curves without each benchmark in turn; leverage is how much the piecewise-linear curve's extra last-year growth over the logistic ( points a year with all benchmarks) shrinks when that benchmark is dropped. Right: on each benchmark, how much higher its own free curve places last year's models than the logistic does, relative to the year before (ECI points). Look at them on the main chart, piecewise-linear curve, implied-capability view.

Curve options compared

Change in held-out error (RMSE in score units) relative to ECI's logistic; lower is better. Bars are 90% intervals from resampling models. Random: 5-fold over all scores. Later models: curves fitted only on models released before , then used to predict half of each later model's scores ( scores), which tests extrapolation.

Every option on one benchmark

Joint fits: free curves and a second factor

Here capability is re-estimated together with the curves, comparing ECI's logistic with the piecewise-linear monotone curve, with one or two factors per model.

Each benchmark's curve, free shape vs sigmoid

Each panel shows one benchmark's scores against model capability θ from the one-factor nonparametric fit. The solid line is that benchmark's monotone curve, with a bootstrap 80% band. The dashed line is the best sigmoid on the same θ, which is the shape ECI assumes. Curves are drawn solid only where the benchmark has scores. Under each name, the held-out (5-fold) error of the sigmoid → the free curve. Click a panel for detail. CV-tuned curves use the penalties that predict held-out scores best, and they end up close to the sigmoid. Lightly penalised curves show where the scores alone would pull each curve. They overfit (held-out RMSE about 0.096) but are the better diagnostic for where a benchmark departs from the ECI shape.

All 60 curves against the sigmoid

Each benchmark's free curve is redrawn in its own sigmoid's units. x = slope × (θ − midpoint), the sigmoid's own log-odds. y = how far the free curve's log-odds sit above (+) or below (−) that sigmoid. ECI's assumption is the zero line for every benchmark. A curve that rises at the right is steeper at the top than a sigmoid, and one that dips at the left starts slower. Curves are drawn only where the benchmark has scores and the fit is between 1% and 99%. The curve switch above (CV-tuned / lightly penalised) applies here too. Click an area in the legend to hide it.

Held-out error

Root-mean-square error in score units (0–1), on scores the fit never saw. Random: 5-fold over cells, each model keeping at least 3 training scores. Forward: curves are fit only on models released before . Each later model is placed using half of its scores, and the other half are predicted, so this tests extrapolation into a capability range the curves haven't seen. Bars are 90% intervals from resampling models.

Two dimensions: models as points, benchmarks as directions

In the two-factor model, each model is a point and each benchmark a direction. A benchmark's score rises, along its own monotone curve, with a model's projection onto that direction. Axis 1 is on the ECI scale (Claude 3.5 Sonnet = 130, GPT-5 = 150). Axis 2 is in the same units and centred at 0. Colour is release date. Rays show the benchmark directions, and the strip on the right lists their angles.

Does the ranking change?

Each model's capability in the one-factor nonparametric fit (θ) against its published ECI. Points on the diagonal are placed the same by both, and the largest movers are labelled. With a free monotone curve only the ordering on this axis is identified. Its spacing comes from the ECI starting point and the two anchors (see method).

What one factor leaves over

Residuals (score − in-sample fit) for every scored cell. Benchmarks are rows, sorted by the capability at which their curve crosses 50%. Models are columns, sorted by θ. Orange = scored higher than predicted, blue = lower, blank = not scored. A block of one colour is structure a single factor can't explain. Switch to the two-factor fit to see how much of it a second axis absorbs.

The frontier, measured on each benchmark's own curve

The thick line is frontier θ (running best by release date, from GPT-4 on). Each thin line reads the same frontier models off one benchmark's free curve and converts back to capability as if that curve were a sigmoid: midpoint + log-odds ÷ slope. If ECI's sigmoid were exactly right, every thin line would lie on the thick one. Where they fan out, the measured rate of progress depends on which benchmark's scale you trust. Lines are drawn only where the frontier is inside a benchmark's scored range and the curve is between 2% and 98%, since inverting a saturated curve is meaningless. The curve switch in the first section applies here too.

Method