Tom Cunningham

Correlation Implies Causation

Tom Cunningham — Sat, 08 Aug 2026 07:00:00 GMT

Note

I wrote this around 2020, when working at Facebook, trying to reconcile practical decision-making with how economists talk about identification.

In a sense thes points are all well known. The idea is treated formally in Imbens (2003), Manski’s identification bounds; Oster (2019). I think Mostly Harmless Econometrics discusses the point informally. Still I found this a useful way of explaining the distinction.

Correlational evidence is always informative about causal effects.

The correlation is the sum of the causal-effect and the confounding-effect. To update your estimate of the causal effect you need to filter it through your prior means and uncertainties about each.

We use correlations all the time to help estimate causal effects.

The slide-decks that circulate in businesses and governments are full of scatter plots and conditional means, and that’s a good thing. It would be madness to make decisions without collecting that data.

Statisticians teach taboos and fetishes.

There’s a taboo against consuming raw correlations because they can make you sick. The taboo is there for a reason but the truth is that they are very nutritious when prepared correctly.

There’s a fetish for identification – finding circumstances where we can assume the confounding-effect is exactly zero – but that’s way too conservative. There are lots of cases where the confounding-effect is sufficiently contained that we can learn a lot about causation from correlations, without thinking it’s exactly zero. We do this every day but unfortunately statisticians are typically prudish and so rarely give useful advice on how to do it well.

Why the taboo?

The taboo is because sometimes people forget entirely about confounding: they get a regression coefficient and say “this is our best estimate of the causal effect”. Doing that is positively harmful, and if that’s the only thing you’re doing with the data you’re probably better off not looking at the data at all.

The right way of interpreting correlations goes something like the following: “the coefficient is above what I expected by 40%, and my uncertainty about the causal & confounding effects are roughly similar, and so I’ll revise upward my estimate of the causal effect by 20%.” If you’re relatively more certain about the causal effect, then you update relatively more on the confounding, and vice versa.

This sounds complicated, but we do it intuitively all the time.

Another reason for the fetish for zero-effects: it lets you keep things objective.

As soon as you allow for some non-zero confounding effect then your interpretation relies on more inputs – you have to put in your best-estimates of means and ranges for different variables.

This is hard in science: it typically means we have to allow subjective judgment to enter the inference process, and that can cause pain – through people being either insincere, or self-deceiving, or clumsy in reporting their judgments. So if you don’t trust other people, or don’t trust yourself, maybe you should stick with well-identified regressions.

Put more broadly: insisting on identification is approprirate in a low trust environment.

The taboo is holding back science.

There are a lot of scientific fields where it’s hard to find good identification, and it feels like those fields have been stuck for decades looking for their keys under lamp-posts: economic development, returns-to-education, nutrition. (Obviously this claim deserves a substantial argument to back it up, & I don’t have anything prepared, but this is how I feel).

Some best practices for interpreting causal effects.

show rich scatter plots to get a full picture of correlations;
benchmark against other correlations;
brainstorm differential diagnoses, with a causal diagram;
be explicit about ranges for both the causal and confounding effect you’re inferring from the correlational data.

Identification comes from assuming some causal effects are zero, but they rarely are.

We often talk about whether some causal effect “is identified” or “is not identified” as if it’s a binary thing. Realistically it’s a continuum. Identification happens when we identify a causal effect with a correlation. The simplest case is univariate regression: . When you assume unconfoundedness, i.e. , then you get identification: . You get identification from trimming the causal tree.

A lot of academic social science uses identification based on an assumption that some effect is probably pretty small relative to the effect of interest, such that we can ignore the bias (e.g. instrumental variables, diff-in-diff, matching, etc), and they use 0 as a stand-in for “small”. But by searching for places where we have clean identification, we’re passing over mountains of good evidence.

Doesn’t it depend on your assumptions about the true model?

People talk about different DAGs as if one of them was true and the other ones not (“does X affect Y, or does Y affect X?”). But we know that everything affects everything, and the idea that some causal arrows have coefficients of zero is just a convenient fiction to help us simplify a complex problem into a simpler one, and achieve identification.

Definitions of Recursive Self-Improvement

Tom Cunningham — Fri, 05 Jun 2026 07:00:00 GMT

This page surveys definitions related to recursive self-improvement.

Most of the content was put together by LLM agents, there are validation checks but it’s possible this contains errors. Any corrections or additions would be very welcome, send me an email!

Inclusion criteria (source of truth)

Coined (red diamond) — the publication is the earliest in this reference to use the term in its modern sense. Each term is coined exactly once.
Later use (open circle) — the publication defines the term, or uses it in a way that is load-bearing for its argument.
Concept anticipated (dashed grey square) — the publication anticipates the idea but predates, or does not use, the term.
Excluded (no marker) if the term appears only as an offhand or passing mention; a reference-list citation; an organisation or event name (e.g. “Singularity Institute”, “Singularity Summit”); an explicit disavowal (“this is not the same as X”); or — for “singularity” — the purely economic “growth-to-infinity” sense (which is tracked separately under the growth-explosion terms).

Coverage. For the shared-vocabulary terms — “intelligence explosion”, “singularity”, “seed AI”, “recursive self-improvement”, “artificial superintelligence (ASI)”, and “software intelligence explosion” — every qualifying use is marked. For all other terms, only coinages and notable (re)definitions are shown.

Scope. In scope: terms for the dynamics of AI improving AI — the feedback/explosion conditions, the milestones or capability thresholds said to trigger them, and the superintelligence or economic-explosion outcomes they imply. Out of scope: general technological-acceleration framings not specific to AI-improving-AI (e.g. Kurzweil’s “Law of Accelerating Returns” and The Singularity Is Near, deliberately omitted); broad AGI / AI-capability taxonomies and alignment vocabulary; and any work that merely mentions these ideas in passing. “Artificial superintelligence (ASI)” is tracked only where it bears on the self-improvement loop. Within scope, only reasonably prominent works are catalogued — those that coined a term, are widely cited, or are otherwise notable in the debate — rather than every paper that uses these terms.

Validation Checks

An Apple-Picking Model of AI R&D

Tom Cunningham — Tue, 07 Apr 2026 07:00:00 GMT

Thanks to Nate Rush, Thomas Kwa, Beth Barnes, Eli Lifland, Chris Ong, Basil Halperin, Tom Houlden, Parker Whitfill, Phil Trammell, & Andy Haupt for comments.

An apple-picking model of AI R&D.

Many people are talking about how AI is autonomously able to contribute to frontier R&D, yet it’s only picking low-hanging fruit: Andrej Karpathy, Terence Tao, Nathan Lambert, Ryan Greenblatt.

In this note we describe a simple apple-picking model of AI R&D, to help measure the contribution that autonomous agents are making.

The model takes the metaphor literally: an AI agent helping you to optimize an algorithm is like a robot helping you pick apples. It will take care of all the apples up to a certain height, and it may find apples you haven’t found yet, but there will still be apples out of its reach.

The model implies that agents can push the frontier forward, but the returns will sharply diminish. It also implies we can measure an agent’s ability with a human-equivalent time horizon, e.g. as of March 2026 agents seem to be able to find optimizations on frontier algorithms worth about 1 week of professional human labor, yet those effects are not additive.¹ You can’t get two weeks labor by running an agent twice.

The basic ideas can all be seen in the drawing below. Here the human and robot are both picking trees. The robot is cheaper to run, but it can only reach the low apples. In the illustration they have both picked 4 apples, yet left the tree in a very different state, such that the robot isn’t ready to replace the human yet:

We can write the core equation, where is the height of the robot, and are the expenditures on human and agent effort, and and are the rates at which they find unpicked apples:

The plots below show two implications of the model:

The agent will make quicker progress than the human at first, but eventually get overtaken (tortoise-hare dynamics)
The agent will be able to accumulate more apples (leave fewer unpicked) if they start from a stronger human starting point

The model sits between a few different literatures in economics, but as far as I can tell doesn’t already exist:

Models of tasks or knowledge hierarchies.² The apple-picking model differs in that tasks are performed cumulatively, instead of a fresh set of tasks each period.
Models of discovery.³ The apple-picking model models the discovery landscape in a different way (as far as I’m aware) which gives very simple clear results.
Models of knowledge accumulation.⁴ The apple-picking model differs in that the state of knowledge is not represented by a single scalar, e.g. the stock of ideas, instead the state is the combination of low-hanging and high-hanging ideas.

Implications for AI R&D

Start with the history of AI R&D.

We start with a graphical representation of AI progress: suppose there have been 10,000 person-years of R&D investment, and the loss metric (e.g. compute efficiency) has fallen by a factor of perhaps 10,000:

We wish to predict how the cost will fall in the future, given that the effort will now be assisted by AI.

Traditional models of AI-assisted R&D imply a permanent acceleration.

There are two standard models of AI’s effect on R&D used in the RSI literature: (1) it multiplies the effectiveness of human researchers; (2) it replaces human researchers with computer researchers (see more discussion below). Both models imply that when AI capabilities get better it’ll lead to an acceleration in the rate of progress, which persists:

Apple-picking implies one-time jumps in progress.

The apple-picking model implies a different pattern: for each model generation, agents will autonomously advance the frontier, but they will then quickly hit diminishing returns. When the marginal returns to human and agent expenditure are equalized, then we will return to investment in human optimization:

We give quotes below from a variety of domains with claims that (1) agents are autonomously advancing the frontier; (2) those advances have hit diminishing returns. We show below in the theory section that the optimal allocation of expenditure will be to first invest in agentic optimization, then switch back to humans.

As new more powerful agents are released, we should expect a sort-of punctuated equilibrium, as each successive branch of apples is picked. Terence Tao says, on Erdos problems: “Maybe the next time there’s a big advance in the models, they will try it again, and a few more will be breached.”

In reality aggregate progress is likely to appear smooth for a few reasons: (1) updated LLMs are released fairly frequently, and harnesses are constantly being updated; (2) each human discovery opens up room for agent discoveries (assumed away in the model); (3) LLMs are used to augment human activity, as well as autonomously do R&D.

We can calibrate agentic value by time horizon.

Because of the tortoise-hare behavior of agents we can calibrate an agent’s ability by the point at which a human and an agent, given equal expenditure, will make equal progress. This is, very loosely, the way agent time horizon is identified in Wijk et al. (2025) and Kwa et al. (2025).

An important implication is that an agent’s time horizon is sensitive to the starting point, in a way that differs from human effort. If we have a starting-point that has only been optimized by humans we expect agents can push it forward a lot. But if we have already applied some agent labor to the algorithm then further agentic labor will have much lower returns, i.e. time horizons will be much shorter. Concretely: one agent can be as good as one human (or human-week), but two agents are not as good as two humans.

Based on a very loose reading of the evidence we could say that agents (as of March 2026) are able to push forward the frontier on optimization problems by the equivalent of around a month of professional effort. However they then hit a wall and need either stronger models or better harnesses.

A low-hanging apple is whatever is easy for the agent.

The model defines a “low-hanging” optimization according to whether an agent can do it. There is a separate discussion of what specific types of tasks and ideas are within reach of agents (e.g. well-defined, hill-climbable, short-time-horizon).

The fact that there are stark differences between human and AI capabilities is often summarized with the phrase “jagged frontier”.

The effects of agentic R&D depend on the type of problem.

We can draw some very heuristic diagrams representing the likely potential for agentic optimization across different parts of the AI stack:

We have drawn pre-training as top-heavy, meaning it’s relatively hard for agents to optimize, arguably because the feedback cycle is long and so it is expensive to tell whether a change will work at scale so may require deep human thought.

In contrast other parts of the stack can be more easily hill-climbed, cheaply iterating on ideas to find improvements, e.g. innovations to elicitation (e.g. prompting, harness iteration) is a space that can be relatively cheaply explored, and so agents may do well at.⁵

Agents could kick-off recursive self-improvement.

Agentic R&D will be self-sustaining if, without human input, each successive generation is able to improve the next generation by a larger amount than the previous generation.

Concretely, if we measure agent ability with METR’s time horizon, then the condition for explosive growth is this: if each doubling of agent time horizon increases optimization ability sufficiently to cause more than 1 doubling of the next generation of models.

An implication is that the potential for RSI will depend on the shape of the tree, i.e. the stock of relatively simple optimizations that remain undiscovered, vs the necessity for finding deep optimizations.

Miscellaneous points.

Uplift. In practice AI will both (A) accelerate humans; (B) replace humans. The apple-picking model only focuses on the second. In the long-run we expect the second effect to dominate, but it’s not clear where we are now.
Directed search. We assumed that the probability of finding an apple is independent of other apples already found. Realistically people have an ability to direct their attention, it’s not clear whether the implications would significantly change.
Bottlenecks. The discussion has assumed that progress is purely a function of thinking. In practice there are other bottlenecks, most concretely for AI R&D the reliance on using scarce compute to run experiments. It’s arguable how important compute scarcity is, see for example Whitfill and Wu (2025).

Evidence on Autonomous AI R&D

Agents are improving on the state-of-the-art in well-studied optimization problems.

Google’s AlphaEvolve (Novikov et al. (2025), June 2025) is an evolutionary coding agent powered by Gemini models. They report a 23% speedup on a key Gemini training kernel and 0.7% of Google’s worldwide compute recovered through a better scheduling heuristic.
Andrej Karpathy’s autoresearch (March 2026) gets an agent to reducing validation loss from pretraining a GPT-2-small model given a fixed compute budget (one H100, ~5 minutes per training loop). Over ~2 days the agent tried ~700 changes and found ~20 additive edits, yielding an ~11% improvement in “Time-to-GPT-2”. Andrej Karpathy says “all the adjustments are ‘real’, I didn’t find them manually previously, and they stack up and actually improved nanochat.”
nanoGPT speedrun (Jordan and contributors (2026)) is a public competition to minimize training time for GPT-2 given a fixed target loss, which has brought training time from 45 min down to 1.4 min over 77 records since May 2024. Four recent improvements are tagged as contributed with the help of “AI systems”.
TTT-Discover (Yuksekgonul et al. (2026), January 2026), a test-time training method, optimized the TriMul GPU kernel used in AlphaFold, achieving >15% improvement over the best human implementations. The authors of the TriMul task, expert kernel engineers, called it “legit” and noted the strategy was “similar to the current best humans, but executed better,” with most human solutions falling behind on fusing more complex operators together.

Some optimizations are deeper than others.

The nanoGPT speedrun provides a useful case study as a public ledger of cumulative human effort on an AI R&D problem. Some deep contributions, from humans, are:

Muon (October 2024) came from original research in the nanogpt codebase on Newton-Schulz orthogonalization, cutting training time by 21% (31.4 → 24.9 min). It has since been adopted widely, including by Kimi K2 (1T MoE), GLM-4.5 (355B MoE), and Arcee Trinity (400B MoE), and is now part of PyTorch’s standard optimizer suite.
U-Net skip connections (November 2024) applied an encoder-decoder pattern from 2015 computer vision to transformer layers, yielding an 8% speedup (7.8 → 7.2 min). This became foundational and later records kept building on it.
Paired Head Attention (January 2026) is a novel attention mechanism that interleaves K/Q/V across head pairs to double the effective sequence length in attention.

These required theoretical insight, cross-domain transfer, or novel architectural ideas.

Many other records in the speedrun are imported from other projects, e.g. bundling known techniques like ReLU² and QK-norm (32% speedup), importing FlexAttention (30% speedup), (5% speedup), or upgrading PyTorch (8% speedup).

Others could be described as “shallow”, e.g. lowering the logit softcap from 30 to 15.

Agent optimizations are often described as shallow.

Several AI R&D and optimization benchmarks, such as MLGymBench (Nathani et al. (2025)), GSO (Shetty et al. (2025)), and SWE-fficiency (Ma et al. (2025)), report that agents achieve “surface-level speedups” but “fail to discover algorithmic innovations.” For instance, one of the largest speedups in AlgoTune (Press et al. (2025)) is a 142× on a graph communicability task, achieved by replacing pure Python with BLAS calls.
In nanoGPT speedrun the AI-contributed patches appear to be shallow relative to the optimizations above (e.g., Muon’s 21% speedup): replacing Python loops with faster library calls (hiverge.ai, ~1.2%) and combining two GPU operations to avoid writing intermediate results to memory (Locus, ~0.9%). These can be classified as typical optimization techniques that apply to many problems.
In autoresearch Karpathy says “It’s not novel, ground-breaking ‘research’ (yet), but all the adjustments are ‘real’”. The improvements that worked were things like adjusting AdamW constants, adding a scalar multiplier for QKnorm, and even making random seed changes. Overall he notes that the agents feel “cagey” on open-ended ideas.
Terence Tao has described the contribution of AI to mathematical discovery:

“What AI has been very good at is systematically exploring this long tail and knocking off the easiest of the problems.” (ref):

“Fifty-odd problems have been solved with AI assistance, which is great, but there’s like six hundred to go. People are still chipping away at one or two of these right now.” (ref)

Discussion

This is distinct from other models of recursive self-improvement.

We can put many existing models of AI R&D in two buckets:⁶

Apples can make the human pick faster. Here AI R&D speeds up humans, typically by automating some of the tasks that are necessary for doing R&D (Aghion, Jones, and Jones (2019), B. F. Jones (2025), Kwa (2026)).
Apples can be used to make more human-sized robots. Here AI R&D is already able to replicate human-level R&D, but the bottleneck is the cost of compute, and further AI progress lowers that cost, thereby expanding the AI R&D workforce (Davidson (2021), Davidson and Houlden (2025), Ho and Whitfill (2025), Eth and Davidson (2025), Davidson et al. (2026)).

Broadly these could be thought of as models of “early” and “late” RSI. The early models assume humans are still required, the late models assumes that AI has already reached parity with humans, and then it is only a matter of further scaling up, and so bottlenecks on cost become binding (though it’s worth noting that early-RSI models can still predict explosive growth).

There is some awkwardness in fitting both these classes of model to our current situation, because AI seems to already be autonomously contributing to AI research, yet not replacing humans, i.e. autonomous agents are not perfect substitutes for humans.

The model in this post is distinct from both: we assume there are plenty of robots, but the robots are limited in ability, and successive generations can reach higher. Thus agents cause neither a multiplicative boost to human productivity, nor a perfect substitute for human labor.

A critical distinction is how we represent the state. Most existing models summarize the level of productivity (or stock of knowledge) with a single number, meaning there’s no distinction between a shallow and deep contribution to the state of knowledge, all that matters is the total stock of ideas (or the efficiency of the algorithm). The apple-picking model keeps track of the state with two numbers: the share of apples remaining above (the robot’s height), and the share below . For the RSI version of the model we instead track the robot’s height () and the share of apples remaining above that threshold.

Apple-picking is a simplification of a landscape-navigation problem.

We think of the apple-picking model as a simplification of a more general landscape-navigation model, where you are trying to find a minimum over a bumpy landscape. You can represent an optimization problem as , where you’re trying to choose an to minimize , given some unknown . An immediate observation from landscape-navigation problem is that the current elevation is not a sufficient statistic for your state, unless you have a degenerate landscape, like the random landscapes in Kortum (1997).

A reasonable conjecture seems to be that AI agents are good at local optimization, e.g. climbing small hills, worse at global optimization, i.e. finding distant hills.

Terence Tao has a similar landscape-navigation metaphor

“These AI tools, they’re like jumping machines that can jump two meters in the air, higher than any human. Sometimes they jump in the wrong direction, and sometimes they crash, but sometimes they can reach the tops of the lowest walls that we couldn’t reach before. We’ve just set them loose in this mountain range, hopping around. There was this exciting period where they could actually find all the low ones and reach them. Maybe the next time there’s a big advance in the models, they will try it again, and a few more will be breached. But it’s a different style of doing mathematics. Normally we would hill climb, make little markers, and try to identify partial things. These tools either succeed or they fail. They’ve been really bad at creating partial progress or identifying intermediate stages that you should focus on first. Going back to this previous discussion, we don’t have a way of evaluating partial progress the same way we can evaluate a one-shot success or failure of solving a problem.”*

The apple-picking model can be thought of as navigation of a multi-dimensional landscape, where the payoff is separable in subspaces, where each subspace represents an apple.

Static Model

Setup.

Apples are spread on [0,1] with density function .

A human can find apples over , but an agent can only find apples over , with (at least for now).

Humans find unpicked apples at rate , agents find apples at rate , and we use and to represent the expenditure on humans and agent searching.

We can then derive apples remaining:

We can illustrate the state of the apple tree:

Implication: agents asymptote to a higher level of remaining apples than humans.

Here we illustrate agent-only and human-only search curves: the agent curve falls more quickly (), but asymptotes to a higher level of remaining apples ().

We see roughly this shape when comparing human and AI scaling on continuously-scored AI R&D tasks, e.g. in RE-Bench (Wijk et al. (2025)):

RE-Bench scaling

Implication: agents can improve on human SoTA, but only by a limited amount.

The plot below shows the returns on human and agent performance. The y-axis shows human-only apples remaining, which you can interpret as the remaining gaps after the effort of a single human or the cumulative effort of humanity. At each vertical point, moving to the right (adding agent optimizations) lowers the remaining apples, but only by a limited amount.

Implication: first spend on agents, then on humans.

We now overlay the optimal expenditure as budget increases: the blue line shows optimal expenditure, which is first to spend on agents (which are cheaper), but then on humans. Equating the marginal returns and , the ratio depends only on , giving a fixed threshold :

where , provided (agents are initially preferred).

Implication: as the budget rises, spend first goes to agents, then to humans.

The same optimal path can be plotted against the budget directly. On the left, expenditure goes entirely to the agent until the threshold , after which all additional budget goes to humans.

On the right we plot each actor’s counterfactual value: how many apples would be picked if the other actor were dropped, holding own expenditure at its optimal level. The agent’s value is the low apples it clears on its own, ; the human’s value is everything it would pick alone, . Because the two are substitutes over the low apples, these values overlap: once both are active their sum exceeds the total apples actually picked, so counterfactual value is not additive.

Implication: the agent asymptote depends on the starting point.

The plot below shows a variety of agent trajectories, each starting after a different amount of human work.

You could interpret this as starting an agent at different points in the history of optimizing some algorithm, e.g. nanoGPT.

The model implies that if you start an agent from the original unoptimized version of an algorithm it will quickly reduce the remaining apples, but asymptote to a value well above the human state-of-the-art.

If you start an agent after some human optimization has been performed the agent will contribute less at first (because fewer apples remain), but it will be able to achieve a lower asymptote.

Implication: agents lower the level of loss, but the rate of human returns stays the same.

The same static model also describes a sequence in time: humans optimize first, an agent is then released and clears the low apples it can reach, and humans then resume. In each period we apply the model expression , holding the idle actor’s expenditure fixed.

We run the agent only until its marginal return falls to the human’s. The agent’s marginal return is and the human’s is ; setting them equal gives the threshold derived above. We stop the agent there and switch back to humans, so the loss curve has no kink at the switch: its slope is continuous because the two marginal returns are equal at that point. The dotted line shows the counterfactual of spending only on humans (no agent); the agent pulls loss below it, and the gap persists once humans resume.

Plotting makes the human rate of return visible. The human marginal return is , a constant fraction of what remains, so regardless of how much has already been picked: human effort gives a constant proportional return. In the plot the two human segments are therefore parallel straight lines of slope — the agent shifts the line down but does not change its slope — while the agent segment is a concave drop whose return declines as it exhausts the low apples it can reach.

Dynamic Model (Recursive Self-Improvement)

This is a very simple model.

We can close the model above, and get a dynamic model of recursive self-improvement. The model below is just a proof of concept, the main conclusion is just that you’ll get recursively-self-improving robots if the low-hanging apples are sufficiently dense. Nevertheless I found it useful for my own thought, & I would love to spend more time filling this out to help with making some quantitative predictions.

Let the robot’s height depend on apples harvested.

The previous model applied to agents working on an arbitrary optimization problem. Now we focus on agents working on AI R&D, which in turn increases agent ability.

We make two changes:

We now track progress in picking apples over periods. Each period represents a generation of AI models, and we assume that agents pick all the apples available to them in each period (apples below ). This makes things easier to model because the state of the tree can be summarized with just two variables (robot height and human time), instead of the entire profile of the tree. It also seems like a reasonable assumption: AI research labs will keep spending money on agent-optimizing their algorithms until there are low returns to additional use.
We assume that the robot’s height in period is a function of cumulative apples harvested by period , i.e. the robot is eating the apples and getting taller.

The figure below shows the basic model.

State variables and dynamics

Assumptions.

Apples are distributed with cumulative distribution function , agent reach in period is , and human reach is always 1.

Apples harvested.

Each period, agents pick all the apples below , and humans pick a fraction of the apples that remain between and 1, thus we can write:

Thus we can write:

Self-improvement.

Finally to close the model we let next-period robot height () depend on cumulative apples harvested (). We assume height is linear in apples, after surpassing a minimum number of apples ():

These two equations above give us a recursive system.

Observations

Human-only labor.

With human labor only then apples will grow with diminishing returns:

This will characterize progress prior to the point at which we can build the first useful robot, , after which progress begins to accelerate.

Recursive self-improvement.

With robot labor only (holding fixed), then progress will be self-sustaining if and only if apples are sufficiently dense in the neighborhood of that robot height, i.e. if .

Embedding Apple-Picking into Jones-model R&D

We can embed the apple-picking model into the workhorse R&D function from C. I. Jones (1995) as follows, with the following implications:

Human effort has constant elasticity (from a human-only starting point).
Agent effort has declining elasticity.

We start with this function:

This can be rewritten in a cumulative form, where represents the total research (adjusted for congestion) up to time , then (assuming is small):

We can compare this to our apple-picking function, with only human effort :

These can be reconciled if we assume ideas is a nonlinear function of apples-picked ():

We can then consider progress in ideas with both humans and robots picking apples, :

References

Acemoglu, Daron, and Pascual Restrepo. 2022. “Tasks, Automation, and the Rise in US Wage Inequality.” Econometrica 90 (5): 1973–2016. https://doi.org/10.3386/w28920.

Aghion, Philippe, Benjamin F. Jones, and Charles I. Jones. 2019. “Artificial Intelligence and Economic Growth.” In The Economics of Artificial Intelligence: An Agenda, edited by Ajay Agrawal, Joshua Gans, and Avi Goldfarb, 237–90. Chicago: University of Chicago Press. https://www.degruyterbrill.com/document/doi/10.7208/9780226613475-011/html?lang=en.

Agrawal, Ajay, John McHale, and Alexander Oettl. 2019. “Finding Needles in Haystacks: Artificial Intelligence and Recombinant Growth.” In The Economics of Artificial Intelligence: An Agenda, edited by Ajay Agrawal, Joshua Gans, and Avi Goldfarb, 149–74. Chicago, IL: University of Chicago Press. https://www.degruyterbrill.com/document/doi/10.7208/9780226613475-007/html?lang=en.

Carnehl, Christoph, and Johannes Schneider. 2025. “A Quest for Knowledge.” Econometrica 93 (2): 623–59. https://doi.org/10.3982/ECTA22144.

Davidson, Tom. 2021. “Could Advanced AI Drive Explosive Economic Growth.” Open Philanthropy 25. https://www.openphilanthropy.org/research/could-advanced-ai-drive-explosive-economic-growth.

Davidson, Tom, Basil Halperin, Thomas Houlden, and Anton Korinek. 2026. “When Does Automating AI Research Produce Explosive Growth? Feedback Loops in Innovation Networks.” https://basilhalperin.com/papers/singularities.pdf.

Davidson, Tom, and Tom Houlden. 2025. “How Quick and Big Would a Software Intelligence Explosion Be?” https://www.forethought.org/research/how-quick-and-big-would-a-software-intelligence-explosion-be.

Eth, Daniel, and Tom Davidson. 2025. “Will AI r&d Automation Cause a Software Intelligence Explosion?” https://www.forethought.org/research/will-ai-r-and-d-automation-cause-a-software-intelligence-explosion.

Garicano, Luis, and Esteban Rossi-Hansberg. 2006. “Organization and Inequality in a Knowledge Economy.” The Quarterly Journal of Economics 121 (4): 1383–1435. https://researchonline.lse.ac.uk/25550/.

Ho, Anson, and Parker Whitfill. 2025. “The Software Intelligence Explosion Debate Needs Experiments.” https://epoch.ai/gradient-updates/the-software-intelligence-explosion-debate-needs-experiments.

Ide, Enrique, and Eduard Talamas. 2024. “Artificial Intelligence in the Knowledge Economy.” https://doi.org/10.1086/737233.

Jones, Benjamin F. 2025. “Artificial Intelligence in Research and Development.” NBER Working Paper 34312. National Bureau of Economic Research. https://doi.org/10.3386/w34312.

Jones, Charles I. 1995. “R&d-Based Models of Economic Growth.” Journal of Political Economy 103 (4): 759–84. https://doi.org/https://doi.org/10.1086/262002.

Jordan, Keller, and contributors. 2026. “Modded-Nanogpt.” https://github.com/KellerJordan/modded-nanogpt.

Kokotajlo, Daniel, Eli Lifland, Brendan Halstead, and Alex Kastner. 2025. “AI Futures Model: Dec 2025 Update.” https://blog.ai-futures.org/p/ai-futures-model-dec-2025-update.

Kortum, Samuel. 1997. “Research, Patenting, and Technological Change.” Econometrica 65 (6): 1389–419. https://doi.org/10.2307/2171741.

Kwa, Thomas. 2026. “Research Note: A Simpler AI Timelines Model Predicts 99% AI r&d Automation in ~2032.” https://www.lesswrong.com/posts/uy6B5rEPvcwi55cBK/research-note-a-simpler-ai-timelines-model-predicts-99-ai-r.

Kwa, Thomas, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, et al. 2025. “Measuring AI Ability to Complete Long Tasks.” arXiv Preprint arXiv:2503.14499. https://doi.org/10.48550/arXiv.2503.14499.

Ma, Jeffrey Jian, Milad Hashemi, Amir Yazdanbakhsh, et al. 2025. “SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?” https://arxiv.org/pdf/2511.06090.pdf.

Nathani, Deepak, Lovish Madaan, Nicholas Roberts, et al. 2025. “MLGym: A New Framework and Benchmark for Advancing AI Research Agents.” https://arxiv.org/pdf/2502.14499.pdf.

Novikov, Alexander, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, et al. 2025. “AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery.” arXiv Preprint arXiv:2506.13131. https://doi.org/10.48550/arXiv.2506.13131.

Press, Ori, Brandon Amos, Haoyu Zhao, et al. 2025. “AlgoTune: Can Language Models Speed up General-Purpose Numerical Programs?” https://arxiv.org/pdf/2507.15887.pdf.

Shetty, Manish, Naman Jain, Jinjian Liu, Vijay Kethanaboyina, Koushik Sen, and Ion Stoica. 2025. “GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents.” https://arxiv.org/pdf/2505.23671.pdf.

Whitfill, Parker, and Cheryl Wu. 2025. “Will Compute Bottlenecks Prevent an Intelligence Explosion?” https://arxiv.org/abs/2507.23181.

Wijk, Hjalmar, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, et al. 2025. “RE-Bench: Evaluating Frontier AI r&d Capabilities of Language Model Agents Against Human Experts.” https://arxiv.org/abs/2411.15114.

Yuksekgonul, Mert, Daniel Koceja, Xinhao Li, Federico Bianchi, Jed McCaleb, Xiaolong Wang, Jan Kautz, et al. 2026. “Learning to Discover at Test Time.” arXiv Preprint arXiv:2601.16175. https://test-time-training.github.io/discover.pdf.

Footnotes

E.g. Ryan Greenblatt says “I tentatively believe the AI made somewhere between a few days and a bit over a week of progress on this task relative to a strong human professional.”↩︎
E.g. Acemoglu and Restrepo (2022), Garicano and Rossi-Hansberg (2006), Ide and Talamas (2024)↩︎
Carnehl and Schneider (2025), Agrawal, McHale, and Oettl (2019)↩︎
C. I. Jones (1995), Aghion, Jones, and Jones (2019), B. F. Jones (2025)↩︎
Note that we should expect the returns to human labor to be roughly equalized across each of these trees, so the thickness of the trees isn’t important for agent:human contribution, but the relative density of low and high apples.↩︎
An exception to this classification is Kokotajlo et al. (2025), which models “research taste” in both humans and AIs. However that model also represents the state with a scalar (“software efficiency”), so I do not believe it could reproduce some of the behaviors described upfront.↩︎

When You Overtake More Runners than You’re Overtaken by

Tom Cunningham — Sat, 07 Mar 2026 08:00:00 GMT

Suppose you overtake 10 times as many runners as overtake you. What can you say about your speed relative to the other runners?

For concreteness, suppose you’re looping around the Viveros Coyoacan, everyone is running in the same direction forever, and each person started at a random point.

Under reasonable assumptions your relative frequency of passing runners will exaggerate your position in the distribution of running-speeds.

We can write the general relationship as follows:

The ratio of slower/faster will not generally be equal to the ratio of passing/passed. The reason is that your propensity to encounter another runner depends on the difference in speeds: you will rarely pass or be passed by those who run similar speeds to you.

If the distribution of speeds is symmetric and log-concave, your frequency of passing runners will exaggerate your position in the distribution of speeds. E.g. if you pass 10x as many people as pass you then there are fewer than 10x as many people who are slower than you relative to people who are faster than you. This covers normal, logistic, & uniform distributions.

Some additional implications:

If you’re passing the same number of people as pass you, then you must be running exactly the mean speed.
If the distribution of running speeds is sufficiently fat-tailed (e.g. a -distribution with ), then you can get the opposite implication.
(due to Matt Parry) If mean and variance exist, then your pass/passed ratio also gives an upper bound on how many standard deviations above the mean you can be: where is your ratio of pass/passed.

(original FB post)

Derivation

This last expression is the most important. It implies you can figure out your rank among runners if you know the relative distance in speeds among those who are faster and those who are slower. If the speed differences are identical then the two ratios are the same. For the Normal distribution, the conditional expectations are such that the slower runners are closer than the faster runners, implying the exaggeration property.

Derivation of the final condition:

One sufficient condition for exaggeration: let

Then

so exaggeration holds if stochastically dominates . Their survival functions are

Now suppose the density is symmetric about and log-concave. Then is log-concave, so for each fixed the function

is increasing in . By symmetry,

and if then , so

for all . Therefore stochastically dominates , so

and exaggeration follows.

Bound in standard deviations (assuming mean and variance exist):

Uniform Distribution of Speeds

With a uniform distribution you can easily see that the number of passing events is proportional to the area of the triangle, and so proportional to the square of the number of runners below/above your speed.

Normal Distribution of Speeds

Note on derivatives:

Simulations

Each cell in the table below shows first the ratio pass/passed, then the ratio slower/faster, for a given speed . We can see that exaggeration holds in the Gaussian and examples, but not in the example.

Note that with is weird because it doesn’t have an expectation, i.e. never goes to zero, so it is harder to think about.

speed ()	N(0,1)	t(df=1)	t(df=2)
-1.0	0.077 / 0.21	0.53 / 0.34	0.26 / 0.28
0.0	1 / 1.1	1 / 1.1	1 / 1.1
0.5	3.5 / 2.4	1.4 / 1.9	2 / 2.1
1.0	13.0 / 5.8	1.9 / 3.1	3.8 / 4
2.0	238 / 48.7	3.2 / 5.8	10.3 / 10.3
10.0	1.46e+25 / Inf	23.3 / 27.8	249 / 202

References

Mathoverflow (2010) “You pass X people and Y people pass you: how relatively fast are you?”

There are a number of sketches of answers, but no explicit solutions. A few answers wrongly conclude that the ratio of overtakes to being overtaken is equal to the ratio of slower to faster runners.

Clevenson, Schilling, Watkins and Watkins (2001) “The Average Speed on the Highway”

They show that if the number of cars you pass is equal to the number of cars that pass you then you must be driving the average speed (not necessarily the median speed).

Schilling (2006) “Do You Know Your Relative Driving Speed?”.

This seems to have most of the same results as the note here, though I don’t think they give a sufficient condition for exaggeration (symmetry + log concavity).

Knowledge-Creating LLMs

Tom Cunningham — Fri, 06 Feb 2026 08:00:00 GMT

Thanks to Zoë Hitzig & Parker Whitfill, among others, for helpful comments.

It’s useful to make a distinction between two types of LLMs:

Knowledge-sharing LLMs. Traditionally LLMs have been trained with human judgment as the ground truth, as a consequence they rarely exhibit superhuman performance. Their economic value mainly comes from sharing existing knowledge, and the natural business model is to sell access broadly.

Knowledge-creating LLMs. Recently LLMs have been trained against the real world, as a consequence they can extend the limits of human knowledge. The demand for new knowledge is different from the demand for old knowledge, and there’s reason to expect LLM-creators to sell access exclusively.

Obviously it’s a continuum but we’re clearly setting out on the trajectory from the first to the second, and I haven’t seen much discussion of the implications. I expect that the 2026 impacts of AI will be dominated by these considerations.

Below I give a longer discussion of this distinction, implications for IP, and a galaxy-brain theory that there are only a few dozen deep problems in the world. I sketch some considerations for a more formal economic model.

Knowledge-Sharing vs Knowledge-Creating LLMs

Knowledge-sharing LLMs.

I find it is useful to think of LLMs as lowering the price of sharing existing human knowledge (see some of my previous writing on this: AI & Imitation, a pocket model of AI).

Traditionally LLMs have been trained with human judgment as the ground truth, using labels from paid raters or from customers. As a consequence they can answer questions and solve problems up to the limits of human expertise but rarely beyond (with some exceptions, see the literature on LLM Transcendence, Abreu et al. (2025)).

If we model the economic effects of LLMs as coming from sharing existing knowledge, recorded in the training data, this has a number of implications that seem to fit the data.¹

LLM use will be higher among those junior in their careers, facing problems that are new to them.
LLMs will be disproportionately used by people outside their area of expertise, e.g. lawyers will use them relatively more for medical questions, doctors will use them relatively more for legal questions.
LLMs will be disproportionately used in well-documented domains, e.g. relatively more for popular programming languages than for proprietary programming languages.
LLMs will decrease knowledge rents – the premium earned by people and firms whose value comes from knowledge.
LLMs will increase home production – LLMs let you solve problems yourself (see our ChatGPT paper, Chatterji et al. (2025)), insofar as this substitutes for market-provided knowledge this can decrease measured GDP.
LLMs decrease the returns to innovation and news-gathering, because they increase the speed of knowledge diffusion and thus diminish the rents that can be earned from new knowledge.
LLM use has high fixed costs (collecting the knowledge) and low marginal costs (sharing the knowledge). The returns to tokens on an individual problem rapidly diminish when you hit the frontier of existing knowledge. One ChatGPT query can tell me what an expert would think about my problem, additional ChatGPT queries have much less use.

Knowledge-creating LLMs.

Over the past 18 months it has become much more popular to train LLMs directly against a source of ground truth, e.g. Reinforcement Learning against Verifiable Rewards (RLVR). Accompanying this there has been a steadily increasing stream of announcements of new discoveries by LLMs.

New LLM-based discovery techniques (e.g. AlphaEvolve (Novikov et al. (2025)), TTT-Discover (Yuksekgonul et al. (2026))) are distinct from prior AI discovery applications (e.g. AlphaFold, AlphaTensor) in that they are general methods, they can relatively quickly be adapted to any arbitrary optimization problem.

Some potential applications for knowledge-generating inference: (1) optimize every part of the AI R&D and serving stack; (2) do drug discovery; (3) discover new algorithms which can be used in new software (e.g. new codecs, new scheduling algorithms); (4) build better trading algorithms; (5) if you have a sufficiently high-quality verifier to human preferences, then build very high-quality cultural products, e.g. movies.

Knowledge-creating LLMs will differ from knowledge-sharing LLMs in a number of ways:

Knowledge-creating LLMs will have qualitatively different benchmarks: instead of seeing if they can answer questions which we already know the answer to (most existing benchmarks), we want them to answer new questions, e.g. solve an unsolved mathematical problem (FrontierMath Open Problems) or set a new record on an optimization problem (e.g. GSO-bench, Shetty et al. (2025)). We can use these new frontier benchmarks as indices for capability, but they are more challenging to interpret because the frontier is always moving.
Knowledge-creating LLMs have high returns to compute on individual problems, unlike knowledge-sharing LLMs for which returns asymptote quickly. It can be worth spending billions of tokens to solve a single problem if the solution is generally applicable.

The demand for new knowledge is much less elastic than the demand for existing knowledge because there are high returns to exclusivity of new knowledge. Thus LLM-providers are likely to either (1) use the knowledge themselves; or (2) license the new knowledge to partners, rather than expose the knowledge-generation technology through a general-purpose API. Sarah Friar, OpenAI’s CFO, said in January 2026:

“As intelligence moves into scientific research, drug discovery, energy systems, and financial modeling, new economic models will emerge. Licensing, IP-based agreements, and outcome-based pricing will share in the value created.”

Will knowledge-creation be bottlenecked on data?

A common claim is that AI knowledge-creation will be bottlenecked on the ability to run new experiments. E.g. an automated biologist still needs a lab (Amodei (2024)), and an automated AI researcher will still need a lot of GPUs to do experiments. Whether this is true depends on the shape of the optimization landscape. If the world is intrinsically high-dimensional, then there is no substitute for collecting data. But if there exists a low-dimensional structure then there are high returns to just thinking harder (more discussion in an earlier post). We have found many domains which appeared to be high-dimensional, but turned out to be intrinsically low-dimensional.

A Visual Illustration

Here we draw the cost for a set of 3 humans across a range of tasks, assuming each has a specialty area where they have the lowest labor-cost. The knowledge-sharing LLM aggregates knowledge, and so is the lower bound across all three agents.

We can then illustrate a knowledge-creating LLM as pushing below the human frontier at some set of tasks:

There are Only a Dozen Deep Problems

If you squint, a billion problems resolve into just a dozen common problems.

In many domains we can reduce the set of problems down to a much smaller set of equivalence-classes or canonical problems. We can then consider problem-solving as having two parts: (1) map to a canonical problem; (2) make progress on that canonical problem.

Consider three types of problems which LLMs are often asked to solve:

(1) Constraint-satisfaction problems.

Optimization theory textbooks will often have two types of results (1) proofs on the optimality of certain algorithms; (2) proofs that one type of problem is logically equivalent to another type of problem (e.g. a very large class of problems are equivalent to 3SAT, or can be reduced to 3SAT in polynomial time). Some algorithms are known to be optimal but many others are being continually improved.

(2) Factual problems.

We can reduce a factual question into (1) find the documented facts that are relevant to this question; (2) infer the answer from those facts. Once you have collected the existing documented facts you hit a ceiling, which can only be advanced by collecting more facts.

(3) Statistical inference problems.

For many classes of supervised learning problems there exists an existing “best practice”, e.g. a recent article says “Over hundreds of Kaggle competitions, we’ve refined a playbook that consistently lands us near the top of the leaderboard”.

This is a difficulty for LLM benchmarking.

The mapping between idiosyncratic and canonical problems is a difficulty for LLM benchmarking. If each problem can be mapped to a canonical problem, and there exists a best-known-algorithm for each of those canonical problems, then it’s difficult to test the model’s intelligence. A reasonably smart LLM will know how to map a new problem into a canonical problem, and will know the textbook best-practice for that canonical problem (XGboost, ARIMA, gaussian process, branch-and-cut, PPO, etc.). Thus we either have to devise problems sufficiently weird that it’s difficult to map them to textbook problems, or instead ask LLMs to advance the knowledge frontier on one of the existing canonical problems.

The remaining good domains for LLM benchmarking are tasks that are an emulsion of data and logic. They’re neither pure statistical inference nor pure deductive reasoning, but require both. Complex video games with a rich state-space are a good example (VideoGameBench appears to be far from saturated), also rich simulated real-world environments like VendingBench.

Labs will spend a lot on fixed inference, a little on variable inference.

If this perspective is accurate then it has deep implications for the economics of AI: the marginal cost of solving an idiosyncratic problem is small (you just need to map it to one of the canonical problems, and apply that solution), but there’s very high value in making progress on the canonical problems. So we would expect AI labs to be spending huge amounts of compute on advancing the SoTA on the few deep problems of the world, and providing a service that solves idiosyncratic problems very cheaply.

There will be a land-grab in intellectual property.

If we maintain the same intellectual property law then this implies there will be a land-grab: firms will rush to be the first to discover new technologies which they can patent. But it seems plausible that the exclusivity will be inefficient, i.e. it wasn’t necessary to motivate the research, the new technology would’ve been discovered anyway.

Appendix

Recent examples of LLM optimization ability:

Recent examples of

Novikov et al. (2025) (Alpha-Evolve):

“AlphaEvolve orchestrates an autonomous pipeline of LLMs, whose task is to improve an algorithm by making direct changes to the code. … When applied to optimizing critical components of large-scale computational stacks at Google, AlphaEvolve developed a more efficient scheduling algorithm for data centers, found a functionally equivalent simplification in the circuit design of hardware accelerators, and accelerated the training of the LLM underpinning AlphaEvolve itself. Furthermore, AlphaEvolve discovered novel, provably correct algorithms that surpass state-of-the-art solutions on a spectrum of problems in mathematics and computer science, significantly expanding the scope of prior automated discovery methods (Romera-Paredes et al., 2023).”

Yuksekgonul et al. (2026) (TTT-Discover):

“We report results for every problem we attempted, across mathematics, GPU kernel engineering, algorithm design, and biology. TTT-Discover sets the new state of the art in almost all of them: (i) Erdős’ minimum overlap problem and an autocorrelation inequality; (ii) a GPUMode kernel competition (up to 2×faster than prior art); (iii) past AtCoder algorithm competitions; and (iv) denoising problem in single-cell analysis. Our solutions are reviewed by experts or the organizers.”

The Opus 4.6 system card reports results on a kernel optimization task. It’s notable that their internal scaffold shows much higher performance than their public model.

“Claude Opus 4.6 obtained a 427× best speedup using an experimental scaffold and a 190× best speedup using our standard scaffold.

Economic Model Sketch

A promissory note.

I’m working on a more full-specified model, but I wanted to get this blog post out sooner. I feel this area is incredibly ripe for modelling, I’d love to find a theorist collaborator, & it’d make me very happy if someone else writes something about this.

Embodied and disembodied knowledge.

I find it useful to distinguish between embodied knowledge (the knowledge only applies to my own labor), and disembodied knowledge (I can scale my knowledge, e.g. hiring other workers or building machines). Knowledge-sharing seems to mostly apply to embodied knowledge, e.g. I use ChatGPT to learn medical knowledge I can use myself. Knowledge-creation is more applicable to disembodied knowledge, e.g. firms innovate.

A simple model of disembodied knowledge.

In a very simple model we have:

Aggregate output is determined by the best knowledge.
Aggregate profit is determined by the distance between the best and second-best knowledge.

Suppose there are people, each has 1 unit of labor, and there is just one good. Each person knows some subset of recipes , and each recipe yields some cost of producing the good from labor, . Then person ’s effective cost is the lowest cost among the recipes that they know. Each person can rent labor from others to produce the consumption good, and we assume labor is allocated via Bertrand wage competition among recipe-holders; workers are price-takers and work for the highest wage.

We can order the costs from lowest to highest, . In equilibrium the lowest-cost agent will rent the labor of all others, produce the good at the lowest cost , and then sell the good back at a price equal to the second-lowest cost (), and keep the remainder as profit:

Two simple implications:

Knowledge-sharing spreads output. If we share the best recipe among the whole population, now . Total output is unchanged, but profit is eliminated, and the output is spread equally among all actors.
Knowledge-creation increases output. Suppose we can improve the best recipe, . Total output will increase. The effect on profit will depend on (1) whether the identity of the lowest-cost producer changes; and (2) the degree of improvement.

We then wish to turn to the market for knowledge. The answers are somewhat sensitive to the knowledge-seller’s ability to commit & exclude knowledge-sharing. But broadly it’s clear that the demand for new knowlege is much less elastic than the demand for existing knowledge: it’s far more valuable to be the sole owner of new knowledge.

References

Abreu, Natalie, Edwin Zhang, Eran Malach, and Naomi Saphra. 2025. “A Taxonomy of Transcendence.” https://arxiv.org/pdf/2508.17669.pdf.

Amodei, Dario. 2024. “Machines of Loving Grace: How AI Could Transform the World for the Better.” October 2024. https://www.darioamodei.com/essay/machines-of-loving-grace.

Chatterji, Aaron, Thomas Cunningham, David J. Deming, Zoe Hitzig, Christopher Ong, Carl Yan Shan, and Kevin Wadman. 2025. “How People Use ChatGPT.” Working Paper 34255. National Bureau of Economic Research. https://doi.org/10.3386/w34255.

Footnotes

Many other technologies share knowledge – speaking, writing, printing, the internet – LLMs just continue this progression but further lower the costs of sharing.↩︎

Citation

BibTeX citation:

@online{cunningham2026,
  author = {Cunningham, Tom},
  title = {Knowledge-Creating {LLMs}},
  date = {2026-02-06},
  url = {tecunningham.github.io/posts/2026-01-29-knowledge-creating-llms.html},
  langid = {en}
}

For attribution, please cite this work as:

Cunningham, Tom. 2026. “Knowledge-Creating LLMs.” February 6, 2026. tecunningham.github.io/posts/2026-01-29-knowledge-creating-llms.html.

LLM verification

Tom Cunningham — Tue, 30 Dec 2025 08:00:00 GMT

A prediction: people will move towards producing documents that are machine-verified. A document will come with a checklist so you can see that it satisfies certain properties, as verified by LLMs:

	Claude	Gemini	GPT
Factual claims are accurate	✅	✅	✅
Logically consistent	✅	✅	✅
Central idea is novel	✅	✅	✅
The writing is readable	✅	✅	✅

If your blog post starts with this checklist I’ll be more likely to read it.

This is already happening for mathematicians and programmers: they verify LLM-produced proofs with a formal verification tool (e.g. Lean), and LLM-produced code with unit tests. I’m predicting that this pattern will spread to all other areas of knowledge work, as LLMs get better at verifying correctness.

Notes

A corollary: there’s a magic prompt.

Instead of saying “answer question Q”, it’s better to say “answer question Q, and give me a way of verifying that the answer is correct.”

You want the LLM to give you a checklist like the one above, decomposing the verification into many subproblems. Programmers have learnt to prompt “write a program to do P, and a set of tests to verify that it does P.”

Examples of criteria you want to check:

An infection prevention plan: verify that the plan is consistent with the relevant protocols.
A tax return: verify that each of the IRS rules are satisfied.
A legal memo: verify that citations are accurate.
An insurance claim: verify that the claim answers all relevant questions.
An insurance decision: verify that the decision is consistent with close precedents.

An analogy with two friends.

You have one friend who is full of new ideas, you have another friend who can tell whether an idea is good or bad. Each friend is somewhat useful, but when you combine them they’re amazing. (I think this is the case for mathematicians: LLMs will produce a fountain of proofs, and Lean can distinguish which are sound or unsound).

Implication: credentials become less important.

Many people are saying that LLMs will make credentials more important, because they make it harder to superficially distinguish high-quality and low-quality work. Ryan Briggs says:

“Prediction: in the short-to-medium term LLMs will make the reputation of the researcher matter more for whether or not we view results as credible because it will become too hard to read everything and people will want shortcuts for filtering. Again, this hits juniors hardest.”

It’s possible this is true but there’s a countervailing force: LLMs are better paper-writers, but also better referees. In fact they may be relatively better referees than they are authors, which would shift balance in favor of the less-credentialed. If we had a perfect test for the quality of work, we wouldn’t need to rely on reputation at all.

If someone entirely unqualified makes a breakthrough in ML or mathematics they can verify it. Historically this has been much harder in soft disciplines like economics, but if the cost of verification falls to zero.

A related point: in an old post on AI and communication I argued that with LLMs reputation will become less important for internal properties (where the ground truth is human judgment, i.e. verification is cheap), more important for external properties (where the ground truth is in the world, i.e. verification is expensive).

A More Precise Story

Different domains have different costs of verification:
- Cheap to verify: whether an image looks good, whether a joke is funny, whether a sudoku solution is valid, whether a formalized proof is sound, whether code passes a specific test.
- Costly to verify: whether a medical paper works, whether an academic paper is high quality, whether a human-written proof is sound, whether code fulfills a specification.
LLM-verification will be a big benefit in domains where it’s costly to verify.
There is a complementarity between LLM-generation and LLM-verification, the value of both is more than the sum of the value of each.
When doing LLM-generation it’s useful to ask the LLM to self-verify. E.g. by (1) generating a Lean proof and validating it; (2) generating unit tests and running them; (3) generating a checklist and asking an independent LLM to check each box.
LLM-generation can hurt communication equilibria where verification is costly, when LLM generation lowers the cost of accidental attributes (not essential attributes). E.g. if LLMs make it cheap to fix spelling errors, or to adopt idioms of the discipline, then there will be less separation in equilibrium.

Formal Models

A couple of very hasty models to sketch how to formalize this. It would be nice to have a single model which incorporates all the mechanisms above.

Model 1: quality vs polish.

Suppose you care just about intrinsic quality , but your signal is where is polish. You know that and are positively correlated (better books have better covers), so is a highly reliable signa of quality.

Suppose LLMs lower the cost of polish, so now everyone has high . This makes the signal-extraction problem worse, and you’ll rely relatively more on another signal, e.g. the author’s reputation (assuming it’s another signal correlated with ).

Suppose instead that LLMs lower the cost of directly observing quality . This will then imply putting relatively less weight on the author’s reputation.

Implications:

LLMs lowering the cost of polish will cause more weight to be put on reputation.
LLMs lowering the cost of verification will cause less weight to be put on reputation.

Model 2: search.

You have ideas with unobserved iid payoffs, and you can pay cost to find the true payoff (AKA Weitzmann’s Pandora’s box problem).

Claim: there’s a complementarity between the number of ideas you have () and the cheapness of verification (inverse of ). Formally:

From inspection we can see that the expression has a complementarity between and .

Forecasts of AI & Economic Growth

Tom Cunningham — Thu, 06 Nov 2025 08:00:00 GMT

Validation Checks

Economics and Transformative AI

Tom Cunningham — Thu, 02 Oct 2025 07:00:00 GMT

Thanks to comments from Daniel Björkegren, Andreas Haupt, Philip Trammel, Brent Cohn, Nick Otis, Andrey Fradkin, Jon de Quidt, Joel Becker.

This is a long collection of notes about economics & AI, prompted by two excellent workshops I attended in mid-September: the Windfall Trust’s “Economic Scenarios for Transformative AI” and the NBER’s “Workshop on the Economics of Transformative AI”. I had just left OpenAI’s Economic Research team, after releasing our paper, How People Use ChatGPT.

I’ll also try to explain a little why I’m excited about joining METR (I’ll explain why I left OpenAI another time).

Observations.

There is no standard definition of machine intelligence. There have been many attempts to give a definition or a metric of machine intelligence but most have been unsatisfactory. The lack of a common language makes work in this field difficult, but it’s also a big opportunity.
There is no standard model of AI’s economic impact. Economists have been using a wide range of assumptions to model AI’s impact, there is no standard framework. There seems to me an opportunity for ambitious economists to propose deep models of AI’s impact. A promising line would concentrate on AI’s ability to find low-dimensional representations of the world.
GDP will be a poor proxy for AI’s impact. AI’s benefits are likely to elude GDP for two reasons: (1) it will reduce the necessity for exchange (and GDP measures exchange); (2) it will lower the labor required for services, and the value-added from services are typically imputed from the wage-bill.
Transformative AI will raise the relative value of resources, and possibly lower the value of labor. If computers can do all human work then there will still be scarcity in natural resources (land, energy, minerals). Because humans require resources to do work (energy, land), demand for human labor will fall, creating a gap between land-rich and land-poor.
AI will likely have a discontinuous impact on science and technology. Many existing models treat computers as substitutes for humans in the R&D process, but there is reason to expect AI to have a qualitatively different effect on scientific progress.

A common thread is this that I feel economics is slightly under-fulfilling its potential. I feel there are a thousand important ways in which AI will change society, but not many attempts at grand unifying theories.

The Workshops

The Windfall Trust workshop.

This workshop asked participants to discuss four possible future scenarios. Each scenario had a page-long description of what might happen, but they can be summarized as follows:

Low capability growth (“the incremental path”)
High capability growth without regulation (“the runaway economy”).
High capability growth with labor-market regulation to protect jobs (“the great pushback”)
High capability growth with redistribution (“the post-work society”)

Windfall Trust Workshop

The NBER workshop.: The NBER workshop was a set of chapters written for a volume on the impact of Transformative AI on a dozen different areas: R&D, media, labor, competition, etc. Attendees were presenting the chapters they had written. The workshop was organized by Ajay Agrawal, Anton Korinek, and Erik Brynjolfsson.

NBER workshop

General Observations

My view: we are driving in fog.

My personal view is that it’s reasonable to have high uncertainty about both AI progress and the effects of that progress: (1) it’s possible that AI will get very good very soon; (2) it’s difficult to anticipate what will happen if it does.

We are driving in fog: it’s possible we have miles of open road, it’s possible we’re about to hit a tree. It seems sensible to be prepared for either outcome.

Background: a tension between AI researchers and economists.

Here is a very simplified characterization of AI researchers’ beliefs about economic impacts:

AI will soon be able to replicate most human labor.
This will cause mass unemployment.
We should fund a universal benefit from AI company profits.

Academic economists have often bristled at these arguments:

Economists have often been skeptics about AI capabilities and future AI progress (see below).
Economists argue (i) there are many frictions which prevent rapid changes; (ii) even if computers were better at everything than humans, there’s still comparative advantage; (iii) new technology will create new tasks for humans.
It’s not clear that AI companies will earn very high profits – most of the surplus from AI may go directly to the consumers of AI products.

Having spent a lot of time with both AI researchers and economists, I feel both have substantial blindspots, & could learn a lot from each other. AI people underestimate the frictions and sluggishness of economic processes; economists often fail to appreciate the speed of AI improvement & avoid having hypothetical discussions about future trajectories.

Many economists avoided talking about transformative AI.

Both workshops were intended explicitly for discussion of a hypothetical: what would happen if AI capabilities approached human-level performance on most work?

Despite this, it seems to me that most of the economists resisted the hypothetical. Looking at the NBER papers I think about half were effectively discussing the effects of AI with existing capabilities, not the effects of transformative AI.

This resistance was a common topic of discussion. People gave a few different reasons for concentrating on contemporary AI instead of future AI: (1) they were doubtful that AI would improve very quickly; (2) they thought it’s more important to work on the economics of actually existing AI; (3) they thought the economics of superhuman AI was formally less interesting; (4) they thought superhuman AI was possible and important, but were nervous about being perceived as credulous by their colleagues.

Chad Jones presented a paper which took seriously the existential dangers of AI, and argued that we should be spending hundreds of billions to prevent it. After his presentation the audience gave various comments but nobody (as I recall) gave a serious counter-argument. Nobody tried to dispute the substance of the argument – that an enormous asteroid is on a path to swipe our planet. Some people said this type of preventative work wouldn’t be politically feasible – but nobody seemed to express a sense of urgency you would expect if it was only political feasibility that was holding us back. The conference discussion mostly just drifted on to the next topic.

Defining AI capabilities is a hard problem.

It would be very useful to have a standard terminology for levels of AI capabilities. It would allow us to split our arguments into two parts: (1) when will AGI arrive? (2) what will happen when it gets here? We suffer from not being able to make this distinction. It is very often difficult to tell how much our disagreements are due to disagreement about capabilities progress, vs disagreement about the impact of that progress.

Conceptually the solution is simple but practically it’s hard. Many people have tried to define a set of capability levels but none have been widely adopted: most capability definitions are either too ambiguous or too narrow.¹

I think this is just a fundamentally difficult problem. It feels like something you should be able to figure out in an afternoon (and I spent many afternoons on it), but it has resisted many great minds.

Here are some examples of definitions that are too ambiguous: (1) it can pass a Turing test; (2) it can do “most economically valuable labor”; (3) it can exhibit PhD-level intelligence; (4) it can do the job of a customer service worker. Each of these covers a very broad a range of interpretations, including some such that 2024-level models could already do this.

Here are some examples of definitions that are too precise: (1) it can solve ARC-AGI, (2) it can write a paper that passes peer review at NeurIPS, (3) it can earn $1M. These are relatively unambiguous but it’s also easy to imagine cases where a model passes these tests while it still has limited economic value.

We are making some progress in characterizing AI ability.

In 2023 and 2024 AI ability was often described in terms of its grades on standardized tests (SAT, GRE, LSAT), or by the human-equivalent years of education, e.g. Leopold Aschenbrenner and OpenAI talked about “college-level intelligence” and “PhD-level intelligence.” This was a reasonable way of trying to make comparisons but clearly had limits: LLMs were PhD-level at some tasks, but also they were clearly kindergarten level at others.

In 2025 METR (Kwa et al. (2025)) made a good argument that a more robust metric of AI ability is the human-time-length of tasks that a model can do (see image). It’s an imperfect metric but I think it’s the best we have now.

It seems to me that having a better characterization of AI abilities, and the difference between AI and human abilities, remains a huge open question, and almost all questions about the impact of AI depend on this. This is what I’m personally most interested in working on, and the reason why I’m joining METR.

A graph of model capabilities across domains, indexed by the human-time-length of tasks, from METR.

I don’t think people appreciate the speed of AI improvement.

Speakers at the NBER workshop gave various examples of things the models still couldn’t do: (1) solve ARC-AGI puzzles; (2) extrapolate world models; (3) operate vending machines. These are all reasonable examples of limitations as of September 2025, but we should at the same time mention the rate of change. It seems very likely that the primary examples of tasks that AI cannot do in 2025 will turn into tasks that AI can do in 2026 (the same was true in 2024 and 2023).

Some general observations about the rate of progress:

ARC-AGI is falling quickly. LLM scores on ARC-AGI-1 went from around 10% to 80% over a year, and scores on ARC-AGI-2 went from around 3% to 30% over a few months. It’s true that a large part of the performance improvement was due to adding wrappers around LLMs, but the models can write their own wrappers, so it seems likely they’ll be able to solve the class of problems which can be addressed by an LLM plus wrapper.
Benchmarks are falling rapidly. It typically takes around 18 months for a newly introduced benchmark performance to go from 25% to 75%. We are frantically making up new tests to map out the limits of machine intelligence.
Consumer utility is growing dramatically. Use of chatbots has been more than doubling each year, both on the intensive and extensive margins. People far prefer answers from newer chatbots to older chatbots. The Elo score of models on chatbot arena is growing at around 150 points/year. This implies about a 70% win-rate, and equivalent to the difference in skill between the top-rated chess player in the world and the player ranked 100.²

From ObservableHQ.

The AI pessimism has mostly evaporated.

Over summer 2024 many economists predicted that AI would have small economic impacts:

Daron Acemoglu predicted AI would add 0.06% to annual productivity growth over the next 10 years.
Josh Gans said “I don’t think it will boost growth appreciably” over the next 10 years.
Robert Gordon predicted small effects.
Paul Romer predicted small effects.

I have heard much less of this kind of talk in 2025.

GDP forecasts have been modeling diffusion, but ignoring capability growth.

Two quantitative forecasts of AI’s GDP impact are Acemoglu (2024) (0.06%/year) and Aghion and Bunel (2024) (1%/year). Both of those papers model AI’s future economic impact as primarily a diffusion process: they treat the arrival of AI as a one-time shock like electricity or the steam engine which is gradually adopted and adapted, but asymptotes.

I think this a bad assumption because AI’s capabilities have been getting dramatically better over time (discussed more above), and we don’t know where the ceiling is.

Conceptually, we can decompose the growth in LLM adoption over a given period into (i) quality growth; (ii) diffusion. It’s very difficult to separate these, but I think a reasonable guess is that growth over the last 12 months (Sept 2024 to Sept 2025) is about 1/2 diffusion of existing capabilities, and 1/2 the causal effect of new capabilities (precisely: growth would’ve been half as large if models had been frozen in Sept 2024). If this is true then a forecast which was just based on the diffusion of existing capabilities would have dramatically under-estimated the impact of AI, and the gap would become larger as the horizon got longer.

Forecasts of diffusion are, by construction, forecasts which assume that AI progress will stop. But neither of these papers gives an argument why we should expect that.

GDP will miss a lot.

There are two reasons why focussing just on GDP will miss important effects:

AI will change relative prices. It seems likely that AI will lower the price of services (especially digital services) much more than the price of goods. If we just talk about the effect of AI on output overall, without distinguishing across domains, I think we will miss a lot. (I talk more about resource prices below).
AI services won’t show up in GDP. AI is already providing a great deal of value (there are 700M ChatGPT users), but the value mostly won’t show up in GDP by our normal accounting methods.³. In fact it’s plausible that AI reduces GDP because it reduces demand for expertise: I no longer call my garage-door-repair guy, because ChatGPT tells me how to fix the door. Services are generally accounted for in GDP just by the wages paid to service-providers. If people substitute from human service-providers towards AI then measured GDP will fall even though true output has increased. If service-providing firms pass through their cost-savings to customers then their measured contribution to GDP will fall. The same argument applies to much older technologies – the printing press, the encylopedia, YouTube.⁴

In 2024 Bloomberg reported on an OpenAI project to define five levels of AI capabilities (I worked a little on this). Also in 2024 Google Deep Mind released a 5-level framework, based on the percentile of human ability (level 2 means 50th percentile of skilled human).↩︎
It likely be impossible to approach a 100% win-rate, because (1) a fraction of people give noisy answers; (2) some easy queries have a unique correct answer; (3) some queries are ambiguous, meaning different people who ask the same query will have different preferences, and so no answer could ever get 100% of the responses.↩︎
Collis and Brynjolfsson (2025) estimate that generative AI is worth around $100/month to users in the US (i.e., they say they would require $100/month to give up these apps). This is far higher than the avg revenue or cost to provide these services.↩︎
Varian (2011) discusses various ways of estimating the value of google, e.g. a finding in Chen, Jeon, and Kim (2014) that people can answer questions twice as fast with a search engine as with a library. These issues were also discussed in Coyle and Poquiz (2025) at the NBER workshop.↩︎

We don’t have a standard model of AI

AI will change everything, but we don’t know how.

We want to predict the effect of AI on dozens of different areas of life – on medicine, on law, on entertainment, on news, on competition and markups, on scientific discovery, on personal relationships.

I will complain a little about the state of theorizing about the economic effects of AI. I feel that there’s not enough ambitious work which tries to model the effects of AI on human society in a deep structural way, which would allow us to make predictions across many domains.

If I’m missing some attempts please tell me.

Each paper used a different assumption about AI.

Each of the papers in the NBER workshop used some assumption about how AI changes the production function, but there were many different assumptions. Some of the ways that AI is modelled:

AI allows capital to perform a wider set of tasks (tasks that previously only labor could do).
AI allows humans to do certain tasks more quickly.
AI allows humans to do a wider range of tasks.

In many papers there’s no specific assumption on the types of task, just a parameter that reflects the share of tasks affected or the size of the effect. In some papers there are more structural assumptions on how tasks are affected, but there are a wide variety of assumptions, e.g. whether AI gives better predictions, or AI shares information. In some papers there are empirical assumptions, e.g. using an index of “AI exposure”.

Brynjolfsson, Korinek, and Agrawal (2025) is a “research agenda for the economics of transformative AI”: it seems to me that the paper lists many good questions but cites relatively few papers, and the papers it does cite are generally treated only as exploratory conjectures.

Structural models of AI.

Here are two classes of somewhat more structural models.

Agrawal, Gans, and Goldfarb (2019) – here AI improves the ability to predict some outcome, & the authors argue these predictions are typically a complement to human judgments. A couple of notable features: (1) in this model AI and human intelligence are qualitatively different, they assume that only humans can exercise judgment, which I find hard to interpret; (2) the model doesn’t seem to naturally predict the knowledge-sharing feature that I would regard as the primary economic effect of LLMs (see below).
Ide and Talamas (2024) – here each human has a certain level of knowledge (on a single dimension), and AI is able to substitute for that knowledge. This is an application of Garicano and Rossi-Hansberg (2006)’s model of the effect of information technology on organizational structure.

A pocket model: LLMs share knowledge.

Here is a simple mental model that I often use: LLMs share knowledge. The model is unsatisfactory in many respects but has the virtues of being very simple and very general. Consider an LLM as just a database of answers to questions, containing the set of answers that already exist in the public domain (i.e., in the LLM’s training set).⁵ LLMs therefore lower the cost of access to existing knowledge, and people will consult an LLM when they encounter a problem for which (i) they do not know the answer, but (ii) they expect that someone else does know the answer (and the answer was included in the training set).

This is a very crude model of an LLM but I think it gives a reasonable characterization of their adoption and effect so far. Around 1/3 of adults in rich countries are regularly using chatbots, and I think it’s fair to say the majority of the use is solving problems outside the domain of the user’s own expertise, but inside someone else’s expertise (see our ChatGPT paper). This knowledge-sharing model predicts that LLMs will flatten comparative advantage, so we should see more home production (people solve their own problems), less trade, and lower returns to experience.

The model has a number of imperfections as a general model of AI: (1) LLMs are often used to do tasks that don’t require knowledge outside the user’s domain, e.g. solving a problem that requires time and patience but not knowledge such as certain types of computer programming, writing, or creating images; (2) the model treats LLMs as strictly bound by the limits of human knowledge, this was a good approximation for early LLMs but it’s clear that AI is progressively expanding the boundary of human knowledge in a variety of ways.

This model is related to the Garicano-Ide-Talamas models in which an AI shares existing knowledge.

Distribution of queries from “How People Use ChatGPT”

Can we use existing models of human cognition?

There has been a lot of work by economists on the limits of human cognition over the last 50 years (AKA behavioral economics). This would seem like a natural quarry where we can get material for a theory of AI’s economic impact, however I’m not optimistic that there is much we can use.

Ideally we could start with a model of human decision-making and change the parameters to fit a model of computer decision-making. However there are not many clear candidates, theories in behavioral or psychological economics tend to emphasize biases: prospect theory, hyperbolic discounting, ambiguity aversion, heuristics, social preferences, rational inattention, two systems.

These are primarily theories of the weaknesses of human judgment, as such they do not seem to offer much explanation of the extraordinary strengths of human judgment – illustrated by the decades that it’s taken for computers to catch up with humans. These models don’t help explain why it has been so difficult to build a computer to do very basic human judgment and decision-making.⁶

A Conjecture About Deep Models

AI’s effect on a domain will depend on that domain’s statistical structure.

This will be a somewhat vague statement: my guess is that the most satisfying explanations of AI’s impact across different domains of human life (entertainment, news, hiring, shopping, etc.) will refer to the statistical properties of those domains, such as the latent dimensionality of that domain, or the strength of correlations in that domain.

Put another way: are there aspects of the statistical structure of the domain which a human brain can comprehend, but a computer brain cannot? are there aspects which a computer brain can appreciate but a human brain cannot? We already have well-established statistical theory on which types of estimator perform better on which types of data-generating process, the conjecture is that this type of theory will help organize a lot of observations about the impact of LLMs on social processes.

This line of thinking is inspired by some standard theory underlying the success of deep learning: that neural nets are able to learn low-dimensional structures in high-dimensional data (the “manifold hypothesis”, Bengio, Courville, and Vincent (2013)).

Examples of economic implications from statistical structure.

Here are a few brief cases in which the equilibrium economic effect of AI is determined by the underlying statistical structure of the domain. My conjecture is that these types of observations could be formalized in a common framework.

The concentration of the market for AI depends on the dimensionality of the world. If the world is intrinsically high-dimensional then the returns to model scale will be steadily increasing, and so we should expect high concentration and high markups. If instead the world is intrinsically low-dimensional then the returns to scale will flatten, and there should be low concentration (high competition) and low markups.
The effect of AI on scientific progress depends on the structure of the world. I give this argument below: if the world has a simple latent structure then progress will be bottlenecked more by intelligence than by data, and so advances in AI will dramatically accelerate scientific progress, without being bottlenecked on more data collection.
The wages paid to an occupation depends on the work’s latent dimensionality. If the work consists of tasks with high latent dimensionality then the returns to experience and ability will be high, and so wages will be high. As AI changes the incremental effect of human experience and intelligence we should expect it to change the structure of wages.
The demand for compute will depend on the self-similarity of the world. If 7 billion people all have very different problems then there are few efficiencies we can make in inference (through caching and distillation) and the share of GDP paid to compute will be high. If instead they have similar problems then the returns to additional compute will fall rapidly (demand will be inelastic) and the share of income paid to compute will be small.
The value of a matching algorithm depends on the dimensionality of preferences. Suppose we are predicting the quality of a match between a person and an item (e.g. a viewer and a movie, or a person and a job). If the latent structure of match-qualities is very simple then classic collaborative filtering algorithms will be very efficient, and neural nets will have small additional value (e.g. suppose preferences over movies can be largely expressed on a single latent dimension). But if preferences are highly idiosyncratic then more advanced AI, and wider data sources, will have big effects on equilibrium outcomes.

If AI can do everything then wages will fall.

What will happen when computers can do everything?

A lot of the discussion at both workshops was about the hypothetical world where computers can do everything that humans can do. Suppose there will be zero intrinsic demand for human-provision, i.e. people would not pay a higher price or accept a lower quality if a service was provided by a human instead of a machine. What would happen to human employment and wages.

Many papers argue that in this case human wages will increase: Pascual Restrepo made this argument at the NBER workshop (Restrepo (2025)), see also Caselli and Manning (2019), Smith (2024), Trammell and Korinek (2023) and Korinek and Suh (2024).⁷

The argument follows the standard trade argument: if a farmer moves in next-door who can grow every vegetable more efficiently than you then it’s good news: you can just specailize in the vegetables you are relatively better at and trade with the neighbor at an advantage. (If capital is a complement to labor then robots might cause wages to decline in the short-run but they would recover in the long-run, see Caselli and Manning (2019)).

If resources are scarce, wages will drop.

This reassuring conclusion depends on there being no other scarce inputs which humans and computers compete for. All of the papers above mention this qualification, they say that the conclusions might change if there are fixed factors, but they do not put emphasis on this point. However it seems to me that if we take seriously the hypothetical (that computers can do all work that humans can) then the resource constraints will very quickly bind.⁸

Humans have resource inputs - say 100 square feet and 2000 calories/day. If a computer can do every task at a lower resource cost than a human, and there are plentiful computers, then there would be no humans employed in equilibrium.⁹ Concretely, humans who do not own land would starve: the price of their labor, denominated in energy and land, would fall below subsistence level.

Among people who are still alive (because they own land, or from charity), would they work? Only if the incremental resource cost of working was below the resource cost of using a computer for that job.

An analogy: suppose we have a stock of A100 chips, then we start introducing more powerful H100 chips. Assume the H100 can do more tasks/hour across all tasks. As we acquire H100s we will keep using A100s for the jobs they are comparatively better for. But once we have sufficiently many H100s we will start to unplug the A100s to make room.

A feudal world.

Here is a sketch of a world with transformative AI. Probably I’m missing important things but I find it helpful to be concrete. Make these assumptions:

Every service and every good can be produced by a robot, with 1hr of robot labor exactly equivalent to 1hr of human labor.
A human requires 100 square feet to live, but a robot requires 1 square foot. I will treat land as the only fixed resource, you can imagine this as also representing energy and scarce minerals.
Suppose robots are sufficiently plentiful that they are rented at their resource cost, i.e. 1 square foot of land.
Suppose half of all humans own land while the other half do not. I.e. for half of all humans their only asset is their labor.
Suppose there is no intrinsic demand for human labor, people only care about the quality of the output, not who made it.

For people who own property you now have as many workers as square feet of land. You can effectively order anything you want from Amazon, or get any arbitrarily high-quality service (medical, massage, education, entertainment). Your primary constraints are space and time, not quality or quantity of goods and services.

However suppose you do not own land, and pay rent every month. In order to provide a service cheaper than your landlord’s robot your wage needs to be equivalent to 1 square foot of land, i.e. less than subsistence.

Taken literally this implies that people without assets will become dependent on charity. It seems plausible that the landowners would provide land to the land-poor, but still there would be no employment in this scenario.

A model with labor and land.

For completeness, here’s a simple model. Let land and labor be gross complements, and let AI be a perfect substitute for labor, and suppose we have an arbitrarily large quantity of AI. Then the marginal product of labor falls to zero and the entirety of the output will be held by the land-owner. Human labor no longer has any value and workers must live off the charity of the land-owners.

Formally:

as then the marginal product of labor goes to zero, and all income goes to land.

Notes:

I assumed labor and land are gross complements (). If production was Cobb-Douglas then making labor free implies we would get infinite output from a finite amount of land. As long as there is some limit on total output then land and labor must be gross complements beyond some point.
I assumed AI labor is free. We could instead assume AI has some resource cost. In that case the price of labor will be driven down to the input cost of AI labor. The input cost of AI labor could be defined in a few ways but they all appear to me low compared to exiting human wages: the land cost of a GPU is a few square inches, the energy cost is 400 watts.
We could distinguish between two sectors: a sector that requires land (goods) and a sector that requires only labor (services). If we introduce AI as free labor then human wages will retain the same purchasing power for services but their purchasing power for goods will collapse. Concretely: if you try to exchange your labor for goods you will have nothing to offer because the land-holder already has unlimited labor. The model predicts that workers will be not benefit from AI-produced-goods because AI requires land inputs.

Korinek and Suh (2024) argue that as AI progresses it will causes wages to increase then decrease, but this is a consequence of a specific assumption: that AI will has identical relative productivity across tasks to humans, among the tasks it can do, and so as AI becomes able to do more tasks then the equilibrium price vector first moves away from, then returns towards, the vector of human productivities.↩︎
Restrepo gives examples of work that humans might do: “For many socially intensive tasks—such as care work, hospitality, or therapy—the compute required to emulate human warmth or social intuition may be enormous. Even if such work is technically automatable, it may remain economically impractical to do so. As a result, these domains could continue to offer meaningful work for people.” However it’s not clear to me why “emulating human warmth or social intuition” would have an “enormous compute requirement”, more than, say, driving a car, or filing records.↩︎
Assuming computers are in sufficiently plentiful supply that we can ignore every other cost apart from their resource cost. If manufacturing computers requires resources then we could include their amortized manufacturing cost. If we have a scarce supply of computers for other reasons then of course this will keep human wages high.↩︎

AI scientists will be unlike human scientists

Will efficiency curves start dropping faster?: A good way of making the AI R&D question very concrete is to look at historical input-efficiency curves across a lot of different areas, and try to predict where they will go in the future. Should we expect them to start dropping faster? Which ones?; In fact I think these efficiency curves are a very good subject for making forecasts about: both as an output (expressing the practical impact of AI), and as an input (a way of expressing the capability of AI, to then make conditional forecasts with).

Our World in Data: Technology Costs over Time.

Most models of AI R&D are based on human R&D.: Models of AI’s impact on technological discovery are typically modelled on human R&D, e.g. (1) AI increases the effective supply of human scientists; or (2) AI automates one component of the R&D process. However both are modelled on a production function fitted on data with human researchers, and it seems to me likely that AI will qualitatively change that production function.
There’s another way to model this.: My instinct is that there’s a different way of modeling this that is more structural. Suppose we have an unobserved landscape, and it can be explored either by a human brain or a computer brain. Human brains have been exploring the landscape, finding successively lower local minima, and also finding general patterns in the landscape (e.g. physical laws). We wish to understand how much computers will speed up exploration of the landscape.

Random landscape. Here there’s no structure: every is an independent random draw.

Rugged landscape. This shows a Wiener process (random walk). Here there’s local correlation but no long-distance dependence.

Regular landscape. Here there’s some latent structure, implying that you can make long-distance predictions from local observations.

The effect of AI depends on the type of landscape.

Represent the landscape with a function , and each period we choose an to minimize . This is a well-defined explore-exploit problem, and we can characterize the expected progression of efficiency over time (the decline in over time) as a function of the statistical structure of the landscape:

Random landscape: If each is completely independent there’s no intelligence needed in choosing (beyond keeping track of which locations you’ve already tried). This is just drawing balls from an urn. The growth in efficiency as a function of draws depends on the distribution of values of (Muth (1986), Kortum (1997)).
Rugged landscape: If is correlated across but the correlation is local (e.g. if is a Weiner process) then the best-estimate of for a new will depend only on the neighboring values of . Callander (2011) and Carnehl and Schneider (2025) characterize the optimal strategy. Again we are not constrained on intelligence, only on data: the extrapolation algorithm is fairly simple.
Regular landscape: Finally suppose the landscape has some deep latent structure. In this case the best-estimate of will depend on the entire collection of previously-observed pairs , and so we do expect that predictions could be improved with more intelligence, and so AI should have a big impact.¹⁰

Implications of landscape regularity.

If the world has a regular landscape then we are not primarily constrained on facts, we are constrained on intelligence. Thus if we build a sufficiently powerful pattern-matching machine our progress might accelerate rapidly, without any new data collection.

Good examples of random landscapes are when we are mapping out specific features of the world. If we are making a list of specific objects (planets) or species (viruses), then the observations cannot be well-predicted from first principles, and we inevitably need new observations. Similarly, if we are mapping a genome then the exact sequence of base pairs requires individual observations, it cannot be accurately predicted from already-available data.

A good example of regular landscape is folding proteins: here we are learning a function from a sequence of base-pairs to a 3D shape. The function is high-dimensional but we have reason to expect a low-dimensional representation which would make it tractable - and AlphaFold found one.

AI R&D has already lead to discontinuities.

Many fields which have been progressing slowly show a discrete change in the rate of progress when computers took over:

Progress in solving optimization problems.
Progress in proving combinatorics theorems (four-color theorem in 1976)
Progress in chess strategy (Elo has fallen quicker since 1997)
Progress in protein folding.

Some of these have hit provably global minima: the four-color them; sphere packing; Ramsey numbers; Nash equilibrium of checkers, connect 4, texas hold-em, and chess endgames.

These accelerations have occurred when computer intelligence has surpassed human intelligence for a particular type of pattern-matching. As computer intelligence becomes more general then we should expect more and more lines of progress to start accelerating.

Aristotle already had the jigsaw pieces.

There’s a nice analogy for this: suppose we resurrected Aristotle, is it enough to show him our theories, or do we also need to show him the data we’ve gathered? Was he constrained on facts or intelligence? Did he already have enough jigsaw pieces, he was just lacking the insight?

It seems to me plausible that we could persuade Aristotle of some of the following from noting connections among the facts he was already aware of: that the sun is a star, that the earth goes around it, that the whale is not a fish, that the human is a monkey, that force is mass times acceleration, that temperature is motion, that pitch is frequency.

Predicting the effect of AI on R&D is intrinsically difficult.

The landscape model I described above implies that we should expect AI to have a big effect when some domain has a latent undiscovered structure. But in many cases this is very difficult to know in advance: we don’t know where the floor is. It seems conceivable that there are some very simple undiscovered principles explaining cancer, fluid motion, evolution. But maybe these domains are irreducibly complex.

References

Acemoglu, Daron. 2024. “The Simple Macroeconomics of AI.” National Bureau of Economic Research. https://economics.mit.edu/sites/default/files/2024-04/The%20Simple%20Macroeconomics%20of%20AI.pdf.

Aghion, Philippe, and Simon Bunel. 2024. “AI and Growth: Where Do We Stand.” https://www.frbsf.org/wp-content/uploads/AI-and-Growth-Aghion-Bunel.pdf.

Agrawal, Ajay, Joshua Gans, and Avi Goldfarb. 2019. “Prediction, Judgment, and Complexity: A Theory of Decision-Making and Artificial Intelligence.” In The Economics of Artificial Intelligence: An Agenda, edited by Ajay Agrawal, Joshua Gans, and Avi Goldfarb, 89–110. Chicago, IL: University of Chicago Press. http://www.nber.org/chapters/c14010.pdf.

Bengio, Yoshua, Aaron Courville, and Pascal Vincent. 2013. “Representation Learning: A Review and New Perspectives.” IEEE Transactions on Pattern Analysis and Machine Intelligence 35 (8): 1798–828. https://doi.org/10.1109/tpami.2013.50.

Brynjolfsson, Erik, Anton Korinek, and Ajay K. Agrawal. 2025. “A Research Agenda for the Economics of Transformative AI.” Working Paper 34256. National Bureau of Economic Research. https://doi.org/10.3386/w34256.

Callander, Steven. 2011. “Searching and Learning by Trial and Error.” American Economic Review 101 (6): 2277–2308. https://doi.org/10.1257/aer.101.6.2277.

Carnehl, Christoph, and Johannes Schneider. 2025. “A Quest for Knowledge.” Econometrica 93 (2): 623–59. https://doi.org/10.3982/ECTA22144.

Caselli, Francesco, and Alan Manning. 2019. “Robot Arithmetic: New Technology and Wages.” American Economic Review: Insights 1 (1): 1–12. https://doi.org/10.1257/aeri.20170036.

Chen, Yan, Grace YoungJoo Jeon, and Yong-Mi Kim. 2014. “A Day Without a Search Engine: An Experimental Study of Online and Offline Searches.” Experimental Economics 17 (4): 512–36. https://doi.org/10.1007/s10683-013-9381-9.

Collis, Avinash, and Erik Brynjolfsson. 2025. “AI’s Overlooked $97 Billion Contribution to the Economy.” Wall Street Journal, August. https://www.wsj.com/opinion/ais-overlooked-97-billion-contribution-to-the-economy-users-service-da6e8f55.

Coyle, Diane, and John Lourenze Poquiz. 2025. “Making AI Count: The Next Measurement Frontier.” Draft chapter for NBER volume {\it Economics of Transformative AI: A Research Agenda}. National Bureau of Economic Research. https://conference.nber.org/conf_papers/f227496.pdf.

Ide, Enrique, and Eduard Talamas. 2024. “Artificial Intelligence in the Knowledge Economy.” https://doi.org/10.1086/737233.

Korinek, Anton, and Donghyun Suh. 2024. “Scenarios for the Transition to AGI.” National Bureau of Economic Research. https://arxiv.org/pdf/2403.12107.pdf.

Kortum, Samuel. 1997. “Research, Patenting, and Technological Change.” Econometrica 65 (6): 1389–419. https://doi.org/10.2307/2171741.

Muth, John F. 1986. Search Theory and the Manufacturing Progress Function. Columbus: Ohio State University. https://doi.org/10.1287/mnsc.32.8.948.

Restrepo, Pascual. 2025. “We Won’t Be Missed: Work and Growth in the Era of AGI.” NBER Chapters. https://www.nber.org/papers/w34423.

Smith, Noah. 2024. “Plentiful, High-Paying Jobs in the Age of AI.” https://www.noahpinion.blog/p/plentiful-high-paying-jobs-in-the.

Trammell, Philip, and Anton Korinek. 2023. “Economic Growth Under Transformative AI.” National Bureau of Economic Research. https://doi.org/10.3386/w31815.

Varian, Hal. 2011. “Economic Value of Google.” https://dl.icdst.org/pdfs/files1/f87de5ba3c43760ebcbc2a1d90950dbc.pdf.

Citation

BibTeX citation:

@online{cunningham2025,
  author = {Cunningham, Tom},
  title = {Economics and {Transformative} {AI}},
  date = {2025-10-02},
  url = {https://tecunningham.github.io/posts/2025-09-19-transformative-AI-notes.html},
  langid = {en}
}

For attribution, please cite this work as:

Cunningham, Tom. 2025. “Economics and Transformative AI.” October 2, 2025. https://tecunningham.github.io/posts/2025-09-19-transformative-AI-notes.html.

On Deriving Things

Tom Cunningham — Thu, 30 Jan 2025 08:00:00 GMT

I’ve spent a lot of time trying to prove things.

With diagrams and algebra, back and forth between clipboard whiteboard blackboard & keyboard.

I can’t talk about what it’s like for a good mathematician but I can talk about it for a hack.

I can prove true & interesting things occasionally but only after wallowing in it for a long time, and after a half-dozen proofs of things that I later realize are not interesting or not true.

The representation of a problem becomes gradually more abstract.

I start with a full set of equations, then I gradually omit things that seem inessential, and I introduce a new symbol which feels like it denotes something but I’m not sure what, and then another higher level of abstraction, a jenga tower to the ceiling. I’m holding it very delicately and it’s crucial not to lose that concentration as I can see it teeter. Sometimes I stare out the window too long and when I look back down at the page the symbols no longer make sense: was this a matrix and that a vector? Why was I multiplying them together in this weird way?

I take a problem and present it to my intuition to see if there’s a glimmer of recognition.

I show it from different angles: vary the notation, write worked examples, draw arrow diagrams, draw venn diagrams, hoping that one of these approaches will trigger a memory of something similar.

I walk around the house checking the windows and doors. I check whether the answers are in the right units: dollars/person, quantity/year, and I check whether the signs go in the right direction.

I lose track of which way the inequality should be pointing, left or right.

My intuition doesn’t give me a strong signal so I just choose one and stick with it hoping that once I arrive at the conclusion I can work backwards and fix the directions.

Margaret Bray told me that, when she worked correcting Stiglitz’s proofs in the 1970s, she would sometimes find an odd number of sign errors instead of an even number. When that happened she would tell him it and he would quickly rewrite the discussion of that result: substituting an intuitive explanation of why the effect of on F was negative for what had been an explanation of why the effect of on F was positive.

Sometimes the goal is to maximize the distance between the assumptions and the conclusion.

I’m trying to make it seem more impressive. At first the starting point and end point are within sight of each other, and I hack away at either side, loosening the assumptions, tightening the conclusions, until the path stretches as far as possible.

I sometimes end up chasing what looks impressive. I twist the wording of the assumptions to make them seem weaker than they are, the conclusions to seem stronger, or hide ancillary assumptions in the body of the derivation. Looking back to some of my old papers I see signs of this and I’m ashamed of it.

There are a half-dozen different things I need to keep ambiguous.

I’m not sure which disambiguation will be the right one - whether to use strict or weak preferences; whether the numbering should start at zero or one; whether naturals or wholes or integers. Making a decision at one stage has implications for others. This part is Sudoku, finding something which fits many constraints.

I switch between two modes: is this true? How do I show everyone it’s true?

In the second mode it’s just trying to keep everything straight to nail it down but then occasionally a sickening feeling arises that it’s not my clumsiness that’s holding me back but perhaps the thing isn’t true after all, and I have to return to the first mode.

Sometimes I’m stumbling through equations trying to derive one from another but I have lost track of what they mean.

Like going through a room in the dark grasping for a door handle. I want to go back and draw some diagrams to get an intuition of what’s going on, some mental picture, but sometimes I try and try and nothing comes.

My coauthor suggests modifying an assumption in the setup.

Taking apart a Lego car to rearrange the unsatisfactory front wheels but it’s hard to anticipate all the downstream consequences of that; I don’t remember all the considerations I had in mind when I put together the front wheels.

Eventually the relief of finding the hidden door.

Suddenly I see the inequalities holding down variables like saxophone keys, which gives me the lemma: a condition for a set of inequalities between the elements of two vectors to imply an inequality between the sums of the vectors. I don’t need to write it down, I can take a break and go for a run, it’s enough to remember the idea of saxophone keys. Later I find out this is called Hall’s marriage theorem.

I’ll be writing out an equation for the 3rd time and suddenly a dusty image will be released.

When I quiet my brain to concentrate on just one thing the background noise becomes audible. I can hear the mice scratching in the walls. Not just the familiar memories, unfamiliar long lost memories come out too.

A sparse park in Manchester, a man who blanked me in a corridor in London, a cheese shop in Stockholm.

My desk is covered with notes: should write up what I have or push on towards the summit?

It’s a dangerous tradeoff. In retrospect it seems like I’ve often made the wrong decision: too-often tried for the summit, failed, and then left without a proper writeup. I come back to make another attempt, see my old footsteps and regret that I didn’t spend more time hammering nails into the rock.

Sometimes a gap opens up between what’s important and what’s orthodox.

If I make an assumption that is somewhat stronger than typical then everything is much more elegant and it feels like I can keep focus on what’s important. But then I get memories of being in seminars and seeing the speaker making an unorthodox assumption and how the energy drains out of the room.

For a while I tried to solve things falling asleep.

Once, in Goodenough College in 2008, as my mind untethered itself from its dock I suddenly glimpsed that the integral could be taken vertically instead of horizontally. The problem was solved. For a few months afterwards I would put my book aside before I was ready to go to sleep and bring to mind some other unsolved problem but it never worked as well again.

More memories are released.

Beer at the dreary Birkbeck bar: we were all grasping around for things to say to each other to fill up an evening with. When we heard the kids roaring by the pool table we all looked up.

Saturday morning on the way back from the vegetable market in Farringdon. I sit down with a sandwich in a churchyard and I see my German neighbors are there too. I’m self conscious that my loneliness is exposed.

Making Will’s grandmother a shandy of beer and lemonade. Noticing the shadows of the roses in her garden, how they would change when I stood near them, the proximity of my own shadow would somehow make the shadows of the roses come into focus.

Perhaps 1/3 of the time is actually concentrating on the problem.

If it’s writing or programming I can just bring up a window and type away. If it’s deriving things then my mind is constantly drifting, trying to find some other path to go down – in reverie, doodling Zeus, refilling my pen, or looking something up.

On a good day it’s like swimming in cold water.

I don’t want to get in but once I’m in I don’t want to get out.

Once my chair is firmly pushed in at the desk I don’t want to get up to get a snack, to check the mail, I don’t want to go to the bathroom, I’ll be working through proofs while shaking my legs trying to stretch my bladder out.

I procrastinate starting work on a revision but once the document is open and the seal is broken I see things to fix and it’s difficult to stop.

When daughter was a baby she would push away a bottle of warm milk until the nipple was in her lips and then she’d clutch it tight while she’d drink.

On a bad day I’m rotating slices of a Rubik’s cube.

I’m checking to see if it’s solved then rotating again in some other direction.

Often I’ll over-estimate what I can accomplish.

I’ll have some cocky confidence that, this time, I can prove something I’ve failed to prove before. Or I’ll have a sketch and decide that it’s good-enough, that I can fill in the details later. Then later get a sickening feeling discovering that it’s really nothing, the bits omitted from the sketch were exactly the difficult parts. The voice in my head that causes these over-optimistic judgments causes other griefs in my life I think.

I get flashes of recollection of other peoples’ seminars and papers.

They seemed so boring and now my paper seems so similar to theirs.

The ideal state of mind has both (1) the clarity of fresh eyes; and (2) the suppleness of familiarity.

For that reason it makes sense to spend an entire day thinking about it, not little blocks of hours.

Weaselly unworthy thoughts bubble up.

I find myself thinking that a reader will be impressed by the quantity and the complexity of the derivation and give me credit for that.

When I’m trying to concentrate on something my weasel thinks of something I could order on Amazon.

I can suppress that. My weasel tries to get me to look up how to do a LaTeX symbol that would be useful (delta over equals). I can suppress that. The weasel sees a pair of brackets that could have an extra space. I give in. I fix the brackets. But when I’m fixing brackets I’m more vulnerable to each of the other temptations, and then after a while the weasel tells me it’s almost lunchtime. I look back on my morning it looks like swiss cheese.

I’m a grown adult but my concentration is still not under my control.

I put out treats for it, entreating it like a dog. Teach it bad habits. Give in to its whims.

When I switched from programming to studying economics I missed those numb hours that would pass by.

Some people seem to get into that state when they’re deriving things but it’s difficult for me to achieve that.

When you’re programming you get incremental feedback: you can see the mountain peak and you’re slowly getting closer to it. With proofs you’re going through the jungle and you don’t know if you’re getting closer or farther away. You could be on the brink of emerging into a clearing or it could be months more in the forest.

Why am I doing this?

As I’m feeling around a problem I think of a guy I knew from seminars at Harvard who would pause when a new slide came up and then nod quietly when he understood. Why do I spend my time at this when so many other people are better at this than me? Am I doing this for intrinsic or extrinsic reasons? One instinct is to untangle the threads of means and ends to see where they lead, another instinct tells me to leave it tangled, and I trust the second instinct more.

I look up from the corner I’m stitching.

I see the quilt goes to the end of the bed, out the window, across the fields all the way to the horizon.

Reading through an old draft like a landlord returning to his estate after a long trip.

To be reminded of the state of things. I’m more often surprised unpleasantly than pleasantly. My memory must be pickling things in a sweet vinegar.

Coming back to writing a paper after 5 years at Facebook I had a different kind of confidence.

Old anxieties were about failing to do things in the orthodox way. Now I feel that if it makes sense to me I won’t be embarrassed by it. I might be wrong but I’m not a fraud.

I’m comparing myself, and it seems unhealthy, but then also necessary.

Am I clearer sighted than I was last month? My decision about whether to attempt to solve this problem depends on that judgment.

I see that the problem I’m attempting is one that was notably not solved in a paper by Benabou and Tirole: whether I should give up now depends on how I assess my ability compared to theirs.

I’ve spent two hours trying out different matrices to see if they have a certain property, looking for a generalization about which matrices satisfy it, I still haven’t got one and now I don’t recall why I felt this was important – how much do I trust my decision to go down this road, and at what point do I go back to reconsider whether this problem is worth solving?

In Tel Aviv I would swim in the sea in the mornings.

By the third week I was still going down to the beach, getting in the water, and starting to swim, but after 30 seconds I would let my legs fall to the sand and just walk through the water staring into space, thinking. After a while I would remember to start swimming again.

Some things I proved

It’s likely that some of these had been proved before, but they each seemed useful enough to prove in the time and place.

The interpretation of an experimental outcome will depend on other outcomes.: Your estimate of the treatment effect on one outcome will depend on the observed effect on the other outcome, and the weight will be proportional to the difference in covariances between the treatment effects and the noise (if everything is joint Normal).
If item values are Normally distributed then selections will look like ellipses.: If you’re ranking items by predictions, e.g. ranking by , and the probabilities have a joint Gaussian distribution, then the overall tradeoff between total and total will be an ellipse, equal to an isovalue of the joint distribution of .
Modularity in the brain will cause characteristic inconsistencies in decisions.: If information is dispersed in the brain, and aggregated sequentially, then (a) mistakes will occur when there are interactions in your posterior (nonseparabilities), and (b) certain characteristic inconsistencies which will reveal those nonseparabilities.
Implicit preferences can be inferred from choices. (with Jon de Quidt): Observing choices over outcomes which differ in attributes can reveal the difference between implicit and explicit preferences – where implicit preferences can be due to either (a) tacit knowledge, (b) signaling motives, or (c) constraints on which decisions are allowable.
Noisy signalling advantages senders. (with Ines Moreno de Barreda): Suppose you’ll admitting a student to your PhD program only if they have a GRE score sufficient to imply their ability is above median. If students can exert effort to inflate their scores then you’ll end up admitting more than half of the students, even if you rationally adjust for the inflation.

Anatomy of a Mistake

I missed a malignant ambiguity because there was another ambiguity next to it, and that other ambiguity was benign.

There were two versions of a claim, and I was a little vague about which I was assuming:

My conclusion required the strong version but my assumptions only justified the weak version.

Why did I miss this? Because there was another ambiguity that was floating around, claim B:

It happens that assuming (never mind what that is) implies both (B) and (A-strong). My thought process:

I don’t want to restrict to .
I knew that (B) is not true for all , but it is true for sufficiently small values, i.e. for some .
So I was juggling two versions of an assumption about ( is zero, is small), and two versions of (A).
When my mind wandered onto whether claim (A strong) was justified I was reassured by remembering that it’ll work when (true for both claim A and B), and additionally that for it’ll be OK (true only of claim B).

The key thing: if claim B had not have interfered I wouldn’t have made this mistake.

In the same way a pickpocket will wait for you to move before taking something off you, because the friction you feel on your buttock is attributed to the walking, not to the wallet leaving your pocket.

Too Much Good News is Bad News

Tom Cunningham — Thu, 26 Dec 2024 08:00:00 GMT

Here are two nice pieces of Bayesian logic observations that help explain everyday intuitions:

When an outcome is the sum of two components then your belief about the contribution of the thinner-tailed component will be first increasing then decreasing in the realization of the outcome.
When you observe an outlier in some process, which is the sum of multiple components, then:
- If the components are thin-tailed, then the outlier implies each of the components is elevated.
- If the components are fat-tailed, then the outlier implies just one of the components is elevated and the others are at their ordinary levels.

Too Much Good News is Often Bad News

In many cases a signal which is good news eventually starts to become bad news:

The longer you wait for a bus the likelier it is to be about to arrive, until at some point it’s more likely that you’ve missed it.
The lower the price the better the value until it becomes suspiciously cheap.
If a drug is associated with a 5% higher rate of birth defects it’s probably a selection effect, if it’s associated with a 500% higher rate of birth defects it’s probably causal.¹
If an AB test shows an effect of +2% ( 1%) it’s very persuasive, but if it shows a an effect of +50% ( 1%) then the experiment was probably misconfigured, and it’s not at all persuasive.²
When reading a biography each detail makes the the subject seem more impressive until you start to doubt the neutrality of the biographer.

In each of these cases I think it’s because the noise has a fatter-tailed distribution than the signal. As a consequence when you see a very-high observation you conclude it’s mostly noise, which implies that after a certain point an increase in the observed outcome becomes bad news instead of good news.

Outliers Usually Have One Cause, Not Many

This is a related but slightly different point. Scott Sumner argued “extreme events generally have multiple causes”, with a few examples:

Italy’s very high COVID death rates are probably due to a number of different factors, not one.
The great depression was due to “multiple policy errors, on both the supply side and the demand side.”
Bob Beamon’s record long-jump in 1968 was probably a coincidence of multiple causes.³

I think Sumner is wrong in his generalization: extreme events typically have a single cause, not multiple causes. Formally (following Nair, Weierman and Zwart below), extreme draws from a sum of thin-tailed influences tend to have many causes, but extreme draws from a sum of fat-tailed influences tend to have one cause.

We often cannot observe the distributions of the individual components but we can observe the distribution of the final aggregate outcome, and if the final outcome is fat-tailed then at least some of the components must be fat-tailed.

I’m not sure about Italy’s COVID and the great depression, but for Bob Beamon’s jump in 1968 it looks like a non-Normal outlier, not the sum of orthogonal influences (see the spike on the blue line at right). Thus from this evidence we should expect, all else equal, that Beamon’s 8.9 metres was due to one big cause not many small ones.

Formalizing This

We can summarize the theory with three observations.

Suppose you observe a variable that is a sum of multiple components. Then:

If components are IID and thin-tailed: outliers are due to many small causes (“conspiracy”).

If components are IID and fat-tailed: outliers are due to a few big causes (“catastrophe”).

If components are a mixture of thin-tailed and fat-tailed: low observations are due to the thin-tailed components, high observations are due to the fat-tailed components (“rejection”).

Literature. De Finetti wrote a paper in 1961, “The Bayesian Approach to the Rejection of Outliers”, giving a simple example where, with fat-tailed noise, a Bayesian will discount outliers.

There seem to be two modern strands of this literature that use somewhat different terminology:

Anthony O’Hagan and Luis Pericchi (2012) Bayesian heavy-tailed models and conflict resolution: a review. They give a number of conditions under which you get “conflict resolution” or “rejection of outliers”, broadly speaking when the noise has fatter tails than the signal. The critical condition is the relative speed of decline of the tails of the distributions of signal and noise. The illustration at right is very elegant, showing how the posterior will be bimodal when you have the sum of two heavy-tailed distributions.
Nair, Weierman and Zwart have a chapter “Catastrophes, conspiracies, and subexponential distributions” in their book “The Fundamentals of Heavy Tails”.
- Suppose you observe the sum of a set of IID random variables. They discuss two polar ways in which your posteriors about the components will depend on the sum:
- If the components are heavy tailed you get the “catastrophe principle”: the probability that the sum exceeds some value will be approximately equal to the probability that the maximum of components exceeds , as .
- If the components are light-tailed you get the “conspiracy principle”: the probability that the sum exceeds some value dominates the probability of the maximum exceeding , as .
- Applications of the catastrophe principle: If a certain year has many earthquake deaths then probably there was one large earthquake, not many small ones. If a random group of people has a high average number of Twitter followers, probably one member of the group is a big outlier, and the others have an ordinary number.
- Applications of the conspiracy principle: if a random group of people has a high average height then probably each individual is tall.

Note: non-monotonicity if and only if the sum is log-convex.: Suppose we observe which is the sum of and . Tweedie’s formula will give us from the empirical distribution of , as long as the distribution of is from an exponential family. If is Normal with variance then we have:; There are two interesting bad news cases (i.e. non-monotonicity):; These correspond to:; A nice observation: if the empirical distribution is log-convex at some point , then the expectation of the Normal component must be decreasing at that point. (Note that log-concavity is a common way of characterizing light-tailed distributions.)
Note: bus arrival times.: Suppose the bus arrival time is , then at time the expected wait time is . We’ll assume , then there are two interesting quantities:; Here the expected wait time is uniformly decreasing in . However suppose that if we’ve missed the bus then the next expected arrival time is (e.g. tomorrow). Then we have expected wait time:

which will be decreasing, then increasing, then decreasing in .

You could supplement this model by saying you arrived at the bus-stop at time , and it’s now time . then we’ll have terms like , but I think the basic non-monotonicity will still hold.

Footnotes

Bradford-Hill: “the mortality of chimney sweeps from scrotal cancer was some 200 times that of workers who were not specially exposed to tar or mineral oils.”↩︎
This case is somewhat subtle: we generally think that treatment effects are fat-tailed, and we can be confident that noise is Normal because it’s the sum of many IID variables. However there’s an additional source of noise from implementation error which has even fatter tails than the distribution of treatment effects.↩︎
“While Beamon received mostly accolades, there also were detractors. The critics harped on the conditions — a following wind of 2.0 meters per second (the maximum allowable velocity for a record), a lightning fast runway and, most important, the thin air of Mexico City. Beamon’s defenders point out that the other competitors, which included the world record co-holders, had the same factors going for them and they didn’t jump close to Beamon.”↩︎

Premature Optimization and the Valley of Confusion

Tom Cunningham — Fri, 10 May 2024 07:00:00 GMT

When formalizing tradeoffs our decisions typically get worse before they get better.

Those who climb the mountain of efficiency first pass through the valley of confusion.

Data scientists are often asked to provide a formula to calculate the expected costs and benefits of a decision but this is hard. There are often many subtleties which we grasp intuitively but do not know how to formalize. For this reason using a simple model is often worse than no model.

Some examples of common mistakes:

Causal effects assumed to be equal to correlations.
Decisions depend on a sharp threshold of statistical significance.
Decisions depend on average rather than marginal value.
Lack of accounting for opportunity cost.
Lack of accounting for option value.
Lack of accounting for network effects.
Confusion over whether an outcome is intrinsically desirable or instrumentally desirable.
Estimating the topline impact of product improvements with estimates from AB tests (i.e. accounting only on effects for existing users, not new users).

These are all avoidable mistakes but it takes time and experience to model them correctly. In fact many of the tools we use to formalize optimization problems are relatively recent discoveries: over the last few centuries we figured out how to write down probabilities and expected value, linear programming, and dynamic programming. But before these discoveries we were still able to make quite subtle and complex decisions – we were able to build pyramids, welfare states, aeroplanes – our informal and intuitive methods were sufficient.

This is independent of whether decisions should be formalized. There are some situation where no formalization of costs and benefits will be sufficient, for various reasons. My point is different: even when we truly care only about a well-defined outcome it’s often better to rely on our instincts than to use a model if we don’t have time to deeply invest in that model.

It’s a common idea that formalizing the costs and benefits of a decision can lead to worse decision-making. People talk about “MacNamara’s fallacy,” but this can be taken in two ways: whether problems are intrinsically unquantifiable or just difficult to quantify. In tech companies I think many problems are quantifiable in principle but it’s sufficiently hard that it’s sensible to leave off quantifying until you’re confident that you have a good model.

The process of formalizing a decision often looks like a dialogue. When you try to write down a model to represent the tradeoffs of a situation very often it feels like a dialogue with your intuition: you first write a simple model, and see that it implies that we should be making a radically different decision. You try to think through, intuitively, the implications of that decision, and see that there is some important factor that the model is missing, and so revise the model, and then look again at the implications.

Avoiding the valley of confusion. I have been talking about the choice between using judgment or relying on a model. There is an alternative route: instead of replacing judgment the goal should be to augment it. Broadly speaking by trying to summarize data in a way that helps inform intuitions about tradeoffs. If we continue this process we may end up at a comprehensive formal model which we can trust.

Some examples:

If we are trying to improve decisions about which experiments to ship then we can build a table of prior experiments, and for each new experiment summarize how it compares with prior ones. (My post on experiment interpretation expands on this.)
If we are trying to improve decisions about the value of metrics as surrogates for causal effects, we can start by visualizing the distributions of each metric and the correlations between them to help build judgment about how they relate to user experience.
If we are trying to improve evaluation of a given initiative we should start by mapping out prior initatives and compare their inputs (e.g. headcount, time) and outputs (metrics), to benchmark estimates.

General considerations that apply to all evaluations:

All models should be described as fundamentally aids to human judgment, and it’s always legitimate to override their recommendations.
Model output should not be described with a single number but a visualization to show how the broad patterns in data.
We should summarize what decisions other people have made in similar situations, both inside and outside the company.

Peer Effects, Culture, and Taxes

Tom Cunningham — Sun, 28 Apr 2024 07:00:00 GMT

In short:¹

¹ First version January 23 2023. Thanks to comments and conversations with many, incl. Jon de Quidt, David Atkin, Donald Kenkel, Dmitry Taubinsky, Charlotte Paul, & Ruth Cunningham.

The majority of variation in preferences is due to the influence of peers. Peoples’ tastes reflect what they are accustomed to, either from their upbringing or from their contemporaries. Tastes for food, alcohol, tobacco, music, art, and leisure, vary much more between societies than within societies. This is a banal observation but I think when combined with standard economic analysis it has fairly radical implications for policy.
Peer influences imply that group behaviour will respond to changes in the environment with a lag. Contemporary preferences will reflect the prior environment of a group, because peer effects will cause persistence. If your society once had a reason to choose potatoes over bread then the preference for potatoes will linger after the reason has disappeared.
The slow adjustment of preferences implies welfare can be improved. Peer effects are an externality and so taxes which offset it will hasten convergence and raise welfare. This would justify taxes to discourage goods that have been recently been discovered to be unhealthy: tobacco, sugar, trans fats & saturated fats.

Summary

Most variation in preferences is due to peer effects. Around the world there is substantial variation in choices over food, clothes, education, working hours, fertility, etc., much of which does not seem to be a consequence of economic constraints, but must reflect a difference in preferences. We can decompose variation in preferences into (1) genetic, (2) shared environment (upbringing and peers), and (3) idiosyncratic differences.

The majority of the variation seems to be due to shared environment because most of the variation is between group rather than within group. I.e. we can predict most of a person’s preferences from knowing the preferences of their peers, and it’s implausible those differences are genetic. I discuss evidence further below.

Aggregate demand will be more elastic than individual demand

Implication: aggregate demand will be more elastic than individual demand. Peer effects causes everyone to bias their behaviour towards what other people choose, as a consequence each individual’s response to changes in the environment (e.g. an increase in price) will be damped in comparison to the aggregate response.

Implication: today’s preferences reflect yesterday’s prices. If peer influence is based substantially on peers’ historical behaviour (as opposed to being forward-looking) this will tend to retard the adjustment of decisions to circumstances, so that a society’s set of decisions at any given point in time will reflect the prior history of circumstances faced by that society.

Peer effects will cause inefficiently slow adjustment to price changes.

Implication: adjustment is inefficient. Peer effects mean that consumption decisions have an externality: when you increase your consumption it will cause others to increase theirs. In the long-run this will not cause consumption to be too-high or too-low, but it will make adjustment to a change in circumstances slower than would be efficient.

Implication for policy: hasten adjustment. Everyone could be made better off with a tax to correct that externality and hasten adjustment to the new equilibrium. When we discover new information about the health effects of some good (tobacco, sugar, saturated fats), peer effects will cause an inefficiently slow adjustment of consumption, and everyone would be made better off by a temporary tax to hasten that adjustment process.

A Simple Model of Peer Effects

In short: With peer effects the aggregate demand curve will be more elastic than the individual demand curve. Additionally long-run aggregate demand will be more elastic than short-run aggregate demand.

I give a fuller derivation of this model below, here I just state the setup and the key results.

Each person will choose a level of consumption between their exogenous preference and the population-average consumption (represented by the dashed line).

Setup: Suppose each person has a pre-existing ideal level of consumption (of alcohol, sugar, tobacco, etc.), but they additionally get disutility from departing from the average level of consumption among their peers. Assume the reference-point is the lagged level of consumption among peers.

Implications:

Long-run average consumption will be independent of peer effects. In the long-run equilibrium peer effects will not systematically bias consumption up or down: the average consumption will be equal to the average pre-existing preference.
Aggregate elasticity will be higher than individual elasticity. Society as a whole will be more sensitive to changes in price (or changes in information) than each individual will be.
Long-run aggregate elasticity will be higher than short-run aggregate elasticity. When the price changes (or information changes) then each individual will respond but their response will be retarded due to peer effects from historical levels of consumption. Thus aggregate consumption will take some time to fully reflect changes in the environment.
Aggregate demand will respond inefficiently slowly. When the price changes then aggregate consumption will gradually adjust, but that rate of adjustment will be slower than the utilitarian (welfare-maximizing) rate of adjustment. The implication is not that we should tax unhealthy goods, but that we should tax goods which we have recently learned to be unhealthy, to help quicken the convergence. When we discover negative effects of saturated fat, tobacco, sugar, then we should tax people to bring them quickly to the new equilibrium (and when when we learn things are surprisingly healthy then we should subsidize them).
Biases will be magnified. If there is a bias in decision-making, e.g. myopia regarding health effects, then the aggregate effect of that bias will be larger than the individual effect. Thus a tax calibrated to the individual effects of myopia would be too small.
Variation in consumption will be higher than optimal. When someone shifts their consumption farther from the mean they have a negative externality on everyone else, and so welfare would be improved if everyone conformed somewhat more than they do. This implication was surprising to me but seems to follow from the model.

Relation to Literature on Health Taxes

The key point of this note is that the existence of peer effects would justify a tax on unhealthy goods, independent of any direct externalities or internalities from consumption, along a transition path when we learn new information about that good. I haven’t found discussion of this point in the economics literature on health taxation. The literature in public health often seems to make this argument implicitly but I think it deserves to be made much more clearly.

Most countries have substantial restriction on unhealthy goods (“sin taxes”). Most countries have substantial taxes on tobacco and alcohol, outlaw many drugs, carcinogens, trans fats, and many countries are planning to introduce a tax on sugar.

The economics literature is not in consensus about the appropriate taxation of unhealthy goods. E.g. DeCicca (2022) reviews the evidence on tobacco taxes and conclude that they are higher than would be justified by the purely economic externalities, e.g. through healthcare (“evidence on the magnitude of the externalities does not support current tax levels.”) They then consider the evidence for “internalities,” i.e. myopic decision-making by consumers, and say that the evidence and theoretical framework is too sparse to draw a conclusion (“the empirical evidence on the magnitudes of the internalities from smoking is surprisingly thin.”)

Discussion of optimal taxation rarely discusses peer effects. DeCicca et al. (2022) mention the existence of an empirical literature on the peer effects on smoking, but don’t discuss the findings, or whether it would have implications for setting taxes.

Kenkel et al. (2002) calibrate a model of rational addiction with peer effects. They show that peer effects will tend to magnify distortions due to other externalities, and so increase the size of the efficient tax for a given individual elasticity. In their model peer effects only magnify existing distortions, they do not justify a tax by themselves. However that paper does not discuss the role of adjustment to new information (the focus of this note), a case in which peer effects will cause inefficiencies even without other externalities.

Allcott et al. (2020) argue for a 2c/oz tax on soda based on (1) a 1c/oz externality due to healthcare costs, and (2) a 1c/oz “internality” due to peoples’ myopic decision-making. They do not (I believe) discuss a peer-effect justification for taxation, which would constitute an additional separate justification for a soda or sugar tax.

The Public Health literature discusses peer effects, but it’s unclear what role they play in setting policy. Since the 1990s the Public Health literature has put a lot of emphasis on “community health” or “population health” interventions, in part due to the perceived importance of social norms, AKA peer effects. However this literature rarely includes an explicit calculation of costs and benefits to justify a given magnitude of intervention. As a consequence it’s often unclear whether peer effects are thought to be relevant due to magnifying some other distortion (e.g. a fiscal externality or myopic decision-making), or due to retarding the aggregate adjustment to new information. However I believe the informal reasoning used in public health essentially appeals to the argument I am making: they argue that peoples’ health would be substantially better off with small lifestyle changes, but they are held back by social norms.²

² A highly-cited paper by epidemiologist Geoffrey Rose (2001) argues that public health interventions should focus on community-wide interventions rather than just on high-risk individuals, in part because of peer influences. He says “Eating, smoking, exercise and all our other life-style characteristics are constrained by social norms. If we try to eat differently from our friends it will not only be inconvenient, but we risk being regarded as cranks or hypochondriacs.”

Application to Tobacco Taxes

Consumption of tobacco declined very slowly after the health effects were discovered. Deaths from lung cancer increased by a factor of 10 between 1925 and 1950, and by 1950 it was fairly clear that smoking was the overwhelming cause.³ Rates of smoking peaked around 1960 and have been declining by about 5%/year ever since, and are now about 1/3 of their peak. Thus most of the deaths from smoking have been among people who started smoking after it became clear that smoking caused cancer.

³ Doll and Bradford Hill (1950)

The decline in smoking was primarily due to a change in beliefs. DeCicca et al. (2022) document that in the US the tax rates on cigarettes stayed relatively constant between 1960 and 2022, and they say most studies estimate that cigarette demand is fairly insensitive to price (estimates of elasticity between -0.05 and -0.35). They also find relatively weak effects from other regulations, e.g. bans on advertising, bans on smoking in public places.

Thus the large aggregate decline in smoking seems to have been driven by a change in tastes, which presumably was downstream from a change in beliefs about health effects. That change in tastes has propagated very slowly presumably due to peer effects (AKA “norms” or “culture”).

In retrospect the decline was inefficiently slow: multiple generations got a taste for tobacco from their peers and their parents, and then found the habit difficult to shake. This implies an externality from smoking. In retrospect welfare would’ve been higher if we’d had high taxes in the 1960s to hasten the decline of smoking, and indeed in a simplified model it could’ve been Pareto improving: each person would lose utility from smoking less, but offset by a gain from their peers smoking less.

Suppose we come to believe that saturated fat or sugar has a comparable health effect to tobacco. Then in the long-run we expect their use will decline, but it might take 50 years for that to happen. Thus it would be efficient to apply a temporary tax to correct the peer-effect externality.

Evidence on Peer Effects

This section has a quick sketch of lines of evidence for the importance of peer effects in consumption decisions, but it is not as well organized as it could be.

Adult preferences reflect childhood exposure. Whether you prefer rice, bread, or potatoes; prefer olive oil or butter; prefer shoes on or off; prefer country or rock n roll; is mostly determined by which one you were brought up with. The cleanest evidence I know of is Atkin (2013) and Atkin (2015) who shows that Indians choice of food is highly sensitive to upbringing: those who are brought up eating rice prefer rice, those who are brough up eating wheat prefer wheat, and the effects are so strong that they contribute substantially to malnutrition.

There is more between-society variation in consumption than can be rationalized by between-society variation in prices. Northerners use butter, southerners use olive oil.

Changes in average consumption are primarily due to between-cohort changes, rather than within-cohort. E.g. DeCicca (2022) says that the decline in smoking in the 20th century was primarily due to lower rates of uptake by each new generation rather than an acceleration of the decline in smoking by existing generations.⁴

⁴ Note that this pattern could be rationalized either by (1) within-cohort peer effects, or (2) within-person intertemporal complementarities. However note that the existence of within-person intetemporal complementarities will tend to increase the persistence of contemporaneous peer effects.

Consumption is sensitive to changes in peer behaviour. If your friends cut back on smoking, drinking, and eating donuts, you’ll likely cut back too.

Long-run effects on culture

Nunn (2020) gives many examples of contemporary cultural traits which appear to reflect differences in historical environments that no longer exist today.

Becker (2020): There are large contemporary differences in culture between societies which live in lands well-suited for pastoralism vs agriculture.
Alesina et al. (2011): “the descendants of societies that traditionally practiced plough agriculture, today have lower rates of female participation in the workplace, in politics, and in entrepreneurial activities, as well as a greater prevalence of attitudes favoring gender inequality.”

However these types of argument are notoriously difficult to evaluate: identification requires assuming many other causal channels are zero, and researchers typically examine a lot of data before finding an association that could be publishable.

Evidence from Migration

Alesina and Guiliano (2013) say

“when immigrants move to a place with different institutions, overwhelmingly their cultural values change gradually, if ever, but rarely within two generations.

Evidence from Heritability

Twin studies show fairly high heritability of health behaviors. I give below some very rough estimates for the contribution of shared environment to adult health behaviors from twin studies. The shared environment estimate comes from the degree of correlation between fraternal twins (precisely: the excess relative to 50% of the correlation between identical twins).⁵

⁵ Vink et al. (2015) Heritability of smoking initiation and nicotine dependence – study of Dutch twins. They cite other studies with similar decomposition for smoking inititation. “individual differences in smoking initiation were explained by genetic (44%), shared environmental (51%) and unique environmental (5%).”

Verhulst (2015) The heritability of alcohol use disorders: a meta-analysis of twin and adoption studies: “The best-fit estimate of the heritability of AUD was 0.49 [95% confidence interval (CI) 0.43–0.53], and the proportion of shared environmental variance was 0.10 (95% CI 0.03–0.16).”

Maes (1997) Genetic and Environmental Factors in Relative Body Weight and Human Adiposity: “genetic factors explain 50 to 90% of the variance in BMI”. They don’t seem to give a preferred estimate for shared environment contribution, I’m going to say 10% based on skimming this and other papers.

	heritability ()	shared env ()	unique env ()
smoking initiation	45%	50%	5%
alcohol use disorder	50%	10%	40%
obesity	70%	10%	20%

However twin studies under-state the importance of peer effects. The “shared environment” will pick up the influence of parents and mutual friends of siblings, however the “unique environment” will also include the peer effects from each sibling’s idiosyncratic friendships.

More importantly, twin studies decompose the variance in behaviours for a given population (usually a single country) at a particular time. However for most of these behaviors the within-society variance is small relative to the between-society variance. Thus twin decompositions of variance will only pick up the contribution of local peer effects (your family or neighborhood), not society-wide peer effects.

Relation with Other Economics Literature

Economics literature on social norms mostly treats preferences as fixed. There is a lot of economic literature on “social norms”, and is applied to behaviors like generosity, bargaining, fertility, & medical treatment.⁶ However I believe the majority of this literature treats preferences as fixed so that norms are modelled as multiple equilibria (Schelling (1978), Axelrod (1986), Mackie (1996)), or with a raw preference to conform (e.g. Young, 2005), or a raw preference to sanction those who deviate from a norm. I model norms as essentially absorption of preferences. The key conceptual distinction is how your behavior changes when you change peer groups: purely conventional norms ought to adapt immediately – you drive on the other side of the road when you go to another country. But acquired preferences should remain: you still like the music, the food, the literature, of your home country.

⁶ Young (2005) is a good review.

Relation with the Becker-Murphy (1988) model of addiction. Becker and Murphy claim that consumption of addictive goods (tobacco, heroin) is consistent with rational choice given that there is an inter-temporal complementarity: consuming at time increases the value of consuming at . They say there’s evidence that people are clear-sighted about the complementarity: when people anticipate a cigarette tax in the future then they smoke less now.

Becker-Murphy is about within-person complementarities while the model in this note is about between-person complementarities. The between-person complementarities generate an externality, and therefore (unlike in Becker-Murphy) a reason to introduce a tax.

In addition it seems likely that a person’s earliest choices are partly naive about long-term effects: when a child or adolescent eats, smokes, or drinks alcohol, they are unlikely to fully anticipate the long-run consequences on their health or on their future tastes. If this is true this would be an additional reason (independent of peer effects) to tax or restrict youthful consumption.⁷

⁷ Becker (1992) is a long essay about habits and addiction that acknowledges the existence of acquired preferences though doesn’t seem to pursue the implications. Becker says “[a]ltruistic parents … may refrain from smoking even when that gives them much pleasure because their smoking raises the likelihood that the children will smoke.”.

Discussion

Concrete implications: (1) If you brought up a child in a world where nobody smoked, drank, or ate twinkies, then they would not lose much pleasure but they would live a substantially longer life – a kid who never acquires a sweet tooth will live a life both happy and healthy. (2) If we had banned cigarettes in the 1960s then one generation of smokers would have been unhappy but each subsequent generation would have grown up with no cravings or desire to smoke and substantially better health (smoking reduces life expectancy by 5 years).

Peer effects due to rational imitation. It can be rational to imitate what you see others do because their choices are informative about common costs and benefits (AKA “rational herding”, Banerjee (1992)).

However a basic model of rational herding would not generate the type of peer effects we see in practice. A pure informational peer effect implies that you should look at the behaviour of all peers not just nearby ones, e.g. you should be influenced by the worldwide participation in smoking instead of just in your local group. Also you should be most influenced by those who you believe to have the most information, e.g. the smoking rate among doctors or epidemiologists. In contrast peoples’ choices seem most influenced by their direct proximate peers.

If peer effects are due to rational herding then aggregate responses to changes in the environment will be sensitive to the nature of the change. E.g. when all prices are common knowledge then the aggregate and individual responses to a change will be the same. However when each person has some independent signal about the health effects of smoking then the aggregate effect of information can be far higher than the individual change.

Peer effects due to environmental spillovers. Peer effects in consumption can exist due to increasing returns in production: e.g. if most of your peers prefer X over Y then most shops will sell only X, and most social practices will accommodate X, make you more likely to consume X. If you move to a town where the majority smoke, drink, and eat unhealthy foods, then it becomes more difficult to live as a non-smoker, non-drinker, and healthy eater.⁸

⁸ Some theory and data from media markets is discussed in Anderson and Waldfogel (2005). There must be a literature on consumption spillovers in other markets but I haven’t seen it.

⁹ Galef (1998) discusses history and evidence for imitation. He seems to say that there’s been a lot of conceptual confusion about the definition of imitation and the evidence sufficient to demonstrate it: “[i]t is somewhat surprising that almost 100 years of study of social learning in animals has failed to produce a clear answer to the question of whether animals can in fact learn to do an act from seeing it done,”. There is also good discussion in the Wikipedia page for social learning in animals.

Peer effects due to a hardwired desire. Evolution may have given us a hard-wired preference to do what we observe others doing. Other animals seem to imitate each other: e.g. bird-song and food choices seem to be heavily influenced by observation of others.⁹ The reason we tend to influenced by peers might be for good informational reasons, as discussed above it’s generally reasonable to do what we observe others do. But these imitative instincts might be hard-wired, which would explain why we are influenced far more by near peers than by far peers. This is the basic perspective of evolutionary psychology: Tooby and Cosmides (1992) argue that humans are built with cognitive architecture that is adapative in the evolutionary environment although often maladaptive in the modern environment.

It mostly doesn’t matter why we have peer effects. Below I discuss some different reasons why our choices might be attracted towards the choices of our peers. However for some questions it doesn’t matter what the cause is: if we can measure the slope of the individual and aggregate demand curves then we can (1) make predictions about the effect of interventions; and (2) measure the welfare effect of different policies (using the area under the demand curve). This is less true for informational peer effects, in which case the effect of a change in environment will depend on whether it is common knowledge, and surplus cannot be identified with the area under the demand curve.

Q: How do peer effects interact with other externalities? Suppose there’s a $1 externality from consuming a good, then should we correct that with a $1 tax, given that we know the individual and aggregate elasticities will be different? There’s an argument that the tax should be larger than $1, because when you increase your consumption it will cause everyone else to increase their consumption too, so you cause more than $1 of harm.

Q: would cohort-specific taxes be optimal? New Zealand has a plan to continuously raise the minimum legal age for buying tobacco so that people born after 2008 would never be allowed to buy tobacco.¹⁰. Not clear to me whether the model gives an answer to this.

¹⁰ article

Models

Summary of Models

I describe three models of peer effects, each an extension of the last.

Static model. Implies (1) aggregate elasticity greater than individual elasticity; (2) decentralized decision-making will cause too much variation in behaviour relative to optimal.
Two-generation model. Suppose the second generation wishes to stay close to the first generation’s consumption. Then the first generation imposes an externality on the second generation, and they will fail to respond to anticipated future changes in price (or information).
Infinite generation model. We can compare the rate of adjustment chosen by agents (ignoring their externality on future generations) and the efficient rate of adjustment, and therefore the appropriate tax to restore efficiency.

To add:

An overlapping generations model.
A fixed-cost model. I believe that this peer-effects model is isomorphic to a model with spillovers due to production with fixed costs, i.e. a Hotelling monopolist who must choose a point on a line, as in Anderson and Waldfogel (2005). However I would like to confirm this.
Matrix of effects. I think you can characterize the theory with two derivatives: the externality and the strategic complementarity. I think it would be useful to give examples of the 9 permutations (pos/zero/negative; pos/zero/negative).

Model: Static Peer Effects

Summary. Suppose you buy at price , and utility depends both on distance from ideal-point, and distance from avg level of consumption, . Then we’ll see that:

Aggregate price elasticity will be higher than individual price elasticity because when one person moves it gives everyone else a reason to move.
Average consumption is the same comparing social planner to decentralized.
Welfare-optimal consumption would be more compressed then decentralized consumption, because when we inflict an externality whenever we depart from the mean.
Average consumption moves slowly. Suppose everyone’s ideal points changes due to new information about health but they take last period’s mean consumption as the reference point. Then the society’s consumption will not adjust all the way to the new equilibrium.

Each person will choose a level of consumption between their exogenous preference and the population-average consumption (represented by the dashed line).

Consumption with spillovers. Suppose you choose your level of consumption, , and utility depends both on distance from ideal-point and distance from avg level of consumption:

We get optimal consumption, a function of ideal-point, avg consumption, and price:

We get average consumption ():

Observations:

Average consumption will be more price-sensitive than individual consumption ().
The average level of consumption is independent of the strength of spillovers (): spillovers just compress the variance, they don’t change the average level of consumption.
Suppose your utility depends on prior aggregate consumption, then you get slow convergence to equilibrium (I derive a dynamic model below).

Finally we derive welfare-maximizing consumption:

The first-order condition for wil be:

So we have:

Observations:

The average consumption is the same, whether decentralized or welfare-maximizing, but with welfare-maximizing consumption everyone has moved closer to the mean.

Q: adding direct externalities? Suppose each unit of consumption inflicts $1 non-peer externality on others, e.g. smog, second-hand smoke, congestion, etc. Without peer effects then a $1 tax would appropriately correct for this. Would the efficient tax also be $1 in this model?

Extension: Direct Externalities

Now we will add a direct externality cost, e.g. representing second-hand smoke or fiscal externality (I believe the results would be the same for an internality, e.g. myopic consumption choice). We will simplify the utility function in some other ways: everyone has the same ideal-point (at zero), and we ignore price:

If there were no peer effects then everyone would simply set , but the welfare-maximizing choice would be to set , at which point the marginal personal cost of increasing consumption is equal to the marginal externality.

With peer effects the decentralized choice of will ignore the externality and will simply bias their choice of towards the population mean:

The unique equilibrium will be at , independent of the strength of peer effects.

Now we can derive the welfare-maximizing choice of with peer effects: The third term above (“marginal peer harm to others”) will always be equal to zero (it simplifies to ). T

Observations:

In equilibrium we can set and we will have , i.e. the welfare-maximizing choice is independent of the strength of peer effects, because they’ll all be zero whatever value you choose.
However if we hold fixed everyone else’s choice (take as given), then your choice effectively has two externalities: (1) the direct externality , (2) the .

Model: peer effects with two generations

For the dynamic models I assume peer effects operate solely through the prior generation’s actions: i.e. there’s some cost of departing from what your ancestors do. I assume there are no peer effects within a generation, and so can treat each generation as having just one agent.

Two generation model. Suppose the second generation gets disutility from departing from the first generation’s level of consumption (“habituation”):

Results:

The decentralized solution is inefficient. The first generation doesn’t care about the second generation, & so hurts the second generation.
The centralized solution would alter first generation’s consumption. We push the first generation a bit towards the second generation’s ideal-point. If we have a perfect ability to tax and rebate then we can implement the centralized solution with a tax.
Roughly: efficient tax would bring you 1/3 of the way towards long-run ideal point. Suppose the half-life of adjustment is 1 generation () and there is no discounting (). Then the first generation consumption should be 1/3 of the way towards the long-term ideal point.

Decentralized solution. The first generation just chooses their ideal point, the second generation is somewhere between the habituation point and their own ideal point. If then they’re half-way, meaning the half-life of adjustment is one generation.

Centralized solution. We now adjust the first-generation consumption towards the second-generation’s ideal point (derivation below):

Observations:

If no spillovers () then , .
If we don’t care about the future () then we get the decentralized solution again.

Derivation:

Model: peer effects with infinite generations

Peer effects will cause inefficiently slow adjustment to price changes.

We model an infinite series of agents. Each has an ideal-point of zero, but pays some adjustment cost for departing from the prior level of consumption (). Given some starting point () we wish to know how rapidly consumption will converge to its long-run equilibrium.

Each agent individually will simply bias their consumption towards last period’s consumption, and so the convergence factor will be simply :

However when maximizing social welfare we also need to take into account the effect on future utility. By taking the first-order condition with respect to we can derive a Euler-equation relationship between , , and :

If we conjecture exponential convergence, meaning , we can solve for the convergence factor :

Applying the quadratic formula we get a solution for :

Observations:

Simulation to confirm the solution. We let , implying a law of motion of , and as predicted declines by a factor of 0.38 each generation.

The solution seems to be irreducibly quadratic. If then we can get a slightly simpler expression for the relationship between and :
If (no discounting) and (habituation and ideal-point are equally strong):
- Decentralized convergence: will not be forward-looking, and so each period will be half-way between prior and the long-run equilibrium: .
- Centralized convergence:
  
  We choose the root that is below 1 (the other root would be explosive), and so , i.e. socially optimal convergence will be only somewhat faster than decentralized convergence in this case. If we add time-discounting I believe the difference between the two rates of convergence will get smaller.
If (infinite discounting) then , i.e. the centralized solution is the same as the decentralized solution, because we do not care about future generations.
If (no discounting) and (adjustment costs strong) then decentralized convergence will have and centralized convergence will have .

References

Allcott, Lockwood, Taubinsky (2019) “Should We Tax Sugar-Sweetened Beverages? An Overview of Theory and Evidence” in Journal of Economic Perspectives.

Anderson and Waldfogel (2005) “Preference Externalities in Media Markets” in Handbook of Media Economics

Atkin (2013, AER) Trade, Tastes and Nutrition in India**

Gives a model of habit formation, shows it fits Indian data, derives implications for trade policy.

“Household tastes evolve over time to favor foods consumed as a child.”

“households in the Indian state of West Bengal devoted 48 percent of their food expenditure to rice and 5 percent to wheat in 1987-88. Despite facing similar prices, households in the state of Rajasthan devoted 30 percent of their food expenditure to wheat and 1 percent to rice … If the average household in West Bengal allocated their expenditure on rice and wheat in the same proportion as households in Rajahstan, they would have obtained 23 percent more calories. … These unrealized nutritional gains are striking given that over 50 percent of children in West Bengal were classified as underweight around this time, and presumably these additional calories would have brought nutritional benefits”

Atkin (2015, AER) “The Caloric Costs of Culture: Evidence from Indian Migrants”

“I … show that migrants bring their origin-state food preferences with them.”

Berkowitz (2005) “An Overview of the Social Norms Approach”

Galef (1998) “Imitations in Animals” in Zentall and Galef (eds.) Social Learning Psychological and Biological Perspectives.

Nunn (2020) “History as Evolution” in the Handbook of Historical Economics.

Nunn (2022) “On the Causes and Consequences of Cross-Cultural Differences: An Economic Perspective”

Rose, Geoffrey (2001) “Sick individuals and sick populations”, International Journal of Epidemiology.

Tooby and Cosmides (1992) “The Psychological Foundations of Culture”

Citation

BibTeX citation:

@online{cunningham2024,
  author = {Cunningham, Tom},
  title = {Peer {Effects,} {Culture,} and {Taxes}},
  date = {2024-04-28},
  url = {tecunningham.github.io/posts/2023-01-23-peer-effects-norms-culture-sin-taxes.html},
  langid = {en}
}

For attribution, please cite this work as:

Cunningham, Tom. 2024. “Peer Effects, Culture, and Taxes.” April 28, 2024. tecunningham.github.io/posts/2023-01-23-peer-effects-norms-culture-sin-taxes.html.

Bloodhounds and Bulldogs

Tom Cunningham — Sat, 27 Apr 2024 07:00:00 GMT

This note contains some ideas about hierarchical structure in perception, judgment and decision-making that I haven’t seen explained clearly elsewhere.

1 Summary

A model of encapsulated inference (“bloodhound”). Many puzzling phenomena in perception, judgment, and decision-making can be explained if we assume that initial judgments are formed pre-consciously in a way that both (a) incorporates information not accessible to the conscious brain, and (b) does not incorporate information that is accessible to the conscious brain. This “double encapsulation” of judgment is a necessary condition for biases to occur in a model of the sequential aggregation of information. As an analogy we can think of someone tracking a scent with a bloodhound on a leash.Suppose the dog has private information, i.e. it know things that the human does not: this itself will not lead to inefficient decisions. However when the human also has private information, known only to her, then there can be inefficient decision-making relative to the benchmark of decisions based on pooling information between the person and the dog.
Applied to perception. Many existing explanation of perceptual biases and illusions can be stated in terms of sequential aggregation of information. A prototypical case occurs when a perceptual cue that is ordinarily informative appears in a context where we are consciously aware that it is uninformative. Because the informational value of the cue is known only to the pre-conscious system the cue will still affect our judgment even when we know it to be irrelevant. There is an additional set of biases in judging the strength of raw sensory stimuli, e.g. the light faling on your eye. If the conscious brain does not have direct access to those stimuli then it will be reverse-inferred from the signals received from the encapsulated processes, which will induce spillovers between signals and result in contrast effects and cross-modal effects.
Applied to judgment. The model says that biases in factual and logical judgments (anchoring, framing, etc.) occur in the same way as perceptual biases. A prototypical bias occurs when an ordinarily-informative cue appears in a situation where we know consciously that it is irrelevant. In short: it is rational to trust our instincts, even though our instincts can be sensitive to features of the environment that we know to be irrelevant.
Applied to decision-making. We can treat decision-making as a special case of judgment, but we are judging relative value of each alternative. As in judgment we rationally trust our intuitions of value, despite being uncertain about what influences them. This can cause inconsistencies in decision-making when we are in unusual contexts. Many people have attempted to formalize decision biases (ambiguity aversion, prospect theory, etc.) but I believe those formalizations have limited value because (1) the biases’ fundamental causes are statistical regularities in the environment, and so the biases will be context-dependent rather than universal; and (2) people are self-reflective about their biases and act to correct them, so treating them as hard-wired into utility or perception will give misleading predictions.
I discuss a little the history of the study of human judgment. There have been many dramatic shifts in paradigm and many laws have been proposed which later were found to generalize poorly outside their original domain. There is still today a lack of consensus around the basic reasons why human judgment is good in some domains, bad in others.
Observation: in most cases human judgment is a close approximation of efficient inference. Computers rarely outperform humans when given the same information. In the history of psychology many discoveries of laws in human judgment have turned out to reflect, on closer inspection, optimal responses to the environment. However there are some cases where judgment makes simple mistakes – illusions, biases, anomalies – and so some theory is needed to explain why those happen.
Encapsulated preferences (“bulldog”). I briefly discuss a separate feature of decision-making: the existence of encapsulated preferences, distinct from encapsulated information. Encapsulated preferences are hard-wired instincts that are rational with respect to a broader set of goals given a limited information-set. As an analogy think of a person with a bulldog on a leash who pulls you where they want to go: now the person is influenced by their dog in two ways: by the information received and by the dog’s own preferences.

2 History

The study of human judgment is intellectual quicksand. Many historical paradigms for studying human thought and decision-making have left little trace. Many smart people spent decades working on theories that seem, in retrospect, confused and unnecessarily complicated. There are still multiple schools of thought on most major questions. I believe the most significant progress it the general acceptance that human judgment typically corresponds to optimal inference, discussed further below.

Two factors that might explain the lack of progress: (1) scientists’ own introspection becomes distorted such that it confirms their own theories; (2) the language we use to talk about perception and judgment is highly ambiguous, and this makes it possible for theories to survive a long time, appearing to accommodate many facts, while being in practice unfalsifiable.

Patterns have often failed to generalize. There are many standardized experimental tasks used to study judgment and decision-making: choice between stimuli (psychophysics); the Stroop task; the implicit association task; the Wason card task; the Iowa gambling task; learning with reinforcement; choice between gambles; two-player games.

Many laws have been proposed based on patterns found in these tasks, but there has been a consistent failure for these patterns to generalize outside the specific tasks, and even within these tasks the laws have often been found to be sensitive to minor tweaks of context, e.g. instrumental conditioning, lateral inhibition, probability-weighting, loss-aversion, reciprocity. One interpretation is that behavior is very context-sensitive, and generalization is difficult. Another interpretation, which I think is the principal explanation, is that behavior is typically well-calibrated for the cues in a given situation, meaning that behavior in a given situation reveals more about the statistics of that situation than about the wiring of decision-making processes.¹

¹ I give a more detailed argument for this interpretation of the history of psychology in The Repeated Failure of the Laws of Behaviour, and in Relative Thinking.

3 A Model of Inference

Let us start with a model of judgment as rational inference. There is some unobserved fact about the world, , a set of observations which relate to that fact, , and an estimate formed by the brain using those observations:

A simple prediction of this model is that influences on judgment of should correspond to regularities in the world, i.e.:

Testing the model. The model’s predictions depend on the joint distribution of observed and unobserved characteristics in the decision-maker’s ordinary environment, . This distribution can be difficult to credibly measure, however there have been many cases where laboratory findings about judgment turn out to be neatly rationalized by correlations in the world:

influence on judgment	correlation in world
Blue objects look more distant	Distant objects are bluer (Rayleigh scattering)
Cold objects feel heavier	Heavier objects transfer more heat
People repeat rewarded actions	Rewards are correlated over time
Random rewards cause more persistence	Probability of reward is correlated

In each of these cases psychologists initially discovered an influence on judgment (), and proposed a mechanical explanation, e.g. based on how neurons are wired. Subsequently it was discovered that the influence on judgment explanations noticed that the influence could be rationalized by some correlation in the world. We can test these explanations by altering the context, finding a situation where we expect the real-world correlation between and to reverse, and we predict that the influence on judgment will reverse. Some examples of successful tests: (1) cold objects feel heavier, but hot objects feel heavier too, and this is consistent with the real-world association, because the effect of weight on heat-transfer depends on the difference from skin temperature. (2) when a rat finds a reward in a maze, it is then less likely to follow the same path subsequently (in contradiction to reinforcement learning), intuitively because they expect the food to be gone, i.e. they expect a negative correlation between past-rewards and future-rewards, instead of a positive one.

Feldman (2013):

“[Bayesian] optimality helps explain why the perceptual system, notwithstanding its many apparent quirks and special rules, works the way it does— because these rules approximate the Bayesian posterior.”

So far we have discussed evidence that perception and judgment are consistent with efficient use of evidence. However there are some patterns which resist rationalization, and for those we need a deeper theory.

4 A Model of Encapsulated Inference (“Bloodhound”)

The diagram shows a hierarchical model in which information is aggregated in stages, drawing on inputs that are not themselves directly accessible to the conscious brain.

The intermediate nodes can be thought of as automatic unconscious judgments – e.g. judgments about the distance of an object, the age of a person, or the value of a bottle of wine. The conscious brain only has access to those intermediate judgments, not the raw data, and it is this separation that can cause inconsistencies in judgment.

For simplicity I will assume just one encapsulated node () and two types of cue: “low level” (available only to the encapsulated node), and “high level” (available only to the conscious brain, ). A bias occurs when this sequential aggregation of information reaches a different conclusion than would occur if all information was pooled:

Roughly speaking a bias will occur only when 3 conditions are met:²

² The model of doubly-encapsulated inference is formalized in my paper “Hierarchical Aggregation of Information and Decision-Making”.

The encapsulated system receives information not expected by the conscious brain (),
The conscious brain receives information not expected by the early system (),
The two pieces of information interact, i.e. is non-separable in and .

What information is accessible to each stage?

There are two types of information that are likely private to early processing. I will give discuss evidence below that this information is mostly inaccessible to the conscious brain.

Raw sensory experience - the intensity of stimulation on the retina, the skin, the ear canal.
Associations between outcomes – e.g. the correlations between different cues which are used to make inferences.

The information private to the conscious brain is relatively abstract understanding of the situation.

As a metaphor we can think of a person leading a bloodhound on a leash. Both the person and the dog are trying to find the same object, but the dog’s private information is not accessible to the person except through how they pull on the leash, i.e. the dog’s interpretation of the information. In typical cases the encapsulation of information does not cause problems: the person simply follows where the dog leads. However when the person is aware of additional context important in interpreting the bloodhound’s own information (i.e. there is an interaction) then the person will make mistakes relative to what they would have known if they’d been aware of all the dog’s information.

The idea that judgment is encapsulated or modular is not new. It’s been argued by Helmholtz (1866), Pylyshyn (1980), Fodor (1983), Sloman (1996). However most discussion concentrates on the modules using a subset of the information available to the conscious brain, explaining biases in our automatic judgments, but not biases in conscious judgments. I put relatively more emphasis on the existence of information accessible only to the modules, which makes distinctive predictions about when reflective judgments will exhibit biases (I will call this “doubly encapsulated” processing of information).³ ⁴

⁴

Why would information be encapsulated? Mammalians brains have been evolving for 65M years while the human forebrain grew rapidly in the last 200,000 years. It seems likely that our systems for conscious reasoning are somewhat separated from systems for automatic judgment.

4.1 Implications of Encapsulated Inference

Here I givae the basic predictions of the model, in subsequent sections I discuss evidence relevant to these predictions from perception, judgment, and decision-making.

Judgment will degrade when presentation is unusual. We have very accurate judgments about cases which we have experience with but judgment degrades when some high-level information becomes relevant to interpretation. E.g., judgment will degrade when the same information is available but presented in an unusual way – upside-down, back to front, inverted.
Judgment will be sensitive to irrelevant features. Judgment will be sensitive to a cue which the subjects knows to be irrelevant when (a) that cue is usually relevant in similar situations; (b) the fact that the cue is irrelevant in this situation is high-level information, not accessible to the encapsulated system.
Judgment of hypotheticals will be poor. We will have poor ability to state how our judgment would change if one of the cues changed. E.g.: would this drawing look more like your cousin if the nostrils were more flared? Would you judge this candidate the same way if they were a woman as if they were a man? These are objective questions about your own judgments, but if judgment is encapsulated then people may give inaccurate answers to these questions.
Judgment will be sensitive to comparisons. When someone is judging two objects at the same time the conscious brain will have access to two encapsulated judgments, and therefore will learn something about what cues influences judgment. Thus we should expect to see systematic comparison effects which will reveal the nature of the information that is encapsulated. We also should see irrelevant-influences affect judgment in between-subjects, but not in within-subjects, experiments.
Persistence of biased intuitions. Even when we are consciously aware that an encapsulated judgment is incorrect the encapsulated system will still produce that inference, i.e. people retain a subjective perception which they know to be false (e.g., even after you learn that an illusion is misleading it still looks that way).

5 Perception

5.1 Basic facts

People are good judges of objects in the world.⁵ We are very good at judging the distance, size, or weight of an object, recognizing a face or a scene. Computers have only recently become comparable to humans after decades of dedicated work.

⁵ How various visual cues affect the judgment of distance.

People are poor judges of low-level perceptual stimuli. When asked to judge the magnitude of a raw sensation people are generally poor judges: e.g. comparing the brightness of two lights, the shade of two colours, the length of two lines. For computers these tasks are trivial.

There exist a large set of perceptual illusions. Psychologists have collected a diverse set of perceptual illusions in which people make predictable mistakes in reporting raw sensations (Muller-Lyer, Ebbinghaus, McGurk, etc.). There are still multiple active schools of thought in explaining most of these illusions.

5.2 Predictions of Encapsulated Inference

We can give a simple model of encapsulated inference:

Early perceptual processes infer real-world values () using information about raw sensations (), and about associations between sensations and value.
Biases in judgment about the world occur because the early processes do not have access to high-level information available to the conscious brain ().
Biases in judgment about raw stimuli occur because the conscious infers the value of stimuli from the outputs of the early processes .

It is useful to make a distinction between raw sensations and their causes in the world. The causes can then be subdivided into value and noise, where “value” represents something of interest to the organism.

sensation ()	value ()	noise ()
light on retina	reflectance of object	illumination
size on retina	size of object	distance of object
pressure on hand	weight of object	sensitivity of hand
oriention on retina	orientation in world	orientation of head
motion on retina	motion of object	motion of eye

This gives us a series of predictions:

Sensitivity to presentation. Peoples’ ability to judge and recognize stimuli are notably worse when the stimuli are presented in a way that we’re not accustomed to: e.g. upside down, inverted, tinted, or back to front.⁶

⁶ e.g. Valentine (1988), Galper (1970), Kemp et al (1990).

Influence of high-level context. It’s useful to distinguish three different aspects of high-level context, and the predictions of double encapsulations:

The double-encapsulation theory predicts that automatic judgments about the world are independent of contextual influences (), while final judgments of both are dependent (, ).

I believe that this is broadly consistent with the empirical evidence:

perceptual inferences are resistant to modulation by contextual information, except for attention effects,
considered judgments of both sensations and values are commonly influenced by contextual information.⁷.

⁷ Firestone & Scholl (2016, BBS)

⁸ Cross-modal influences are often been cited as evidence against the encapsulation of perception, because they show influences across perceptual areas. However the fact that they show systematic errors in assessment of sensations demonstrates the existence of vertical encapsulation (i.e. lack of direct access to raw stimuli), at the same time as they show that the limits of horizontal encapsulation.

Judgment of raw sensations. Because the conscious brain does not directly observe the raw stimuli, it must infer them from the encapsulated system’s outputs: and . This predicts cross-modal effects in judgment of sensations, where one sensation influences estimate of another sensation:⁸

This is consistent with many laboratory examples showing “cross-modal” effects, where one stimuli (e.g. auditory) is influenced by some other stimuli (e.g. visual).

Examples fitting this pattern: (1) the McGurk and anti-McGurk effects; (2) the Stroop effect; (3) confusing motion of beeps and flashes; (4) confusing numerosity of beeps and flashes; (5) phoneme restoration; (6) assimilation to expectations - proofreaders’ errors; (7) Simon interference - response location & stimulus location.

Contrast effects in shade, contrast, hue, and size. The central circle is identical in each case.

Application: contrast and assimilation effects. A common finding in perception is a contrast effect, where some quality appears less-intense when placed next to a more-intense neighbor, although we also observe assimilation effects (the opposite) in some circumstances. I argue that (1) contrast effects in real-world perception are due to rational inference; (2) contrast effects in raw sensory stimuli are byproducts of double-encapsulation.

Suppose there are two neighboring objects, each with unobserved value () and additive noise:

If everything is normally distributed then we have the following:

In words this implies we expect an “assimilation” effect when the value is more correlated than the noise, and a “contrast” effect when the noise is more correlated than the value.

The equation above describes rational inferences of the encapsulated system about real-world values, but we additionally observe that contrast and assimilation effects occur in judgment of raw sensations, and for that reason they can properly be called “biases” or “illusions” relative to the full-information case. If we assume the conscious brain infers the sensations from the posteriors of the encapsulated system (), then we get the same comparative static:

I believe this gives a good account of the existence of contrast and assimilation effects across a range of cases:

Adelson (1993) gives a series of examples demonstrating these effects in lightness illusions: contrast effects occur when context implies that noise is more correlated than value, and assimilation effects occur when context indicates that value is more correlated than noise.⁹
When judgment is affected by a dissimilar stimulus (cross-modal effects), contrast effects will tend to occur when the additional cue is associated with the stimulus, assimilation effects will occur when it is associated with the value.
These biases will tend to be smaller for more automatic responses - e.g. grasping responses - insofar as they receive signals prior to high-level processing.

⁹ Some papers also find Bayesian assimilation with very noisy stimuli, which I think is also consistent, but need more argument.

Reproduction. People find it easy to recognize patterns but hard to reproduce them, for example to paint a picture or to transcribe a melody. This is consistent with the model above insofar as the recognition is done pre-consciously. The model also predicts certain biases in reproduction, e.g. that people will tend to fail to account for shadows when they do a drawing, because they are inferring the raw stimuli from the inferred object , and the brain’s inference has discarded most of the information it considers to be noise, such as illumination.

Additional notes.

Differences in estimand. We have assumed that the encapsulated system is estimating the same quantity that the conscious brain is trying to estimate, . In many cases the conscious brain will want to estimate some other quantity, and this discrepancy will cause a distinct type of bias. I think this accounts for the size-weight bias in which people judge a larger object to be lighter, all else equal: if the encapsulated system is inferring the density of an object (from size and weight), and passing that estimate to the conscious brain, this would account for a negative effect of size on conscious estimates of weight.
Bimodal posteriors. When there are two plausible but distinct interpretations of a given stimulus, i.e. when posteriors are bimodal, then people often alternate between perceiving the two interpretations, e.g. the Necker cube and the black/gold dress. It’s interesting that the cognitive system seems incapable of representing bimodal posteriors directly.

6 Judgment

6.1 Basic Facts

There are a set of well-known judgment illusions. There are a large set of simple judgment questions which people give consistently wrong answers to.

Logical judgment: bat-and-ball, Linda, Monty Hall, mathematics problems on exames.
Factual judgment: anchoring, joint-separate inconsistencies.

There are many proposed explanations of these biases but there is still relatively little consensus.

People are generally poor at making statistical generalizations about features. A variety of laboratory tests give people a set of cases and ask them to either (a) judge the strength of correlation between two features, or (b) predict an unobserved feature of a case given the observed features. People are often much worse than simple computer programs at performing these tasks.

Judgment in some domains can be very accurate very accurate. People can learn extremely subtle judgments when playing chess and Go, such that computers have only been able to beat them relatively recently.¹⁰ Other domains are harder to benchmark but it seems fair to say that judgment can be finely calibrated in domains such as interpersonal emotional judgment, medical diagnosis, music or literature.¹¹

¹⁰ Playing a game could be seen as a combination of judgment and decision-making, but because every player shares essentially the same preferences (they prefer to win) it’s informative about judgment.

¹¹ Kahneman and Klein (2009) say that human judgment tends to be good when subjects have (i) a lot of experience, and (ii) quick feedback.

6.2 Judgment and Encapsulated Inference

The encapsulated inference model says that judgments are largely instinctual and we have limited introspection into what determines them, e.g. when we are guessing the price of a good, judging the likelihood of an event, judging the trustworthiness of a person, we rationally trust our instincts. This has a number of consequences.

Limited introspection. We have limited ability to explain our judgments, e.g. to formalize them in a way that a computer could reproduce them.¹² The history of human sciences is consistent with this: e.g. linguists have worked for centuries on finding the rules which make a sentence grammatical, formalizing knowledge that we all posess intuitively.

¹² There is a long debate on what evidence would establish unconscious knowledge. Experimentalists have shown that subjects can learn a pattern but have trouble describing it. Critics have said (a) the subjects may have learned a different pattern but with the same extension (Quine), or (b) the questions asked of subjects are insufficiently detailed. In my view the problem is in the design of the experiments: implicit knowledge can be defined behaviorally, by observing when low-level and high-level information fail to be integrated.

Hypothetical questions. The model predicts that people will be unable to accurately answer hypothetical questions about their own judgments, e.g.: Would you value this bottle of whisky the same if it had a different price on it? Would you like this house as much if it was a different colour?

Influence of irrelevant cues. A common finding is that judgment is influenced by irrelevant cues, i.e. when told to ignore some fact people will still be moved by it. This occurs in the model when (1) the cue is integrated into preconscious judgments, (2) the conscious brain does not know how heavily the cue is weighted, and so cannot account for it.

Effect of comparisons. When two judgments are being made at the same time – e.g., two items are being judged – then the conscious decision-maker will learn more about the unconscious knowledge. The patterns of comparison effects will reveal the nature of implicit knowledge.

Internal consistency. We should expect that judgment anomalies will disappear in within-subject studies because the decision-maker becomes aware of the inconsistency and adjusts their judgments.

Consistency of biases. A lot of academic literature tries to identify biases in judgment. In our analysis biases will be caused by correlations in the environment, and so we should expect them to vary or reverse from one environment to the next, and indeed we find that many biases reverse sign:

the “contrast effect” vs the “assimilation effect”
the “gambler’s fallacy” vs the “hot hand fallacy”
the “recency effect” vs “confirmation bias”
“overweighting of low probabilities” vs “neglect of rare events”

Effect of incentives. We should see that incentives don’t materially affect biases, beyond the point where they simply get the person to pay attention. This is in contrast to inattention-based theories.

Persistence of mistaken judgments. The encapsulated-inference theory predicts that after someone learns that a judgment is incorrect then the feeling will persist, because the encapsulated system operates independently of higher-level knowledge. Thus in many of the classic judgment illusions, even when you know the right answer the wrong answer still has some intuitive draw.¹³

¹³ Sloman (1996) described this as Criterion S in arguing for two systems of reasoning: “a reasoning problem satisfies Criterion S if it causes people to simultaneously believe two contradictory responses.”

6.3 Implications for Improving Judgments

Organize information in a way that helps your instincts to recognize patterns. For example (a) visualizing data so that visual-processing modules can be used to recognize patterns; (b) describe uncertainty in language that’s similar to the way that uncertainty is experienced: in terms of frequencies (“1 out of 5”) instead of probabilities (“20%”); (c) when looking for different interpretations of data, present it in an unfamiliar way in order to route around your unconscious inferences.
Remove information that you know to be irrelevant. Your instincts will pick up on all available cues, even irrelevant ones. Thus attempting to ignore cues is likely to be ineffective, it’s better to physically remove irrelevant information.
Ask yourself variations on the same question. – it will help to extract more information from your instincts.

7 Decision-Making [UNFINISHED]

7.1 Background

Motivating examples. Decision-making as a whole is an unwieldy topic, I’ll give some motivating examples focussed on economics:

Choosing which wine to buy for dinner.
Choosing how much of your income to spend vs save.
Choosing whether to move cities for a job.

We don’t know much about what influences decisions. There are two polar schools of thought: (1) that decisions maximize some objective outcomes; (2) that decisions are buffeted by all sorts of influences – context, custom, time of day. Concretely, given variation in choices (e.g. savings rate, hours worked, charitable giving, education), we can decompose into variation in (1) budgets, (2) consequentialist preferences, (3) other non-consequentialist influences. Perhaps 1/3 each.

I will treat decision-making as judgment of value. i.e., people judge , and choice is just selecting the outcome with the highest expected-value. This allows me to use the same framework as used in perception and judgment.

In decision-making there’s no objective standard. When discussing perception and judgment we can determine whether people are right or wrong, but it’s not the same with decision-making. Here we never observe the true value of things, we can only talk about inconsistencies among decisions, which itself requires making assumptions over what are reasonable preferences.

We have limited insight into our goals. I think we have limited insight both into ordinary value judgments (why do I prefer this coffee-cup to that one?), and into overarching goals (why do I marry? have children?). Philosophy, after thousands of years, still has not given us clarity regarding our tradeoffs between different ends (pleasure, moral imperatives and religious imperatives, providing for others, receiving love & receiving esteem).

Evolution must play some deep role in decision-making.¹⁴ As with other animals, humans’ decision-making must have been shaped to favor reproduction. However I believe there remains relatively little consensus on exactly how it affects decision-making, outside some specific areas.

¹⁴ I think there’s a common perception that although evolution must have importantly shaped human psychology, the evolutionary perspective has made few fundamental contributions to understanding psychology.

¹⁵ Some of this correlation can come from genetic effects or complementarity (when other people do X, then it’s in your interest to do X). However we see that when people migrate to a different culture in adulthood they retain a substantial part of their old preferences.

We must absorb most of our preferences.¹⁵ A large share of variation in decisions must be due to the preferences that we absorb when we grow up: taste in music, religion, political beliefs, career, who we marry, how many children to have, where to live, all must be heavily influenced by early experiences.

Preferences vs associations. Instead of estimating objective quantity, can estimate instrumental value, payoff, utility.

There are a set of known decision biases. There are some decision problems in which peoples’ choices consistently violate norms of rationality: Allais paradoxes, Ellsberg paradoxes, small-stakes risk-aversion, relative-thinking paradoxes, time inconsistency, anchoring effects, etc. There have been many attempts to fit these anomalies into generalizations about decision-making but there has been little consensus (e.g. with prospect theory, inattention, relative thinking).

7.2 Perspective from Encapsulated Inference

(1) Limited introspection. The model predicts that we have poor ability to introspect, e.g. to explain how different features influenced our decisions, or to predict what we would do in a different hypothetical situation.

(2) Sensitivity to associations. The evaluation of an outcome should be sensitive to whatever details are informative about value, and we still respond to those details when we’re aware that they are uninformative in the current situation, insofar as that awareness is not available to the encapsulated system. Thus we expect that people may have well-calibrated judgment for usual situations, but make bad or inconsistent decisions in unusual situations. This is my interpretation of many laboratory biases such as framing, anchoring, etc..

The decision-theory community has developed sophisticated logic to model many decision anomalies (ambiguity aversion, relative thinking, probability weighting). In my opinion most of these anomalies are due to associations which are very sensitive to context, and formal modelling is relatively unfruitful. The imperatives of publication have, I think, caused too much effort to be put in this direction: if anomalies are context-specific then research is less publishable, for that reason academics keep trying to come up with general theories, beyond the point of plausibility.¹⁶

¹⁶ Examples: (1) people avoid ambiguity not intrinsically, but because it’s associated with bad outcomes in certain classes of situation, and they seek out ambiguity in other situations; (2) people are influenced by the choice-set because it’s often informative about relative value, but the nature of the influence varies drastically between situations (discussion here).

¹⁷ More precisely: when stakes are high enough such that people no longer choose dominated alternatives, most of the between-person inconsistency in choices remains.

(3) Consistency within decisions. The model says that violations of rationality are mostly inadvertent. We therefore expect decisions to be consistent within situations, though they may be inconsistent between situations. This is consistent with laboratory evidence showing that people show large framing effects, but rarely choose dominated options.¹⁷

(4) Implicit knowledge is revealed in comparisons. ¹⁸ When two objects are evaluated side by side, that reveals some of the unconscious information to the conscious brain, and so we should expect systematic comparison effects. In choice, we can show that characteristic intransitivities reveal implicit knowledge, e.g. the intransitive cycle to the right (a “figure 8”) reveals an explicit preference for male over female, but an implicit preference for male over female. Similarly if we observe that some bundle of attributes is evaluated more highly when the comparison bundle becomes more similar in some respect, this reveals implicit knowledge about the attributes (either a positive implicit association about a shared attribute, or a negative implicit association about a non-shared attribute). Jon and I formalize this logic in our paper “Implicit Preferences”.

¹⁸

8 Encapsulated Preferences

¹⁹

Finally I introduce an additional model which I think is necessary to explain a lot of behaviours: encapsulated preferences, (as opposed to encapsulated information). As with the previous model I assume an informationally-encapsulated system, but now assume it has a direct effect on decision-making, e.g. as a simplification we could say the final decision is a weighted average of the output of the two systems:

We could describe these low-level preferences as hard-wired, meaning that they intend to achieve the same value as high-level judgments (), but they respond just to superficial stimuli, here . Put another way: we have an irreducible preference for characteristics which have historically been associated with value. In some cases that association is learned over the course of our life (learned preferences), in other cases it will be over evolutionary history (evolutionary preferences). This has a number of predictions:

Decisions fail to maximize value when the association between superficial stimuli () and value () is different from the historical association. This is a weaker condition than encapsulated inference, because here the high-level system cannot override the low-level inferences.
People will choose to avoid situations where low-level and high-level judgments conflict. This follows if the high-level system has the power to choose future situations: she’ll avoid situations in which she anticipates a divergence between the preference of the low-level and high-level system.

This is broadly consistent with some features of human behaviour:

We are tempted when in situations involving immediate benefits: food, sex, tiredness, addictive drugs. These can be interpreted as evolutionary fail-safes that exist to prevent conscious reasoning from doing too much harm. They are distinct from ordinary preferences in a utility function because they are only activated in certain contexts, and for this reason the conscious brain can work around them, just as an adult can work around a child by putting certain things out of view.
We have an aversion to things that are superficially unsafe. We dislike walking on a glass bridge even if we know it’s safe, we dislike drinking from vials marked “poison” even when we know it’s not poison, we dislike the smell of rotten meat even if we know there’s no danger. This has the same pattern as the examples above, but reversed.
Developmental pathologies are adaptive at an evolutionary timescale conditional on superficial information. There are certain diseases specific to civilization: myopia, allergies, obesity, crooked teeth. For each of these there are persuasive arguments that the developmental response is optimal given some superficial information about the situation, and so the pathology is caused by the divergence between evolutionary-scale associations and contemporary associations.
Nervous-system responses are optimal conditional on superficial information. Many bodily reactions are not under conscious control: heart-rate, sneezing, fever, pain, headaches. We can over-ride automatic responses, e.g. by taking medications, but we often don’t know whether it would be wise because we don’t have access to the private information used by the automatic system, and it’s resaonable to presume that its responses are optimal.

8.1 Additional Notes on Encapsulated Preferences

Puzzle: distortion of conscious reasoning. In many cases situations which activate a visceral temptation also seems to affect our conscious thoughts: we don’t just give in to temptations, we also rationalize it. In the dog metaphor it’s like having a very persuasive bulldog who can talk you into things. You might still avoid the situation but once in the situation you choose whole-heartedly. It’s unclear why we would reason in this way.

Evolved desires vs encapsulated desires. Should be careful not to confuse encapsulated preferences with a different relation, where evolution gives us a goal which we consciously strive for (eat sweet things, have sex) which are only proximal relative to the evolutionary goal of reproducing. Those proximal goals could still be ipmlemented in a single system which would make entirely consistent maximizing choices (but it would be maximizing an outcome that is only correlated with evolution’s goal).

Characteristic behaviors which reveal encapsulated preferences:

Preferences are stronger when made more salient. E.g., (1) more likely to choose a croissant when you smell it; (2) you’re scared by certain things only if you can see them, when giving blood you avoid looking at the needle; (3) you’re more likely to agree to have sex if you’re aroused. Just thinking about certain things can make us aroused or happy or scared, so we can strategically choose what to think about.
Making personal rules. We set a certain time to start & finish working; set a goal number of words to write each day; only getting ice-cream if you’ve been to the gym; setting a rule for how much money to save. Implies you don’t trust your future self, so you make up a rule, even though there’s no way to enforce it.
Strategic choices. We choose to avoid certain options. E.g., (1) buying a house to force yourself to save; (2) avoiding situations where you think you’ll be tempted. Formally you can call this “choice over choice sets”, and there’s a a fair amount of decision-theory on how multiple-selves can be identified if you can observe this kind of choice.
Effect of distraction. You are more likely to indulge in proximal goals when you are distracted - e.g. manipulation of cognitive load.
implicit preferences. More influenced by an attribute in less-direct choice sets.

Other variation in decision by context: Decisions made with salient outcomes, or abstract outcomes; decisions made under time pressure; decisions made with cognitive load; decisions made in advance; decisions over future choice sets; choices made in different moods, or after exerting willpower; decisions made directly or indirectly; from small or large choice sets; and attitudes which are expressed by involuntary responses (response time, skin conductance, pupil dilation).

8.2 Why would we have encapsulated preferences

Two types: (a) evolutionary proximal goals (sugar, sex, putrid smells); (b) learned proximal goals (coffee, wine, tobacco, learned food aversions).

Proximal preferences from evolution. A lot of human preferences make sense as proximal goals for achieving a long-run evolutionary purpose: when we get a kick out of doing X, it’s because doing X was selected for in our evolutionary history. These goals are in some sense hard-wired because people still do X even when it no longer makes any evolutionary sense to do it: (1) we enjoy sex even when it doesn’t lead to reproduction; (2) we like the taste of sugar even when we know it’s bad for us; (3) we get scared of snakes, spiders, of heights, of blood even when we see them behind glass - we are even scared of pictures of these things; (4) we are disgusted by smells associated with infection, even when we know there’s no risk of infection. (There are also arguments that some emotions - such as anger, jealousy, love - are hardwired for a slightly different reason - because they serve as commitment devices in social interaction - Trivers).

Why would these be hardwired? Or rather, instead of hardwiring preferences, why didn’t evolution hardwire beliefs? It would seem to be more efficient to just have innate knowledge: that snakes are dangerous, that sex leads to reproduction, that rotting meat is an infection risk. Then we could treat each of these things as means, not as ends. I guess there are two reasons: (1) perhaps it’s harder to preinstall knowledge than to preinstall preferences, because each brain grows differently; (2) perhaps it’s safer to preinstall preferences, to prevent them being overridden by a malfunctioning conscious brain which thinks it knows better.

Proximal preferences from learning. We often can explain peoples’ preference for X, because X has been associated with good outcomes in the past, yet people still choose X even when they are aware it no longer has those good associations. I.e., X has turned from a means into an end in itself. This can be described as a “habit”, “learned preference.” Some examples: (1) enjoying the smell of coffee, the taste of wine, or a cigarette, when those associations must be due principally to the psychoactive drugs that have been associated with them; (2) a learned phobia; (3) a learned food aversion, even when you know it’s irrelevant (e.g. chemotherapy patients develop aversions to whatever food they were eating during the chemotherapy).

It’s hard to answer why learned preferences should be hard-wired.

Different models which get at aspects of this type of behaviour: (1) a complementarity between a cue and consumption: the consumption becomes more valuable when exposed to an associated cue (Laibson); (2) a temptation cost - you have to pay a cost to not consume something when it is possible to choose it – (Gul and Pesendorfer) (3) preferences which depend on your state of arousal - e.g. willingness to be violent, have sex, changes preditably with context (Loewenstein on arousal).

9 Notes on Literature

Unconscious knowledge. I wrote a discussion of unconscious knowledge here. In short: there have been many proposals of how to identify unconscious knowledge or unconscious preferences over the last 150 years, but there remains little consensus.

Modularity / top-down influences. In the 1950s the “new look” school in psychology argued that peoples’ perceptions are distorted by their beliefs and expectations. There was then a reaction in the 80s and 90s: Fodor & Pylyshyn argued that perception is encapsulated, separated from expectations, & you need this to explain illusions. Both are right, and in fact two types of illusions from different types of private info. System 1’s private info causes theory-biased report of sensation. System 2’s private info causes expectations-ignorant perceptions.

Efficient coding, predictive coding, rational inattention. A general class of theories assumes that there is some kind of noise in the processing of information, so that the conscious brain makes inferences using a noisy signal about the stimuli . These models, like my model, assume an intermediary process, but one which does compression rather than inference, i.e. the signal minimizes error with respect to , instead of . I have further discussion elsewhere, but in short I think (1) efficient-coding theories do a poor job accounting for features of perception; (2) rational-inattention models do a poor job accounting for gross facts of judgment; and (3) it would be an inefficient design for the brain to pass compressed versions of raw stimuli (x), instead of inferences from those stimuli (v).²⁰

²⁰ I need to think more about the relationship with “predictive coding” theories (where only the delta from the expectation is sent). Friston’s “free energy” theory is related to these theories but I haven’t been able to properly understand it.

Citation

BibTeX citation:

@online{cunningham2024,
  author = {Cunningham, Tom},
  title = {Bloodhounds and {Bulldogs}},
  date = {2024-04-27},
  url = {https://tecunningham.github.io/posts/2023-10-24-manifesto-perception-judgment-decision-making.html},
  langid = {en}
}

For attribution, please cite this work as:

Cunningham, Tom. 2024. “Bloodhounds and Bulldogs.” April 27, 2024. https://tecunningham.github.io/posts/2023-10-24-manifesto-perception-judgment-decision-making.html.

The Influence of AI on Content Moderation and Communication

Tom Cunningham — Mon, 11 Dec 2023 08:00:00 GMT

Thanks to many comments, esp. Ravi Iyer, Sahar Massachi, Tal Yarkoni, Rafael Burde, Grady Ward, Ines Moreno de Barreda, and Daniel Quigley.

Summary

It is difficult to anticipate the effect of AI on online communication. AI models have already had big effects on content moderation but they are now starting to have effects on content production, e.g. through generating synthetic spam and deepfakes. How is the balance likely to play out?

This note discusses the likely effect of AI on online communication, summarized with a set of predictions:

The prevalence of policy-violating content on platforms will decline. Classifiers are approaching human-level performance and so the prevalence of policy-violating content (e.g. nudity, hate speech) should decline to very low rates. The same argument applies to censorship: governments will be able to near-perfectly identify messages that include a forbidden sentiment.
The prevalence of “context-specific” violating content will increase. This refers to content which appears to be non-violating to the average person but is interpreted in a policy-violating way by its intended audience.
The prevalence of “known violating” content will decrease. Content that is a match against databases of illegal sexual media (PhotoDNA), IP-protected content (ContentID), or terrorist recruitment content (GIFCT) will become less frequent because AI will make obfuscation harder (i.e. relatively benefit the platform over the obfuscator).
Platforms will not be able to identify bots from their behavior. Behavioral tests like CAPTCHAs will become ineffective against sophisticated actors and so platforms will have to rely relatively more on direct proofs of identity.
People will not be able to discriminate between real and fake media based only on the content. Since the invention of audio and visual recording in the 19th century the existence of a recording has been evidence for the event depicted having occurred. The evidentiary value of media will become much less strong because it will be impossible for a human to discriminate between real and fake.
Platforms will not be able to discriminate between real and fake media based only on the content. It seems unlikely that AI classifiers will be able to reliably detect AI-manipulated media.
Fake media will not have a substantial influence on politics. It has always been possible to forge documents but the influence of forgeries on politics has been limited. Documents have always been evaluted not just on their content but their provenance. Advances in the ability to forge media seem likely to be accompanied by advances in skepticism and greater reliance on provenance.
Computer-generated content will not be hyper-persuasive. LLMs with their current architecture are unlikely to produce text that is more persuasive than a skilled human could produce. Social scientists have warned about “hyper persuasive” technologies for a century and been wrong every time.
Communication will migrate further towards large closed platforms. The rise of synthetic content will make it harder for individuals to discriminate between reliable and unreliable information sources, and so will likely to increase the demand for intermediaries, which implies greater demand for verification and closed platforms.
Things will get weird. Our intuitions about meaning and interpretation are based on experience with human-created representations. When computers can generate representations then strange things are likely to happen.

A useful distinction: internal properties vs external properties. A common thread in this argument is a distinction between two types of property:

An “internal” property of a message is a function solely of the content, e.g. whether an image contains nudity, whether text contains hate speech, whether a joke is funny. These properties hold independent of any outside facts. AI classifiers are rapidly approaching human-level accuracy for these properties and this means that platforms (and governments) will be able to near-perfectly filter by internal properties even if content-producers have access to the same technology.
An “external” property of a message depends on some fact outside the message’s content: e.g. whether an image was computer-generated, whether a claim is true, whether a message came from a specific person. Platforms will get better at predicting external properties but they will be outpaced by motivated actors who can manipulate fakes until they become indistinguishable from genuine articles, and able to manipulate lies so they’re indistinguishable from the truth.

This note additionally has short appendices on the history of forged documents and on relevant forecasting questions from Metaculus.

What I don’t talk about. Two important topics that I do not discuss here: (1) super-intelligent AI and alignment; (2) social and cultural biases in AI.

Predictions

Background on AI. Before discussing the predictions I note a couple of facts about recent progress in computer abilities:

Computers surpassed human abililties at mathematical calculations in the mid 20th century.
Over the last 50 years computers have been steadily getting better at recognizing text and images, with particularly rapid progress since around 2012. Today computers can match the best-performing humans on many classification tasks.
Since around 2018 computers have additionally become good at synthesizing media, e.g. creating text, images, or video, to match a description.

The graph above from Kiela et al. (2023) shows the very recent history of computer performance: AI systems are regularly achieving human performance on benchmark tasks within a year of them being introduced, and so new benchmarks are being introduced more frequently.

The Prevalence of Policy-Violating Content Will Decline

All large internet platforms use automated systems to detect policy-violating content. All major platforms ban or suppress various types of content, e.g. hate speech, incitement to violence, nudity, graphic content. It has not been practical to have a human review each message because the platforms have a high volume of messages being sent with low latency. However automated systems have always been used: early systems simply checked for the appearance of prohibited words or matched against media databases, later systems used classifiers trained on human labels. See a brief history of automated content moderation here.

Simple classifiers have high offline accuracy. Simple classifiers which just look for the appearance of specific words are often useful, e.g. certain words and phrases are highly predictive of whether text would be labelled as “toxic” or “hate speech.” However this method has many false positives (Chen (2022)) and false negatives (Heiner (2022)).

Simple classifiers are easily evaded. It is typically easy to alter a violating message such that humans still think it is violating but the classifier does not. As a consequence the accuracy of these classifiers looks much higher offline than online, as users take steps to evade them.

Gröndahl et al. (2018) note that hate speech detectors can easily be fooled if you “insert typos, change word boundaries or add innocuous words.”
Han and Tsvetkov (2020) note that simple models are poor at detecting “veiled toxicity” which they define as including “codewords, novel forms of offense, and subtle and often unintentional manifestations of social bias such as microaggressions and condescension.”
A. Lees et al. (2021) note that simple models are poor at detecting “covert toxicity” which includes “types of toxicity that may not be immediately obvious. Covertly toxic comments may use obfuscation, code words, suggestive emojis, dark humor, or sarcasm …[and] [m]icroaggressions.” These papers evaluate models trained to identify context-independent toxicity, i.e. where the ground truth is human rating of the text alone without additional information on context or audience.

LLM-based classifiers are approaching human levels of performance. In August 2023 OpenAI described using GPT-4 as a content labeler (Weng, Goel, and Vallone (2023)) and said “[l]abeling quality by GPT-4 is similar to human moderators with light training … [h]owever, both are still overperformed by experienced, well-trained human moderators.”

LLM-based classifiers handle adversarial cases well. Google’s 2022 generation of text moderation models, which use transformer-based LLMs, are able to correctly classify many types of adversarial messages which are designed to evade simpler classifiers. A. W. Lees et al. (2022) say their classifier performs well against “code-switching, covert toxicity, emoji-based hate, human-readable obfuscation, [and] distribution shift.” Google’s 2023 generation spam classifier uses an embedding that is “robust against typos and character-level adversarial attacks” (Bursztein et al. (2023)).¹

¹ Arnaud Norman writes about how algorithms to scrape email addresses are often easy to evade, by adding special characters or other obfuscations, but that ChatGPT can straight-forwardly decode most such obfuscations.

Better classifiers will lower prevalence even if they are available to adversaries. Suppose an adversarial content-producer had access to the same classifier that was used by the platform. The produced could keep testing different variants of a violating post until they found a variant that was truly violating, but not identified as violating by the classifier, i.e. a false negative. However as the platform’s model becomes more accurate there will be fewer possible false positives, and so the task becomes relatively more time-consuming for the adversary, and thus we should expect prevalence to decline.

The prevalence of policy-violating content has declined dramatically. Meta reports that the prevalence of nudity, bullying, hate speech, and graphic content each declined by a factor of between 2 and 5 between 2017 and 2022, and that the share of identified-violating content that was first identified by an ML model (“proactive rate”) is approaching 100% for most categories. I think much of this decline can be attributed to improvements in the quality of classifiers.² Mark Zuckerberg has been making predictions for a long time that human raters could be substituted with AI. Although he was over-optimistic about the pace, I think he has been largely correct, e.g. in late 2018 he said “through the end of 2019, we expect to have trained our systems to proactively detect the vast majority of problematic content.”³

² It is important to remember that the “proactive rate” is the share of detected content that is detected by AI, the share of violating content that is detected by AI will certainly be significantly lower but is not generally reported. See Meta’s Community Standards report and my visualization of this data.

³ Zuckerberg, “A Blueprint for Content Governance and Enforcement”

Employment of human moderators will likely decline. As computer accuracy improves fewer messages will need to be escalated for human review, additionally fewer humans will be needed to label training data.

This prediction also applies to government monitoring and censorship. Many governments use some kind of automated scanning tools to intercept or censor messages based on their content, e.g. the US’s NSA and Cybserspace Administration of China. Better AI will allow these agencies to classify every post with reliability as high as if they had a human read each one, thus we should expect obfuscation will become a much less-effective workaround for censorship.

This prediction would fail if there were hard limits on the performance of AI. It’s conceivable that there are ways of obfuscating content that will remain difficult for an AI to identify for a long time. However even if LLMs cannot identify violating content in real-time it seems likely they could catch up quickly. Suppose humans invent new types of obfuscation, e.g. misspelling words in a particular way. An LLM which is continually trained on human-labeled samples could likely learn the pattern and thus force humans to continually adopt new patterns.

Prevalence will never decline to exactly zero because it’s inherently noisy. An AI model can never perfectly predict human-rater evaluation because humans are themselves noisy: there is both between-rater variation and within-rater variation in labelling for any given piece of content. Thus if the ground truth is human judgment then even an infallible classifier could not be used to drive prevalence all the way to zero.⁴

⁴ Strictly speaking: this will be true if no content has a probability of being labelled as positive by a human of exactly zero.

The Prevalence of Context-Specific Violations Will Increase

Some messages have a violating significance only to their intended audience. We can define a message as violating in one of two ways: (1) has a violating significance to the average person (average citizen or average user), or (2) has a violating significance to the intended audience of that message. I will define a “contextual violation” as a message that is violating to its intended audience but not to the average person. This is stronger than just having a double meaning where both meanings are clear to all audiences. I am specifically talking about messages which are interpreted in distinct ways by different audiences. Of course contextual violations are often unstable, over time the average person will often learn the contextual meaning.

Many messages use contextual violations.⁵

⁵ A related phenomena is people using selective truths to give an impression that is false. E.g. it is common for anti-vaccination groups to post mainly true claims, but only those claims which reflect badly on vaccines. People with a bias against some ethnic group likewise often refrain from posting provably false claims but post only those true claims that reflect badly on the disliked group. Because the pool of claims that are true is enormous it is easy to collect a large set of true claims that collectively give a false impression.

Saying “globalist” when your audience understands it to mean “jewish”
Saying the opposite of what is meant, e.g. a bigot saying excessively positive things about an ethnic group, or a pro-anorexia poster making anti-anorexic statements sarcastically.
Using euphemisms for illegal substances or illegal acts.
Using emojis of eggplants and peaches with sexual connotations.
Using photos without explicit nudity but which can be read as pornographic.

Improved detection of violations is likely to cause substitution towards contextual violations. As AI improves the ability to detect violations it seems likely that there will be at least some substitution towards context-specific violations, however as long as there is some cost to using a contextual-violation then we would expect a less than one-for-one substitution.

Platforms could detect contextual violations if they wanted to. When doing human evaluation then platforms could either (1) provide human raters with detail about the message’s context and audience, or (2) assign human raters to messages based on their experience with that community.⁶ Likewise AI models could be trained to include rich representation of the context. An additional advantage of adding context is that it can identify and exempt posts that violate the letter but not the spirit of the policy.

⁶ Platforms already have some policies that include context, e.g. Facebook’s “Bullying and Harassment policy” bans “repeatedly contacting someone in a manner that is unwanted or sexually harassing.”

Platforms may not want to remove contextual violations. There are reasons why platforms may be reluctant to use context in determining violations: it is more complex, and can lead to awkward PR where the platform is shown to be censoring words and images have a harmless interpretation. Additionally platforms may care more about being seen to restrict harmful content than about the actual harm prevented.

Contextual violations have long existed in broadcast media. There have been many cases where contextual violations have been tolerated: e.g. newspapers would allow classified advertisments for prostitutes if described as masseuses, vibrators if described as massage wands, contraception if described as marital aids, and abortion if described as “removal of obstructions”. Thus it seems plausible that platforms will tolerate a substantial amount of contextually-violating content to remain.⁷

⁷ In Facebook’s Marketplace it is prohibited to list guns for sale. As a consequence people began to list gun cases, with the understanding that a case was standing in for a gun. Facebook then updated their policy to prohibit selling gun cases. In turn people began to list gun stickers as stand-ins for guns. See WSJ reports from 2020 and 2021.

⁸ https://www.axios.com/2022/11/28/china-protests-blank-paper-covid

Government censorship is unlikely to be constrained by context-specific violations. Once a censor discovers that a term has an anti-government significance in a certain context then they are likely to start censoring that term. E.g. China has suppressed online mentions of Winnie the Pooh because it is associated with criticism of Xi Jinping, and in 2022 Hong Kong police arrested protestors for holding blank pieces of paper.⁸

The Prevalence of Variants of Known-Violating Content Will Decline

Platforms typically check content against databases of known-violating content. In addition to running classifiers on content platforms also check content against databases of known-violating content. The databases are often shared across platforms, known as “signal sharing”, e.g. databases of illegal sexual media (PhotoDNA), IP-protected content (Content ID), or terrorist recruitment content (GIFCT).⁹ As a consequence sophisticated uploaders often obfuscate their content, e.g. by adding noise, and platforms expand their matching algorithms using fuzzy matching.

⁹ Other signal sharing programs: National Center for Missing & Exploited Children Child Sexual Abuse Material (NCMEC-CSAM), Non-consensual Intimiate Imagery (StopNCII), ThreatExchange.

Gratuitous violations with known-violating content will go to zero. Suppose someone wants to violate the policy just for the sake of violating that policy, e.g. they want to show a shocking image. Call this “gratuitous” violations. Currently the easiest way to do this is to first find a violating piece of content, then obfuscate it. If attackers can use AI synthesis they no longer need to find existing violating content, they can synthesize new ones. The defensive technique of checking against known-violating content becomes much worse. However if defenders have AI recognition, then by the same argument as above prevalence will go to zero.

Platforms Will Not Be Able to Identify Bots from Their Behavior

Most online platforms struggle with automated users (bots) who are disruptive in a variety of ways. One way of protecting against bots is with behavioral tests, e.g. a CAPTCHA test asking users an image-recognition task¹⁰, or by using on-platform behavior to detect whether a user is human. However improvements in AI mean that computers have human-level performance on image-recognition tasks, and can learn to imitate human-style behavior patterns, thus it seems likely these behavioral tests will become ineffective against sophisticated actors. Searles et al. (2023) finds that most contemporary CAPTCHAs can be solved by computers with higher-than-human accuracy (p10).¹¹

¹⁰ CAPTCHA stands for Completely Automated Public Turing test to tell Computers and Humans Apart.

¹¹ Similarly behavioural fingerprinting will become ineffective against advanced actors, e.g. using voice recognition to verify identity.

¹² The 3rd-party identity providers will themselves have to rely on some other ground truth when accepting signups.

This does not imply that the prevalence of bots will increase. All platforms need some defense against bots so they will have to rely relatively more on other forms of authentication, such as monetary payment, offline identity credentials (government ID, credit card number), hard-to-fake metadata (unique IP address, device ID), or 3rd-party identity provider (Sign in with Google, OpenID).¹² Thus the barriers to signing up for a service, and especially posting on it, will become higher, but the effect on equilibrium prevalence of bots is ambiguous.

Platforms Will Find It Hard to Discriminate between Real and Fake Media

In some cases the ground truth depends on properties outside the content. I will refer to these properties as “external” in contrast to “internal” properties which depend only on the content such as whether a picture depicts nudity. I discuss the distinction at greater length below. Some examples of external properties:

Whether a piece of media was generated in the traditional way (photographing a scene, recording a sound), or has been manipulated or synthesized.
Whether text was written by a human.
Whether text was written by a specific person, e.g. by Shakespeare.

Advances in AI will help with both forgery-detection and forgery-creation. It is clear that a better statistical model of genuine artefacts will help detect forgeries but it will also help create convincing forgeries.

Determined forgers will be able to fool humans. It seems likely that the latter effect will dominate: it will gradually become possible to camouflage computer-generated content such that neither a computer nor a human could tell them apart. If the content-producer has access to the platforms’ model then they can keep perturbing their fake media until it is labelled as non-fake.

We cannot reliably discriminate between real and AI-generated media. As of late 2023, programs to detect synthetically generated media have relatively poor accuracy: OpenAI announced a model to detect LLM-created text in January 2023 but then shut it down in July because of poor performance. In June 2023 the NY Times compared a variety of tools to detect computer-generated images and found that with minimal effort they could all be reliably fooled.

The prevalence of synthetic media will increase on unmoderated platforms. The major platforms have incentives to limit the prevalence of fake media,¹³ and can control the prevalence even without reliable classifiers. E.g. Meta and YouTube dramatically decreased the prevalence of misinformation over 2016-2020 not primarily through real-time detection of whether a given claim is false, but by (1) adjusting ranking to penalize publishers who tend to circulate false claims; (2) punishing publishers who circulate proven-false claims. Thus I do not expect overall prevalence of fake factual media to substantially increase on the major platforms.

¹³ The goals of platforms in content moderation are discussed in my note on ranking, Cunningham (2023).

Fake Media (Deepfakes) Will Not Have a Substantial Influence on Politics

As synthetic media becomes common people will rely more on provenance. As it becomes cheaper to manipulate and synthesize media then people are likely to become more skeptical and rely relatively more on the provenance of information. Thus although synthetic media will likely circulate I do not think it will have a substantial influence on beliefs in equilibrium.

It has always been easy to create misleading documents.It is not difficult to forge or alter documents, or edit video in a misleading way. As a consequence mainstream media organizations typically do not publish leaked materials unless they have either a chain or provenance for the leaks or independent confirmation of their content.

Influential forgeries of documents have been historically rare. In an Appendix below I compile a simple dataset of politically influential document leaks in the US over the past 25 years and estimate around 10% of them were based on forged materials.¹⁴¹⁵

¹⁴ I know of two forged documents that were widely taken as true in the last 25 years, from around 15 substantial leaks that I could find: (1) the “yellowcake” letters from Iraq to Niger, cited in the 2002 US case for war against Iraq; (2) a fake G W Bush military transcript reported on by CBS and Dan Rather in 2004. It’s notable both that these cases are somewhat rare, and that each was passed through a chain of apparently reputable parties.

¹⁵ This argument implies that, prior to AI, anonymously leaked video would be more likely to be published and circulated than anonymously leaked documents, because video is harder to fake. In fact I cannot think of many cases of influential anonymous leaks of videos. When Trump’s “Access Hollywood” tape was leaked to the Washington Post they got confirmation before publishing it. In fact maybe leaked video has always been untrustworthy because it has always been easy to make deceptive edits.

¹⁶ Snopes.com has an enormous database of both false claims and misleadingly manipulated media that has circulated since 1994. A typical recent example is an edit of a Bill Gates interview to make it appear he wants to use vaccination to reduce population growth.

The quantity of false claims circulating on the internet is not primarily constrained by the quality of their content. A great deal of false claims already circulate on the internet, especially in loosely moderated parts: e.g. by email, on Telegram, 4chan, Truth Social, WhatsApp, Twitter. It’s not clear that the quality of the faked media is an important constraint on the volume that circulates. It’s not uncommon to find a clip of an interview with a politician edited to make it appear that they are admitting to a crime or secret agenda.¹⁶ If people already take what they see at face value then adding deepfakes seems unlikely to change their opinions substantially. Alternatively if people are skeptical and look for corroborating sources then, again, deepfakes would be unpersuasive. It seems that deepfakes would only be influential if there are a significant population who are exposed to many lies but are not haded because the documentary evidence is not sufficiently strong.

Communication Will Migrate Towards Large Closed Platforms

Small platforms will be overrun with AI-created content. In particular, AI-created bots, spam, obfuscated violating content, and fake media. This would imply that consumers will tend to migrate to larger closed platforms with more effective defences, and which have more restriction on participation. This continues a general movement over the last 20 years of communication moving from small open platforms (independent email, small forums, mailing lists, independent websites) to large closed platforms (large email providers, large social media platforms).

People will rely more on established sources of truth. E.g. they will rely relatively more on Wikipedia, Community Notes, and mainstream recognized media sources. The ordinary content-based signs of trustworthiness will become less reliable: having a professional website, well-edited text, well-argued reasoning, and documentary evidence.

People will rely more on cryptographic signing to verify authenticity. I am not sure how strong this effect will be: it is generally more efficient for an intermediary to verify authenticity of senders than for users to do it themselves. I think we’ve seen that in other domains: (1) PGP signing of email has been less important than email providers filtering spam and phishing; (2) SSL certificates in browsers have been less important than browsers giving warnings for suspected phishing sites (e.g. Google’s safe browsing database of sites with phishing or malware is used to give warnings in Chrome and Safari).

Pedigree will become more important in publication. As an editor accepting submissions (e.g. an academic journal, a literary magazine, a newspaper letters page) the quality of the work submitted is typically correlated with more superficial features such as the grammaticallity and the length. As it becomes easy to synthesize text then those superficial features will become less informative about quality and editors are likely to rely relatively more on hard-to-fake signals like the pedigree of authors: what have they published before, and which college the author went to.

Entertainment will Become Largely Synthetic

A classifier that can detect whether a photo is pretty can also generate a synthetic photo that is pretty, and a classifier that can detect whether a joke is funny should also be able to generate funny jokes.¹⁷ On average people spend around 3 hours per day watching entertainment (TV, YouTube, TikTok, Instagram). It seems likely that trained models will be able to synthesize content that is highly engaging though it’s hard to anticipate what it will look like.

¹⁷ I think language models haven’t yet been very good at jokes because they generate one word at a time (autoregressive), while jokes typically have a logical structure such that the setup is probable given the punchline, but not the other way around. When we get language models which generate text using different statistical algorithms (e.g. diffusion instead of autoregressive generation) then it seems likely they’ll be able to create good jokes.

Things Will Get Weird

Much of our common-sense understanding of media will be violated when we routinely use AI models to manipulate and synthesize artefacts. Some examples:

People will synthesize completely new violating images/videos. Thiel, Stroebel, and Portnoff (2023) say that, as of early 2023, less than 1% of child sexual abuse media (CSAM) appears to be synthetically generated. However the ability to synthesize has been advancing rapidly, “to the point that some images are only distinguishable from reality if the viewer is very familiar with photography, lighting and the characteristics of diffusion model outputs … it is likely that in under a year it will become significantly easier to generate adult images that are indistinguishable from actual images.”
Producers will synthesize content to sit on the edge of a category. If platforms take action whenever content passes some threshold then adversarial actors will generate or perturb content such that it sits right below the threshold. If a platform removes a photo whenever more than 50% of raters would say it depicts nudity then producers would upload photos which 49% of raters would say depicts nudity. People would upload movies which almost look like an existing IP-protected movie, and students might submit essays that are close to existing sources but don’t quite trigger the plagiarism detector.

Appendix: Historical Observations on Forgeries

Influential leaks of US political documents since 1997:

Tripp Tapes	1997	audio	Linda Tripp to Kenneth Starr
[FORGERY] Iraq letters to Niger (“yellowcake”)	2002	documents	Unknown to Italian intelligence to CIA
[FORGERY] Bush military transcripts (“Killian”)	2004	fax of 1970s memo	Unknown to retired colonel to Dan Rather / CBS
Abu Ghraib photos	2004	photos	Unkown to CBS
Baghdad Airstrike (“Collateral Murder”)	2007	video	Chelsea Manning to Wikileaks
US Iraq war logs	2010	digital docs	Chelsea Manning to Wikileaks
US Diplomatic cables	2010	digital docs	Chelsea Manning to Wikileaks
Romney Fundraiser Tape (“47%”)	2012	audio	Bartender to Mother Jones
NSA Surveillance Leaks	2013	digital docs	Edward Snowden to the Guardian, WaPo
DNC emails	2016	emails	Unknown to Wikileaks
Podesta emails	2016	emails	Unknown to Wikileaks
Colin Powell emails	2016	emails	Unknown to DCLeaks
Panama papers	2016	documents	Unknown to Süddeutsche Zeitung
Donald Trump Access Hollywood Tape	2016	video	Unknown to Washington Post
China Cables	2019	digital docs	Unknown to the ICIJ
Hunter Biden laptop	2020	docs,audio,video	computer shop to Giuliani to NY Post
Los Angeles Council call (“changuito”)	2022	audio	Unknown to Reddit to LA Times

Why are forgeries not more common? I can think of three possible reasons:

It’s difficult to forge credible documents – e.g. even a simple memo is hard to fake because there are lots of small details like the letterforms and formatting and jargon used.
It’s easy to forge credible documents but intelligence agencies and the media won’t believe them without independent confirmation.
It’s easy to forge credible documents and for them to get coverage but not many people are motivated to try.

Other examples of influential forgeries.

Mark Antony’s will	33BC	read out by Octavian in the senate (disputed whether forgery)
Dreyfus letters sharing military info w Germany	1894	fabricated by French military
Protocols of Elders of Zion (Jewish plans for domination)	1903
Castle Document, letter to British govt in Dublin	1916	unclear source, contributed to 1916 Easter rising
Zinoviev letter from Russia to UK Labour party	1924	unclear source
Tanaka Memorial (Japanese plans for world domination)	1929	unclear source
Macron emails	2017	emails

Notes.

The Macron email leaks seemed to include both real and fake content (ref). In fact one report says that Macron’s team sent each other outrageous implausible emails as a pre-emptive defense to make any subsequent leaks seem less credible (ref).
The Steele dossier isn’t really a forgery: it doesn’t purport to have a different author than its true author. The problem with the dossier is that the author makes knowingly false claims of fact.

Appendix: Relevant Forecasts from Metaculus

	deadline	Metaculus Jun 30 2023
Will a deepfake cause damage & make front page of a major news source?	2023	89%
Will a deepfake be blamed by G20 politician for election loss?	2025	80%
Will AI be used in an attack on infrastructure costing >$1B?	2025	4%
Will AI be used in a theft of intellectual property cost >$10M?	2025	30%
Will AI cause a stock exchange to halt trading for >24 hours?	2025	15%
Will AI be used in a major attack on voting systems in G20?	2025	10%

Will a deepfake about politicial running for office get 2M+ views?	2018	(resolved false)
Will a wide-scale video hoax put words in a famous figure’s mouth?	2017	(resolved false

References

Bursztein, Elie, Marina Zhang, Owen Vallis, Xinyu Jia, and Alexey Kurakin. 2023. “RETVec: Resilient and Efficient Text Vectorizer.” https://arxiv.org/pdf/2302.09207.pdf.

Chen, Edwin. 2022. https://www.surgehq.ai/blog/are-popular-toxicity-models-simply-profanity-detectors.

Cunningham, Tom. 2023. “Ranking by Engagement.” http://tecunningham.github.io/2023-04-28-ranking-by-engagement.html.

Gröndahl, Tommi, Luca Pajola, Mika Juuti, Mauro Conti, and N. Asokan. 2018. “All You Need Is "Love": Evading Hate-Speech Detection.” https://arxiv.org/pdf/1808.09115.pdf.

Han, Xiaochuang, and Yulia Tsvetkov. 2020. “Fortifying Toxic Speech Detectors Against Veiled Toxicity.” In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 7732–39. Online: Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.622.

Heiner, Scott. 2022. “Real-World ML Failures: The Violence, Racism, and Sexism Uncaught by Twitter’s Content Moderation Systems.”

Kiela, Douwe, Tristan Thrush, Kawin Ethayarajh, and Amanpreet Singh. 2023. “Plotting Progress in AI.” Contextual AI Blog. https://contextual.ai/blog/plotting-progress-in-ai.

Lees, Alyssa Whitlock, Vinh Q. Tran, Yi Tay, Jeffrey Scott Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman. 2022. “A New Generation of Perspective API: Efficient Multilingual Character-Level Transformers.” In. https://dl.acm.org/doi/10.1145/3534678.3539147.

Lees, Alyssa, Daniel Borkan, Ian Kivlichan, Jorge Nario, and Tesh Goyal. 2021. “Capturing Covertly Toxic Speech via Crowdsourcing.” In Proceedings of the First Workshop on Bridging Human–Computer Interaction and Natural Language Processing, 14–20. Online: Association for Computational Linguistics. https://aclanthology.org/2021.hcinlp-1.3.

Searles, Andrew, Yoshimichi Nakatsuka, Ercan Ozturk, Andrew Paverd, Gene Tsudik, and Ai Enkoji. 2023. “An Empirical Study & Evaluation of Modern CAPTCHAs.” https://arxiv.org/pdf/2307.12108.pdf.

Thiel, David, Melissa Stroebel, and Rebecca Portnoff. 2023. “Generative ML and CSAM: Implications and Mitigations.” https://www.semanticscholar.org/paper/bbcbaedfe893f9cd5d6390adf616cecdabfc651d.

Weng, Lilian, Vik Goel, and Andrea Vallone. 2023. “Using GPT-4 for Content Moderation.” https://openai.com/blog/using-gpt-4-for-content-moderation.

Citation

BibTeX citation:

@online{cunningham2023,
  author = {Cunningham, Tom},
  title = {The {Influence} of {AI} on {Content} {Moderation} and
    {Communication}},
  date = {2023-12-11},
  url = {https://tecunningham.github.io/posts/2023-06-06-effect-of-ai-on-communication.html},
  langid = {en}
}

For attribution, please cite this work as:

Cunningham, Tom. 2023. “The Influence of AI on Content Moderation and Communication.” December 11, 2023. https://tecunningham.github.io/posts/2023-06-06-effect-of-ai-on-communication.html.

The History of Automated Text Moderation

Sat, 18 Nov 2023 08:00:00 GMT

This document describes five technologies for automated text moderation, each roughly correspond to an historical phase.

As a working example we will use the detection of “toxic” comments. In practice many different definitions of “toxic” have been used in the industry, and there are a variety of related concepts, e.g. “hate speech” and “offensive”.

(1) Keywords

The simplest technology is to hard-code a list of words which are considered “toxic”, e.g. a list of curse words. This can be implemented with regular expression. This has obvious limits on the accuracy and cannot be easily maintained, however many platforms still maintain a keyword block list for some sensitive terms.

(2) Simple classifier (“Bag of words”)

We can collect a large set of human-labeled data on whether individual messages are toxic, and then predict toxicity from the appearance of individual words e.g. using logistic regression or naive Bayes. These classifiers will find that certain words are highly predictive of toxicity. Simple classifiers often have reasonable accuracy but will have many important false positives and false negatives, and they are easy to evade by rewording or misspelling text.

1961: Maron (1961) proposes the Naive Bayes classifier

(3) Embedding-based classifier (2013-2018)

These models have two stages:

Pretrain: for each word calculate an embedding (a vector of numbers) which predicts its likelihood of co-occurring with other words. Pairs of words which are nearby in embedding-space typically have similar meanings.
Train: train a model to predict toxicity of a comment using the embedding of the words in a message (e.g. the average embedding).

An advantage over simple classifiers is that these models require much less labeled data for an equal performance, because the pre-training stage has already learned (crudely) the meanings of different words. Thus these models can identify words that are diagnostic of toxicity even if they never appeared in the toxicity training set.

However embedding-based classifiers are still bad at edge cases, e.g. when a word is used inside a negation (“is an idiot” vs “is not an idiot”), or if a word is mis-spelt, or if harmless words are used to express an meaning that is toxic (“your brain is a bowl of jello”).

2013: Word2Vec: a word embedding using a 2-layer neural network, (Mikolov et al. (2013))
2014: GloVe: Global Vectors for Word Representation. They say “training is performed on aggregated global word-word co-occurrence statistics from a corpus” (Pennington, Socher, and Manning (2014)).
2015: fastText: word embedding from FAIR. They released pre-trained models for 294 languages (Joulin et al. (2016))
2017: Jigsaw Perspective Toxicity API v1 from Google.¹

¹ I couldn’t find any authoritative documentation on the architecture of this classifier: I found one reference to it using the GloVe embeddings.

(4) LLM-based classifiers (2018-2023)

These models have three stages:

Embedding: Compute embedding of each token (a token is roughly equal to a word).
Pretrain: Train a deep neural net to predict a token from surrounding tokens (or prior tokens), using attention (i.e. don’t weight all words equally) on an enormous training set of text from books and the internet.
Train: Train a model to predict toxicity from labeled data using the top-level neurons in the net as features.

Conceptually these are similar to embeddings but (1) they can represent the meaning of entire sentences instead of just words, (2) have more layers so tend to have more sophisticated representations of meaning.

2017: Transformer architecture (Vaswani et al. (2017))
2018: BERT transformer LLM, this model has been widely used as base model for a variety of natural language tasks, including content moderation (Devlin et al. (2018))

(5) Zero-shot LLMs (2023-)

These models have three stages:

Embedding: Compute the embedding of each token.
Pretrain: Train a deep net to predict the next token from previous tokens, as above.
Directly ask the model whether a given message violates a given policy, e.g. “is the following sentence toxic? ___”

Notably this method does not use any human-labeled data, it only needs to be told what type of text it is looking for. This is referred to as “zero shot”, meaning it needs zero training data. These models can also use “few shot” learning, where a small number of examples are given instead of the thousands of examples that had ordinarily been used.

This has big benefits: it allows you to very quickly refine policy, and the LLM can generate explanations for why it made a decision.

2020: GPT-3: reasonable zero-shot performance (Brown et al. (2020))
2022: ChatGPT published: very good zero-shot performance on many tasks.
2023: OpenAI provides GPT-4-based content moderation tools (Weng, Goel, and Vallone (2023))
2023: Startups providing LLM-based content moderation: SafetyKit, CheckStep, Hive, Cove.
2023: Stanford CoPE:an open-source LLM for moderation.

Discussion

Q: now that we can use LLMs for arbitrary labeling, will we change policies?

Proposals are coming out of Michael Bernstein’s lab, e.g. Jia et al. (2023), in using LLMs to substantially change how content is ranked.
Dave Wilner has argued that because LLMs offer much greater flexibility then platforms will find it easier to write more complex policies and update them more frequently.

Q: what do we know about degree of accuracy across languages?

AI typically has a strong anglophone bias. Performance in non-English languages tends to be proportional to the distance from English, e.g. European languages tend to be worse. However many also noted that there is typically a large anglophone bias in human moderation.
Some literature shows that LLMs have good performance in languages with relatively little training data, e.g. Armengol-Estapé, Gibert Bonet, and Melero (2021).

Q: will censorship change when using LLMs instead of humans?

Jeff noted that an advantage of human censors over machine censors is that humans might exercise their judgment to refuse to censor while machines will not.

References

Armengol-Estapé, Jordi, Ona de Gibert Bonet, and Maite Melero. 2021. “On the Multilingual Capabilities of Very Large-Scale English Language Models.” https://arxiv.org/pdf/2108.13349.pdf.

Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al. 2020. “Language Models Are Few-Shot Learners.” https://arxiv.org/pdf/2005.14165.pdf.

Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. “Bert: Pre-Training of Deep Bidirectional Transformers for Language Understanding.” arXiv Preprint arXiv:1810.04805. https://arxiv.org/pdf/1810.04805.pdf.

Jia, Chenyan, Michelle S Lam, Minh Chau Mai, Jeff Hancock, and Michael S Bernstein. 2023. “Embedding Democratic Values into Social Media AIs via Societal Objective Functions.” arXiv Preprint arXiv:2307.13912. https://arxiv.org/pdf/2307.13912.pdf.

Joulin, Armand, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. “Bag of Tricks for Efficient Text Classification.” https://arxiv.org/pdf/1607.01759.pdf.

Mikolov, Tomas, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. “Efficient Estimation of Word Representations in Vector Space.” https://arxiv.org/pdf/1301.3781.pdf.

Pennington, Jeffrey, Richard Socher, and Christopher D. Manning. 2014. “GloVe: Global Vectors for Word Representation.” In Empirical Methods in Natural Language Processing (EMNLP), 1532–43. http://www.aclweb.org/anthology/D14-1162.

Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. “Attention Is All You Need.” Advances in Neural Information Processing Systems 30. https://www.semanticscholar.org/paper/204e3073870fae3d05bcbc2f6a8e263d9b72e776.

Weng, Lilian, Vik Goel, and Andrea Vallone. 2023. “Using GPT-4 for Content Moderation.” https://openai.com/blog/using-gpt-4-for-content-moderation.

Thinking About Tradeoffs? Draw an Ellipse

Tom Cunningham, OpenAI. — Wed, 25 Oct 2023 07:00:00 GMT

This material was first presented at MIT CODE 2021. Thanks to Sean Taylor among others for comments.

Thinking about tradeoffs? draw an ellipse. When making a tradeoff between two outcomes, and , it’s useful to sketch out what the tradeoff looks like, and an ellipse is often a good first-order approximation. The ellipse helps visualize the most interesting parameter: the tightness, i.e. how much the rate of tradeoff between and varies as you increase .

In addition we can show that if the Pareto frontier is formed by the sum of vectors, and the vectors are drawn from a joint Normal distribution, then the expected frontier will be exactly an ellipse.

Concrete Applications:

Choosing launch criteria? draw an ellipse. Suppose you have a set of features each of which has some metric impact, and . If we assume that the effects are additive then we can construct a Pareto frontier, i.e. a set of all the aggregate effects on and achievable by selection from the set of features. The frontier will typically look like an ellipse. You can prove that the Pareto frontier will be exactly an ellipse if the set of experiment-effects have a joint Normal distribution.
Choosing ranking weights? draw an ellipse. Suppose you are ranking items for a user using a set of features e.g. p(Like), p(Comment), etc. It is useful to sketch out the Pareto frontier, i.e. the set of outcomes achievable by different ranking algorithms. If the outcomes are additively separable functions of the item-features, and if the joint distribution of features is Normal, then the Pareto frontier will be an ellipse.
Allocating headcount? draw an ellipse. When you shuffle headcount around a company it’s hard to precisely measure the impact on different goals, however I have found it useful to sketch ellipses to make explicit the tradeoffs you face. This is particularly useful for visualizing the differences between within-team vs between-team reallocation of effort.

Tight and Loose Tradeoffs

Suppose we care about two metrics, and . E.g. suppose we care about DAU and time-spent, or revenue and retention, or engagement and misinformation.

It is useful to draw a Pareto frontier. A Pareto frontier will show the set of achievable outcomes for X and Y, to make the tradeoff precise. If we have a well-defined objective function then we can visually represent the optimal choice where the indifference curve is tangent to the Pareto frontier.

If your frontier is “tight” then there is not much tradeoff. The first figure shows a tight frontier, meaning that there is not much tradeoff available between X and Y. With a tight tradeoff it doesn’t matter whether we maximize X or Y or a weighted average, we’ll end up in roughly the same place anyway. Suppose we are choosing among experiments: if we observe a high positive correlation between and then the choice of shipping criteria is relatively unimportant, most criteria would select the same experiments anyway. Suppose instead we are calibrating a recommender system: if we observe a high positive correlation between predictions of the two outcomes then the choice of weights is relatively unimportant, we would end up showing the same items anyway.

If your frontier is “loose” then there is a lot of tradeoff. The second figure shows a loose tradeoff: in this case the outcome does depend substantially on the relative weight we put on and .

Ellipses for Experiments

Suppose we have a set of experiments. Each experiment has some impact on two metrics, and . We visualize such a set of experiments at right.

If the set of features is separable, meaning that the impact of each feature is independent of what other features are launched, then a natural question will be the shape of the Pareto frontier formed by all possible combination of experiments.

If the distribution of experiments is mean zero and joint Normal then the Pareto frontier will be an ellipse, and it will have exactly the shape of an isovalue of the density of experiments. Thus knowing the variance and covariance of experiment results allows us to characterize the nature of the Pareto frontier we face.

Ellipses for Ranking

Suppose we are choosing a fixed set of items to show to a user, based on two metrics and . E.g. pLike and pComment, or pDAU and quality etc. A natural question will be the shape of the Pareto frontier formed by alternative selections of items.

The Pareto frontier will be an ellipse. We show below that, if the predictions are well calibrated, the outcomes are independent (i.e. additive), and the distribution of prediction obeys a joint Normal distribution, then the Pareto frontier will be an ellipse and it will have exactly the shape of an isovalue of the density of predictions. Thus knowing the variance and covariance of predictions allows us to exactly characterize the nature of the aggregate tradeoffs we face.

Ellipses for Company Strategy

Tradeoffs are looser higher in the decision hierarchy. Suppose a company cares about two outcomes, and . Many different people will be making tradeoff decisions between X and Y, we can distinguish between four objectives used at different levels in the company hierarchy:

We can think of each successive level as holding more variables fixed, and so we expect the Pareto frontiers to become successively tighter (Le Chatelier principle). We thus expect the tradeoff to be loosest at the level of overall company objectives, where we reallocate headcount. For this reason we should expect that, if the company as a whole pivots form metric to metric , the principal effect will be a reallocation of effort between products rather than reallocation within products.

We now walk through some of the different levels of optimization:

Different product areas have different Pareto frontiers. Typically two different product areas will have substantially different ability to affect different metrics, and we will often observe a situation like that shown on the right: team A’s choices primarily affect metric , team B’s choices primarily affect metric .

We can also draw a combined Pareto frontier. Here we add up the Pareto frontiers of team A and B. In this case the combined frontier is somewhat tight, because the two constituent frontiers are tight. Neither individual Pareto frontier shows a substantial effect from changing weights (if we restrict weights to be positive), and so accordingly the combined Pareto frontier shows little response to a change in weights.

Note that I have drawn the frontier only approximately, the frontier achieved by combining two ellipses does not have a simple representation. When the two constituent frontiers are straight lines then the combination will be a parallelogram. (Note also that when the two frontiers are circles then the combination will be a circle).

Greater investment will shift Pareto frontiers out. Here we visualize reallocating employees from team B (the frontier shifts in) to team A (the frontier shifts out).

A combined company Pareto frontier will be loose. Here the green curve represents all the possible outcomes as you shift resources between team A and B: we have now turned a tight tradeoff into a loose tradeoff. In this case this represents that a change in company objectives will be reflected mainly in reallocation of effort between teams rather than within teams.

Appendix: Model for Normal Distributions

Suppose we have a set of items with, and , distributed Normally:

We additionally let each item have a score, , which is simply a weighted sum of the two characteristics (normalizing the weight on the first characteristic to be 1):

We can write the covariance between the characteristics and the score as follows:

We wish to know the total number of actions of each type, and , for a given score threshold :

We first calculate the conditional expectations:

Next we will assume that the expected quantity of items is fixed. This implies that both both and will be constant, and we will define:

We thus have expressions for and as a function of the relative weight . We wish to rearrange these to express directly in terms of . To help we turn to Mathematica, with the following input:¹

¹ see notebook

F1[w_,p_,s1_,s2_,g_]:=g(s1^2+w p s1 s2 )/Sqrt[s1^2+w^2 s2^2+2w p s1 s2]
F2[w_,p_,s1_,s2_,g_]:=g(w s2^2 +p s1 s2)/Sqrt[s1^2+w^2 s2^2+2w p s1 s2]
Solve[{X1==F1[w,p,s1,s2,g],X2==F2[w,p,s1,s2,g]}, {X1,w}]
Simplify[First[%1]]

This returns a large expression:

We can however substantially simplify this:

We now wish to show that this curve is equal to an isovalue of the joint distribution of and (illustrated at right). We can write the isovalue of the joint Normal distribution of for any given as follows:²

² From Bertsekas and Tsitsiklis (2002) “Introduction to Probability”, Section 4.7

Solving this quadratic we can write:

We can see that this will be identical to the relationship between and above when .

Appendix: Simulations for Non-Normal Distributions

In this section I compare Pareto frontiers generated from different distribution of from different joint distribution, and then draw the Pareto frontiers. There are a few points of interest:

Other distributions apart from the Normal do not have the property that the Pareto frontier is equal to an isovalue of the joint density.
In my examples the Pareto frontier of many other distributions does look roughly elliptical though not precisely an ellipse. This makes me more comfortable to use an ellipse as a first-order approximation of a Pareto frontier.

These simulations show something slightly different from what is proved in the prior section. These simulations show that as the number of experiments is large () then the Pareto frontier begins to resemble an ellipse. What was proved in the previous section is that for any given the expected Pareto frontier will be an ellipse, where we calculate the expected value of and for a given rate of tradeoff.

In each case the left-hand plot shows the raw distribution, the right-hand plot shows the Pareto frontier.

Joint normal with positive correlation, small :

Joint normal with positive correlation, =100:

Joint t Distribution with 3 degrees of freedom, =100.

Independent Laplace, =100:

Common Laplace factor with independent Gaussian noise, =100:

Independent uniform, =100:

Common uniform factor plus independent uniform noise, =100:

Experiment Interpretation and Extrapolation

Tom Cunningham — Tue, 17 Oct 2023 07:00:00 GMT

Introduction

I give a simple Bayesian way of thinking about experiments, and implications for interpretation and extrapolation.

Thanks to J. Mark Hou for comments.

Setup: The canonical tech problem is to choose a policy to maximize long-run user retention. Because the policy space is high-dimensional it’s not feasible to run experiments on every alternative (there are trillions), instead most of the decision-making is done with human intuition based on observational data, and experiments are run to confirm those intuitions.

The inference problem. The basic problem of experimentation is to estimate the true effect given the observed effect. The problem can become complicated when we have a set of different observed effects, e.g. across experiments, across metrics, across subgroups, or across time.

Two common approaches are: (1) adjust confidence intervals (e.g. Bonferroni, always-valid, FDR-adjusted); (2) adjust point estimates based on the distribution (empirical Bayes). Both have significant drawbacks: my suggested approach is to let decision-makers make their own best-estimates of the true effects but provide them with an informative set of benchmark statistics so they can compare the results of any given experiment to the results from a reference group.¹
The extrapolation problem. Given an effect on metric A what’s our best estimate of the effect on metric B? This problem is common to observational inference, proximal goals, and extrapolation.

There are three approaches to solving this: (1) using raw priors; (2) using correlation across units (surrogacy); (3) using correlation across experiments (meta-analysis). I argue that approach #3 is generally the best option but reasonable care needs to be taken in interpreting the results.

¹ If the decision-maker is not technical then a data scientist or engineer can summarize for the decision-maker their best-estimate of the true impact on long-run outcomes, taking into account the evidence from the experiment and other sources of evidence, including the distribution of effects from other experiments.

I also briefly discuss two additional problems:

The explore-exploit problem. We would like to choose which experiments to run in an efficient and automated way. I think the technical solution is relatively clear but tech companies have struggled to implement it because good execution requires some discipline. I describe a simple algorithm that is not optimal but very simple and robust.
The culture problem. Inside tech companies people keep misusing experiments and misinterpreting the results, especially (1) running under-powered experiments, (2) selectively choosing results, and (3) looking at correlations without thinking about identification.

A common response is to restrict access to only a subset of experiment resuts. However this often backfires because (1) it is difficult to formally specify the right subset; (2) it reinforces a perception that experimental results can be interpreted as best-estimates of true treatment effects; (3) it reinforces a norm of selecting experimental results as arguments for a desired outcome. I think a better alternative is to explicitly frame the problem as one of predicting the true effect given imperfect evidence, and benchmark peoples’ prior performance in predicting the true effect of an intervention. (This section is unfinished, I hope to add more).

Setup

Firms choose their policy to maximize user retention. As a simplified model companies are choosing policies to maximize long-run retention (or revenue). A policy is, for example, a recommendation algorithm, or notification algorithm, or the text and images used in an advertisement or the UX on a signup page. Notice that policies are very high dimensional: there are millions or billions of alternatives, while we usually run only a few experiments.²

² In fact variation in the success of tech platforms is primarily due to variation in the inflow of new users, not due to variation in retention rates. However growth in new users is driven by the attractiveness of the product and retention is a good proxy for this.

Experiments and formal causal inference methods not the primary sources of causal knowledge. People already have substantial knowledge about the effects of their decisions without either randomized experiments or natural experiments (IV, RDD, etc.). We built cathedrals, aeroplanes, welfare states, we doubled human life-expectancy, & WhatsApp grew to 1B users, all without randomized experiments or instrumental variables estimates. These achievements were all based on causal inference but informal causal inference, i.e. using our instinctive knowledge of how to process information without writing down or calculating the assumptions and distributions. Formal causal inference methods are useful but primarly insofar as they augment our already substantial causal abilities, and in most cases they clearly lag far behind humans intuitive ability to draw causal inferences. Inside companies the primary way people learn about causal relationships is raw data (e.g. dashboards) and common-sense reasoning about human behaviour.

Experiments only solve the low-dimensional problem. In most cases the dimensionality of the policy space is far higher than the dimensionality of experiment space, thus the responsibility for choosing policies is primarily human judgment. Humans come up with a few policy variants of interest, and then run experiments to compare their performance.³⁴

³ Statistically the problem is analagous to drilling for oil: you drill test holes trying to understand the distribution underneath. A common algorithm for this is “Kriging” (the invention of Danie Krige in 1960 to model gold mining in South Africa) which in some cases is equivalent to Gaussian process regression.

⁴ There is a nice analogy to machine learning: computers have been able to outperform humans at low-dimensional prediction problems, e.g. linear regression, for the last 100 years, but only in the last 10 years have they caught up in high-dimensional problems like recognition of patterns in images, speech, and text. The implication is that humans have extraordinarily powerful ability to intuitively represent latent structures in high-dimensional data, and we shouldn’t expect simple algorithms to replace that ability.

⁵ For simplicitly assume the experiment doesn’t have any effect on variances or covariances of outcomes, the effects are typically small enough that it doesn’t matter.

Most questions related to experiments can be expressed as conditional expectations. A good workhorse model of experimentation is the following. Suppose we have two metrics #1 and #2. Taking some set of experiments we can think of three joint distributions: the observed effects, the true effects, and the noise:⁵

For simplicity we’ll assume everything is normally distributed and has mean zero, then we get two very simple expressions for conditional expectations, and I’ll argue that these conditional expectations serve as answers to almost all interesting experimentation questions:

Once we have a clear expression in terms of conditional expectations we can add on additional considerations: nonlinearities, fat-tailed distributions, strategic problems, etc..

1 The Inference Problem

There are a number of experiment inference problems that we often find difficult. We will discuss these as pure inference problems without worrying about strategic behaviour (e.g. peeking, cherry-picking).

Estimate the treatment effect given the observed treatment effect.
Estimate the long-run treatment effect knowing the short-run observed effect.
Estimate the treatment effect, knowing the observed effect, and additionally the distribution of observed effects across some set of experiments.
Estimate the treatment effect on a subgroup, knowing the observed effect, and additionally the distribution of observed effects across all other subgroups.

The textbook approach uses p-values. A common approach (NHST) is to treat the true effect as equal to the observed effect if the p-value is below 0.05, and otherwise treat the true effect as zero. This leads to all sorts of well-known difficulties.

Empirical Bayes estimates are often imperfect. We could instead calculate empirical-Bayes conditional expectations, , based on covariances from prior experiments, and treat those as the true effects. However the distribution of prior experiments is only a subset of the full information set available to the decision-maker, i.e. empirical Bayes is not Bayes, and very often there are idiosyncratic details about this particular experiment that are consequential.

My recommendation: report “benchmark” statistics. The ideal decision process lets humans make a judgment about estimated treatment effects given three ingredients:

Raw estimate. The point estimate and standard error.⁶
Benchmark statistic. We should also report a statistic comparing this observed effect to observed effects of other similar treatments. There are many ways of benchmarking and I think they are all convey the same basic information, e.g. the empirical-bayes shrunk estimate (and there are various shrinkage estimators), the FDR-adjusted p-value, or the fraction of statistically significant experiments. We have to use judgment in defining what a “similar” experiment is, and it’s important that we report to the end-user what class of similar experiments we’re using and how many we have. For the remainder of the section I will assume we are reporting an empirical-bayes shrunk estimate.
Idiosyncratic details. We should additional report any information about this treatment relative to the benchmark class, that could be relevant to its effect on this metric. E.g. (1) suppose this experiment only affects iPhone users then it is rational to heavily discount any outcomes on Android use unless they are highly significant; (2) suppose this experiment is a direct replication of a prior experiment, then we will likely wish to shrink our estimates towards that prior experiment rather than towards the mean of all experiments.

⁶ Equivalently, the point-estimate and p-value, or the upper and lower confidence bounds.

Benchmarking solves all the problems above. An empirical-Bayes shrunk estimate represents our best guess at the true treatment effect conditional on the experiment being drawn from a given reference class.

Useful shortcut: using the fraction of significant experiments to do shrinkage. A convenient rule of thumb for doing empirical Bayes shrinkage is to use the fraction of experiments that are statistically significant in some class. If the fraction is 5% then we should shrink all estimates to zero, if the fraction is 20% then we should shrink estimates by about 50%, and if the fraction is 1/2 then we should shrink estimates by about 20%. If everything’s Gaussian and every experiment has the same then the optimal shrinkage factor is , where is the fraction of stat-sig experiments.

1.1 Strategic Problems

There are additionally some strategic problems in experiment interpretation.

Strategic stopping (“peeking”). An engineer will wait until an experiment has a high estimated impact, or low p-value, before presenting it for launch review. A common proposed remedy is that all experiments should be evaluated after the same length of time, or that engineers should pre-specify the length of experiments.
Selection of treatments (“winners curse”). An engineer will run a dozen variants and only present for launch review the best-performing one. A common proposed remedy is that every variant should be officially presented in launch reviews, even the poorly-performing ones.
Selection of metrics (“cherry picking”). An engineer will choose to show the experiment results on the metrics that are favorable, not those that are unfavorable. A common proposed remedy is that the set of metrics should be standardized for all launches, or that the set of evaluation metrics should be pre-specified by the engineer (AKA a pre-analysis plan).

I will argue that the commonly proposed remedies are highly imperfect fixes. These are complicated things to think about because the mix together issues of statistical inference and of strategic behaviour. In the discussion that follows I try to separate those out as clearly as possible.

1.2 Strategic Stopping

I will ignore dynamic effects. For simplicity assume that all effects are constant, so the length of an experiment effectively determines just the sample size of that experiment. I.e. I will ignore time-dependent and exposure-dependent effects.

Stopping rules are irrelevant to expected effect sizes. Suppose an experiment has a given estimate. Does it matter to your estimate of the true causal effect if you learn that the experimenter chose the sample size by a data-dependent rule, e.g. continuing to collect data until the estimate was statistically significant? If you are estimating the true causal effect, then it doesn’t matter, your posterior will be identical either way.⁷ A simple proof: suppose we observe two noisy signals, and : Suppose a peeker will report only if , otherwise they will report . We can compare the expectation of given the sum, depending on whther the engineer peeked: This holds because is a sufficient statistic for the distribution, i.e. does not tell us any additional information. Note that peeking is not irrelevant to interpretation of a result if (1) the engineer can choose to report either or , (2) the engineer can choose to report alone after observing .⁸

⁷ This argument holds if the engineer always has to report the most-recent estimate. If they can choose to ignore later datapoints, and report an earlier result, this is essentially a “selection of metrics” case as below, and so the selection rule is relevant for interpretation.

⁸ See Deng et al. (2016) for a fuller argument that stopping rules are irrelevant, and a review of the prior literature.

Stopping rules would be relevant if we made decisions based on statistical-significance. A stopping rule would be relevant if we conditioned only on statistical-signficance instead of the full estimate. In other words the expected true effect, conditioning only on whether or not the estimated effect is statistically significant, will depend on the stopping rule. For example if people kept running experiments until they were significant then significant experiments would tend to have small effect sizes. However it is clearly bad practice to condition only on this binary piece of information when you have the full estimate, and if you have the full estimate then the stopping rule becomes irrelevant.

The optimal stopping rule is data-dependent. The discussion above took a stopping rule as given, we can also ask what’s the efficient stopping rule. It’s clear that a fixed length is inefficient: we should stop an experiment sooner if it does unexpectedly well or unexpectedly badly, in both of those cases the value of collecting more information has decreased because it’s less likely to change our mind about a launch decision. Thus enforcing a static or pre-specific experiment length will lead to inefficient decision-making.

Considering engineers’ incentives. Now consider the launch process as a game, with the engineers trying to persuade the director to launch their feature. Suppose the director’s ex post optimal strategy is to launch if , and suppose the engineers get a bonus whenever their feature is launched. In equilibrium the engineers will keep their experiments running until , which will cause a skew distribution: the distribution of posteriors will show a cluster just above the threshold. The director’s strategy is ex post optimal but it’s not an efficient use of experimentation resources. In this game the director would likely wish to pre-commit to a different threshold which induces more efficient effort by engineers. However a more direct solution would be to align engineers’ incentives with those of the director by rewarding them for their true impact, i.e. setting their bonuses proportional to , instead of discontinuously rewarding them for whether or not they launched.

1.3 Selection of Treatments

If you learn an experiment is the top-performing variant it should change your asssessment. Suppose we have a result , and we are estimating the true treatment effect, . If we learn that another variant has a lower treatment effect, , then it is rational to update our assessment of :

This will hold whenever , i.e. when we have some shared source of uncertainty about the two treatment effects.⁹ We can write a model for this, however conditioning on this binary information (whether a variant is the winner) is not an efficient way of using the information at your disposal.

⁹ Because and represent independent experiments we’ll have .

¹⁰ Andrews et al. (2019) describes some unbiased estimates for treatment effects conditional on them being winners. In general I would say this is an inefficient use of information, because we know much more about the distribution of treatment effects than just whether a specific variant is the winner. However that paper does argue that empirical Bayes estimates struggle when the sample-size is small or when we are estimating the tails of when variants are non-exchangeable, and in those cases the unbiased estimators may be useful.

It’s better to condition on the whole distribution. In almost all cases we know much more than whether is the winner, we also know the value of , and then this reduces simply to the empirical Bayes problem, i.e. we simply wish to estimate: and we can do that in the usual way.¹⁰ E.g. if we have a Normal prior over treatment effects then we can estimate from and . Once we have conditioned on then it becomes irrelevant whether variant 1 is the winner or not, i.e.:

Put another way: the selection rule is irrelevant (just as the stopping rule is irrelevant) once we condition on the distribution of observed outcomes.

Implication: show the distribution. If we are worried that engineers are selecting variants based on their outcomes then the simplest and cleanest fix is to calculate the distribution of variants and use that to discount any experiment results, either explicitly with an empirical Bayes estimator, or implicitly by showing the decision-maker the distribution.

1.4 Selection of Metrics

Suppose engineers are selectively presenting the most favorable metrics. Suppose there are two outcome metrics from a single experiment, and the engineer will present whichever is the most favorable. Knowing this fact should rationally affect your judgment of the treatment effect on the presented metric:

Implication: engineers should present all outcome metrics.

1.5 On Launch Criteria

Choosing weights on metrics for a launch decisions involves many considerations: network effects, noise, cross-metric proxy effects, and dynamic effects. In addition launch rules serve a bureaucratic role, and engineers will often want the launch rule to be public and without discretion. To make clear decisions it’s important to peel apart these layers, I recommend these steps:

Choose a set of final metrics. These are the metrics we would care about if we had perfect knowledge of the experimental effect. We can define tradeoffs between them, it’s convenient to express those tradeoffs in terms of percentage changes, e.g. we might be indifferent between 1% DAU, 2% time/DAU, and 5% prevalence of bad content.¹¹
Choose a set of proximal metrics. These are the metrics on which we are confident we can detect our experiment’s effect, meaning the measured impact will be close to the true impact on these metrics (i.e. has a high signal-noise ratio). To determine whether a metric is moved we can use the fraction of a given class of experiments that have a statistically-significant effect on that metric: if the share is greater than 50% then we can be confident that the estimated effect is close to the true effect.
Identify conversion factors between proximal and final metrics. These tell us the best-estimate impact on final metrics given the impact on proximal metrics. Conversion factors can be estimated either from (a) long-running tuning experiments; (b) a meta-analysis of prior experiments with similar designs.

A final linear launch criteria can then be expressed as a set of conversion-factor weights applied to each of the proximal metrics.¹²

¹¹ Arguably revenue or profit is a more truly final metric, and these are just proxies, but these are probably close enough to final for most purposes.

¹² For derivation see Cunningham and Kim (2019).

1.6 Comparing Launch Rules

Ship if sum is positive

I find it useful to visualize different launch rules. For simplicity suppose our utility function is linear: we have two metrics, 1 and 2, and we care about them equally: But we only observe noisy estimates .

Kohavi et al. (2020) recommend a stat-sig shipping rule. They say (p105):

If no metrics are positive-significant then do not ship
If some are positive-significant and none are negative-significant then ship
If some are positive-significant and some are negative-significant then “decide based on the tradeoffs.

I represent this in the first diagram (but I treat condition 3 as a non-ship). The dotted line represents .

The stat-sig shipping rule has strange consequences. You can see that this rule will recommend shipping things even with negative face-value utility (), when there’s a negative outcome on the relatively noisier metric. This will still hold if we evaluate utility with shrunk estimates, when there’s equal proportional shrinkage on the two metrics, but if there’s greater shrinkage on the noisier metric it will not hold.

Linear shipping rules are better. In the margin I illustrate (1) a rule to ship wherever the sum is positive; (2) a rule to ship wherever the sum is stat-sig positive. I have drawn the second assuming that . With a positive covariance the threshold would be higher.

The Leontief sandwich. I assumed above that our true utility function is linear. In fact tech companies often explicitly give nonlinear objective functions to teams, e.g.:

This is illustrated at right, the indifference curves are L-shaped so I’ll call it a Leontief utility. Having Leontief preferences can cause some unintuitive decision-making, in particular the tradeoff between and will varies drastically depending on your location. One important observation is that if your goal is assessed at the end of some time-point (e.g. at the end of the half) then optimal launch decisions will depend on your future expectations, e.g. you’d be willing to launch a feature that boosts A at the cost of B only if you expect a future launch to make up that deficit in B.

In practice I think it’s useful to think of this nonlinear objective function as sitting in the middle of the hierarchy of an organization, with approximately linear objective functions above and below it, i.e. a “Leontief sandwich.”

At the highest layer the CEO (or shareholders) care about all the metrics in way that is locally linear, i.e. they do not have sharp discontinuities in how they assess the company’s health. At the lowest layer engineers and data scientists are trying to make individual changes that achieve the Org’s overall goals, but because they only account for a small share of the overall org’s impact they can treat their objectives as locally linear (& likewise in a value function we make linear tradeoffs between objectives because we’re in such a small region). Finally even for orgs which have nonlinear objective functions it’s often reasonable to think of the nonlinearities as “soft”, e.g. if an org comes in slightly below a guardrail the punishment is slight, and if they come in above the guardrail then they will be rewarded. This softening makes the effective objective function much closer to linear, and so I think for many practical purposes it’s reasonable to start with a linear objective function.

2 The Extrapolation Problem

Many problems are predicting the effect one one metric (downstream) given the effect on another metric (upstream). There are a variety of situations in which we cannot measure the effect on the downstream metric, either because it has high noise, or it is in the future:

upstream	downstream
short-run revenue	long-run revenue
click	purchase
engagement on content	response to survey (“do you like this content?”)
engagement on content	retention
exposure to content	retention
time on surface X	time on all surfaces
purchase	repeat purchase
wait-time for delivery	retention
price	quantity purchased

For concreteness we will treat the problem of predicting the long-run (LR) effect of an experiment on DAU from its short-run (SR) estimated effects on all metrics:

There are two obvious ways to calculate this:

Meta-analysis. We can run a regression across prior experiments:

However the coefficients will be biased if we use on the LHS the observed long-run DAU, instead of the true long-run DAU. This bias is often large, and in fact if you run a bunch of AA tests (where the causal effect is zero) you’ll find strong significant relationships between short-run and long-run impacts. I discuss below ways in which to adjust for this bias.
Observational Inference. We can run a regression across users:

We can look at what is most predictive of long-run DAU across users. The problem here is obviously endogeneity, and so it’s worth spending time drawing a DAG and running robustness tests to carefully think through the sources of variation we’re using.

2.1 With Meta-Analysis

With metrics we can write the underlying model as:

Here we are treating and as two different metrics, but for some experiments we only observe the first. We thus want to estimate the effect on long-run retention (DAU) given short-run metrics.

where

Running a Regression will be Biased. The obvious thing to do is run a regression across experiments:

However this will be biased. The simplest way to demonstrate the bias is to show that even with AA tests (where there is zero treatment effect on either metric) we will still get a strong predictive relationship between the observed treatment effects on each of the two metrics (see figure).

A simulated scatter-plot showing 20 experiments, with N=1,000,000, , with correlation 0.8. The experiments are all AA-tests, i.e. there are no true treatment effects, yet a regression of on will consistently yield statistically-significant coefficients of around 0.8.

The bias is because in the regression our LHS variable is estimated retention ( instead of ), and the noise in that estimate will be correlated with the noise in the estimates of short-run metrics. In the linear bivariate case (where we have just one RHS variable) then we can write:

The bias will be small if the short-run metrics have high signal-noise ratios (SNR), . A simple test for SNR ratio is the distribution of p-values: if most experiments are significant then the SNR is high. However in the typical case (1) is the best predictor of ; and (2) has a low signal-noise ratio (i.e. few outcomes are stat-sig). This means the bias is large, and so results are hard to interpret.

2.1.1 Adjusting for the Bias

Here are some alternatives:

Run a regression just using the high-SNR metrics. We could just drop as a regressor because of the bias, but we lose predictive power () so it’s hard to know when this will be a good idea without an explicit model.
Adjust for bias in linear estimator. If we want a linear estimator then we can estimate and adjust for the bias.

If everything is joint normal then the expectation is itself linear, and so this will be optimal. In practice the true distribution of effect-sizes is somewhat fat-tailed, which imply that the conditional expectation will be nonlinear in the observables. Nevertheless I think this is a good start. (One other complication is that the SNR is more complicated to calculate when experiments vary in their sample size).¹³
Use experiment splitting. You can randomly assign users in each experiment to one or other sub-experiments. You now effectively have a set of pairs of experiments, each of which has experiments with identical treatment effects () but independent noise. Thus you can run a regression with LHS from one split, and RHS from other split, and you’ll get an unbiased estimate. Additionally you can easily fit a nonlinear model (Coey and Cunningham (2019) has details of how to do an experiment-splitting).¹⁴
Run a regression just using the strongest experiments. If the distribution of experiments is fat-tailed then the strongest experiments will have higher SNR, and so lower bias. A worry about this is that you’re only estimating the relationship from outliers, so nonlinearities are more of a worry. At the same time the assumption of fat-tailed treatment-effects gives reason to believe the expectation will be nonlinear. (This is roughly how I interpret the Peysakhovich and Eckles (2018) experiments-as-instruments paper. They propose using L0 regularization and experiment-splitting cross-validations, which I think effectively selects the strongest experiments.)

¹³ See Cunningham and Kim (2019), and see Tripuraneni et al. (2023) for a slightly different setup with weaker assumptions. Bibaut et al. (2024) shows the relationship with LIML.

¹⁴ Bibaut et al. (2024) gives another formalization in terms of jacknife-instrumental-variable (JIVE).

Choosing a Reference Class. It is important to think about the reference-class of experiments which we use to calibrate our estimates. The long-run DAU prediction can be though of as an empirical-bayes estimate, which is our best estimate conditional on the experiment being a random draw from this class of experiments.

In many cases a company’s experiments will naturally fall into different classes: e.g. some have a very steep relationship between engagement and DAU, others have a very flat. It’s important to both (1) visualize all the experiments, so that a reference-class can be chosen sensibly; (2) calculate the across experiments, so we can have some sense of confidence in our extrapolation.

2.2 Observational Inference

What we want to know: Given the short-run effect of a content experiment on engagement we want to predict the long-run effect on DAU. We can start with a simple regression along these lines:

We could set up a DAG and discuss the surrogacy conditions. The condition are that (1) all effects of an experiment on DAU are via short-run engagement; and (2) there is no unobserved factor which affects both SR engagement and LR DAU:

In fact we know that engagement doesn’t literally lie on the causal chain, instead we think engagement is a good proxy for content which might lie on the causal chain.

In any case I find the following setup an easier way to think about the assumptions necessary for identification:

We can write it out a simple structural model as follows (for compactness I leave out coefficients):

Some general observations:

We would get a more credible estimate if we could directly measure content quality. E.g. if we could use the quality of the content available to the user on the RHS, instead of just their engagement on that content. This wouldn’t get perfect identification but it would help.
The relative shares of variation in the RHS is important. If most of the variation in engagement is due to variation in content (i.e. high from content), then we don’t need to worry much about confounding from other effects. We can think of introducing control variables as a way of increasing the share of varation in engagement due to content.
We should control for distractions. If we have measures of app-related events that don’t affect content-seen but do affect engagement, e.g. notifications, messages, then we should use those as controls. This will increase the relative share of variation in engagement due to content.
Controlling for pre-treatment outcomes changes variation used. If we control for engagement this will change the relative contribution of each factor in the variation of engagement. Specifically it will reduce the share of the terms with higher autocorrelation. Thus by definition temperament will reduce its contribution. However it’s unclear whether mood or content has higher autocorrelation, and so controlling for pre-treatment could either increase or decrease the relative contribution of content. It’s probably worth doing some simple decomposition of variation in engagement into (1) user, (2) content, and (3) mood (the residual), both statically and over time.
Univariate linear prediction is usually pretty good. In my experience you can get a fairly good prediction of most user-level metrics with a linear function of the lagged values. If you use a multivariate or nonlinear function you’ll get a better fit but only by a small amount (one exception: when predicting discrete variables like DAU it’s useful to use a continuous lagged variable like time-spent). So I’m skeptical that adding more regressors or adding nonlinearity will significantly change the estimates or the credibility of the estimates.
Estimand is not but . Suppose we see that 1 unit of engagement causes a certain increase in DAU over the following weeks. We then want to apply that estimate to an experiment which permanently increases engagement by 1 unit. We thus should take the integral over all the subsequent DAU effects. In the simple exponential case the effect of a shock at period on DAU at period will be , and so the cumulative effect on all subsequent periods will be .
Autocorrelation in content makes things messier. If there is significant autocorrelation in content then the interpretation of DAU~engagement is more difficult. E.g. if we see that engagement on is correlated with DAU on this could be because either (1) content on content caused the DAU on , or (2) good content on is correlated with good content on , which in turn causes DAU on . I don’t think controlling for pre-treatment levels or trends solves this.

Appendix: The Explore-Exploit Problem

What experiments should you run? The prior sections have been just about interpretation of existing experiments, we can now turn to the choice of which experiment to run. The space of all possible experiments is immensely high dimensional and thus most of this process uses human judgment. However in some cases we can reduce the space to a small number of dimensions and use an algorithm to explore that space. We can call this process a “bandit” or “explore exploit” or “adaptive experimentation” or “gradient descent” problem (though gradient descent is typically pure exploration with no exploitation).

Typical cases for explore-exploit:

Tuning parameters on a recommendation algorithm to maximize retention.
Tuning parameters on video or audio streaming to maximize satisfaction and retention.
Tuning parameters on ad bidding to maximize net profit.
Exploring different components of quality in recommendations:
- Content quality
- Producer quality
- User-topic interest
In each case showing some content that is less interesting to the user, but in return for learning more information.

I will focus just on tuning parameters in a recommendation algorithm.

Tuning projects have a high failure rate. I should say that I am not an expert on explore-exploit algorithms and many others have deeper professional experience than I do. However I have seen multiple tuning projects either abandoned because of complexity, or fail to find a set of parameters which yields a non-trivial improvement on metrics. Speaking broadly I think the problems were overly-complicated designs, under-powered experiments, lags in effects, ill-defined outcome variables, or improper use of short-term proxies for long-term outcomes.

If is already close to the global optimum then there will be not much loss from perturbing some users because the loss function should be flat in that neighborhood.

Recommendation: a simple tuning algorithm using weather stations. Here is a crude but easy-to-execute method for dynamically optimizing parameters. It’s less efficient than other algorithms but it’s easy to describe, easy to implement (it uses the existing AB-test system), and easy to visualize and see that it’s working as intended. In short: for each parameter we set up two permanent “weather stations” treatments: 1/3 of users get a slightly higher value, and 1/3 of users get a slightly lower value.

Suppose we have parameters to tune : we run orthogonal experiments, each of which partitions the all users into 3 equal-sized buckets, with either (1) , (2) , (3) , where is the current production level of . If then users would be assigned as such:

If we start at a point above the global optimum then the “low” group benefits and the “high” group suffers, but we can see that any short-term cost will be outweighed by long-term benefit.


1/9	1/9	1/9
1/9	1/9	1/9
1/9	1/9	1/9

The size of the perturbations are easy to adjust dynamically as the data comes in: we can start small and keep increasing until we see a stat-sig difference in the outcome. We monitor the trajectory of each bucket continuously, and once/month make a formal decision about whether to adjust the production parameters, e.g. increasing to or lowering it to . When interpreting these experiments it is important to monitor the full trajectory of outcomes over time, ideally a visualization will show a large matrix of trajectories, with one cell for each combination of experiment-bucket and metric.

We can use the data generated to explore other aspects: (1) whether there are significant interaction effects between the different experiments (e.g. if the users who have both increasing and have a different effect), and (2) whether there are significant heterogeneities in outcomes across subgroups.

This is the simplest general framework I know of for continuous optimization of a set of parameters. I think that simplicity is by far the most important criterion: I have seen a long history of optimization projects get tangled in complexity and fail. Because of the past history of failures I think it’s crucial to do the simplest and most transparent thing at each point until you have a steady rhythm and track record of making progress.

The hard work is the choice of parameters to tune. Once you have a small set of parameters to tune it’s not too hard to find the global optimum. However in typical problems there are thousands or millions or billions of possible parameters, how should you choose which ones to tune?

Appendix: Difficult Cases

Example: Selection of experiments. Your team goal is to maximize podcast_time, and you want to know what other teams are hurting that metric. You find the 10 experiments with the biggest negative effect. Should you take their estimated effects at face value?

Classical advice is to adjust p-values for the number of experiments you selected from (Bonferroni correction). But from a Bayesian point of view it’s irrelevant whether these 10 experiments are taken from a pool of 10 or 1000 experiments.
The set of experiments is informative about appropriate shrinkage. You can use the pool of experiments to estimate the appropriate shrinkage, . E.g. if we assume a Normal distribution we can quickly calculate a shrinkage estimate from the average effect and from the fraction of experiments that are statistically significant.
Shrinkage should depend on plausibility of the effect. You can look at how much each of these experiments moves their primary outcomes. Suppose a music-ranking experiment decreases podcast time-spent by 0.4s, and increases music time-spent by 0.2s: the more-than-proportional side-effect seems unlikely, so there is reason to discount (shrink) the likely effect significantly.
Shrink less if the effect is very significant. If the effect-size is 4 standard-errors then, because the distribution of treatment, this is much more likely to be due to treatment than to noise, and so the effect does not require much shrinkage.

Example: Selection of Experiments #2. An engineer has an experiment with effect +1% (±0.5%) on your goal metric. They mention that they ran 20 other experiments, and this is the experiment with the biggest effect.

Recommendation: shrink heavily towards the average effect.

Finding out about other experiments with smaller effects means you should shrink more. Finding out about the 20 other experiments is evidence about the size of the typical effect, and you should shrink towards that average. If the engineers are only showing you their best ones, that is reason to shrink your estimates.
It matters how selection was done. Suppose the engineer chose the highest-effect one by chance, not intention. You should still shrink by the same amount: the distribution is evidence, not the selection rule. However if they had some independent reason for expecting this experiment would be the most effective, that is relevant evidence.

Example: Subgroup Outcomes. You see that the overall time-spent of a feature holdout is -3.5% (±0.5%), but in Korea it’s -9%(±2%). How seriously should you take the Korean effect?

Recommendation: take it seriously, because (a) very significant, and (b) there is high between-country variance.

Is this effect plausible? I.e., do we have reason to expect the effect of this feature to vary a lot by country, and in particular in Korea? We do generally think user behaviour varies a lot by country.
How significance is this effect? The effect is 9 standard-errors – i.e., extremely significant – which makes it much less likely to be noise (=.00001).
How much variance in effect is between-country vs within-country? Suppose we see that 1/2 of the countries have effects that are significantly different from the global average effect, this implies that there is a fair amount of variance in effect-sizes, and so reasonable that Korea should be such an outlier.

Example: Multiple Outcomes.Your experiment increases music_time, which you expected, and increases podcast_time, which you did not expect.

Implication: The positive effect on podcast_time is bad news about music_time. If outcomes are positively correlated across units but not across treatments then: In this case good news about one outcome is bad news about the other.

Example: Multiple Outcomes #2. You run an experiment on movie ranking intended to increase watches, and it works. You additionally see an increase in comments-given. Should the increase in comments give you more confidence or less confidence in the increase in likes?

Recommendation: Good news is bad news, if the side-effect is unexpected.

If the experiment was expected to increase both metrics - e.g. by increasing overall time spent on feed - then this is good news: it is additional evidence for the effect on likes.
If the experment was expected to have a null or negative effect on comments – e.g. by boosting like-able posts at the expense of comment-able posts – then this is bad news: the positive effect on comments is likely due to noise, and it should make us expect greater noise in the measure of likes.

Given two treatment effects and , and two outcomes, , and two noise variables, then we have the following (in the Gaussian case):

References

Andrews, I., Kitagawa, T., McCloskey, A., 2019. Inference on winners. National Bureau of Economic Research.

Bibaut, A., Kallus, N., Lal, A., 2024. Nonparametric jackknife instrumental variable estimation and confounding robust surrogate indices. arXiv preprint arXiv:2406.14140.

Coey, D., Cunningham, T., 2019. Improving treatment effect estimators through experiment splitting, in: The World Wide Web Conference. ACM, pp. 285–295.

Cunningham, T., Kim, J., 2019. Interpreting experiments with multiple outcomes.

Deng, A., Lu, J., Chen, S., 2016. Continuous monitoring of a/b tests without pain: Optional stopping in bayesian testing. 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA) 243–252.

Kohavi, R., Tang, D., Xu, Y., 2020. Trustworthy online controlled experiments: A practical guide to a/b testing. Cambridge University Press.

Peysakhovich, A., Eckles, D., 2018. Learning causal effects from many randomized experiments using regularized instrumental variables, in: Proceedings of the 2018 World Wide Web Conference. International World Wide Web Conferences Steering Committee, pp. 699–707.

Tripuraneni, N., Richardson, L., D’Amour, A., Soriano, J., Yadlowsky, S., 2023. Choosing a proxy metric from past experiments. arXiv preprint arXiv:2309.07893.

Citation

BibTeX citation:

@online{cunningham2023,
  author = {Cunningham, Tom},
  title = {Experiment {Interpretation} and {Extrapolation}},
  date = {2023-10-17},
  url = {tecunningham.github.io/posts/2023-04-18-experiment-interpretation-extrapolation.html},
  langid = {en}
}

For attribution, please cite this work as:

Cunningham, T., 2023. Experiment Interpretation and Extrapolation [WWW Document]. URL tecunningham.github.io/posts/2023-04-18-experiment-interpretation-extrapolation.html

An AI Which Imitates Humans Can Beat Humans

Tom Cunningham — Fri, 06 Oct 2023 07:00:00 GMT

Thanks to comments from many, especially Giorgio Martini, Grady Ward, Rob Donnelly, Inés Moreno de Barreda, and Colin Fraser.

If we train AIs to imitate humans, will they ever beat humans? AI has caught up to human performance on many benchmarks, largely by learning to predict what humans would do. It seems important to know whether this is a ceiling or we should expect them to shoot out ahead of us. Will LLMs be able to write superhumanly-persuasive prose? Will image models be able to see things in photos that we cannot? There is a lot of technical literature on imitation learning in AI but I haven’t found much discussion of this point (Bowman (2023) is a notable exception).

In a formal model I derive five mechanisms by which imitative AI can beat humans.

Noise. Different humans give different answers to a question, and so if an LLM can consistently give the average answer it will do better than the average human (the error of the average being always smaller than the average of the error).
Specialization. People tend to write about what they know, and so an LLM which learns to predict the typical answer to a given question will sound like a specialist in all areas: it will answer questions about water like a hydrologist and questions about bugs like an entomologist (although it will also answer questions about astrology like an astrologist).
Interpolation. An LLM will interpolate responses from different humans, and this interpolation can be functionally equivalent to inference, meaning an LLM will sometimes be able to reliably answer questions that no human can answer.
Priors. If an LLM has different priors than a human then they could uncover hidden structure that humans do not, e.g. an LLM trained on human observations of astronomical events could conceivably recover cycles in those events, and so give superior predictions to the human.
Tacit knowledge. The majority of human knowledge is tacit, meaning it is used in forming judgments but we do not have conscious access to that knowledge. If AI models can accurately predict human judgments then the weights in those models effectively contain that tacit knowledge, and so the model can be re-engineered to use that knowledge in ways that humans cannot.

The evidence is unclear. There are many reasons why this could theoretically occur but I couldn’t find much evidence for superhuman performance: many benchmarks which we use to evaluate ML models have human labels as the ground truth, meaning we wouldn’t know when computers do pass us by.

This blog post contains:

A graphical argument illustrating the five mechanisms.
A deeper discussion of each of the five mechanisms.
A brief overview of the AI-human gap in various tasks.
Discussion of applications, related literature, and complications.
A simple formal model with a derivation of each of the five mechanisms.

Setup: Modelling Questions and Answers

For concreteness I will describe humans and computers answering questions about the world. However I think the basic framework applies generally to performing tasks or following instructions. Some examples of questions and answers:

question	answer
What’s the capital of Switzerland?	Bern
What’s the best response if white plays c4?	Nf6
Does this picture (🐄) depict a cow?	Yes
How much more likely is someone to buy a Coke after hearing the slogan “Coke refreshes”?	0.1%

Graphical Argument

Here I illustrate all the core points in a graphical framework. For simplicity I am representing a set of questions and answers which can be represented by a pair of numbers, e.g. asking what is elevation of a point along a given line of latitude? I treat the human and the computer as having smooth priors about the world which determines how they interpolate and extrapolate from their observations.

(1) Let the curve above represent the truth about the world. Each question about the world is a point on the x-axis, and the answer to each question is represented by the curve.

(2) A human forms beliefs about the world (red curve). The human asks two questions and gets two answers (red dots) and from these they form estimates of the answer to every other question (red line).

(3) A computer learns to predict the human’s answers (green curve). The human records some questions and their answers (green dots), and the computer learns to predict the human’s answers (green curve). Here I have illustrated a very favorable case, where the human has shared all her observations with the computer, and both the human and computer have the same priors. If the human gives inconsistent answers (imagine a thickening of the green line) then the computer will do better than the human by having less noise (the error of the average always being smaller than the average error).

(4) Add another human (blue curve). Suppose we have an additional human who asks some different questions (blue dots) and so forms different beliefs (blue line). Both humans’ beliefs are are accurate in the neighborhood of their own experience.

(5) Let the computer predict answers from both humans (green curve). Now both humans record their experiences and the computer tries to predict human answers (green curve). Here we can see:

Specialization. The computer’s predictions can match the humans’ responses in each of their domains of expertise
Interpolation. The computer is better than both humans in the intermediate region, i.e. the computer effectively combines information from both humans.

(6) Let the computer have superior priors. If the computer and human have different priors then they will make different extrapolations from the same dataset. Suppose the world has a strong cyclical structure, as shown in the black oscillating line. The human does not appreciate the regularity and fits their datapoints with a simple nearest-neighbor algorithm, but the computer, with different priors, could get a superior fit to the true model. A simple hypothetical: suppose we trained a language model to predict records of astronomical observations, it could conceivably discover cycles in these observations even if no human was aware of those cycles, such that computer predictions of out-of-sample human observations of the world would constitute super-human predictions about the world.

(7) Let the human have tacit knowledge. Finally, suppose the human always knows correct the answer when they see the question (red), but their conscious understanding of the relationship (pink) is imperfect. When asked abstract questions about the world they cannot use their tacit knowledge, e.g. if asked what is the maximum of this function they would choose the maximum of the pink curve (conscious beliefs), not the red curve (tacit beliefs). However the computer could learn the tacit knowledge from observing sufficiently many answers, and then algorithmically find the maximum of this curve, substantially outperforming the human at these abstract questions.

Five Reasons for Superhuman Performance

(1) Noise. For many tasks there is very high within-human and between-human variation, so any model which is deterministic will have a substantial advantage. Thus averaging multiple answers tends to do much better (the “wisdom of crowds”, and the “crowd within”). A computer with a deterministic outcome will thus have a substantial advantage. Zhang et al. (2024) shows that an LLM which is trained to predict chess moves of a group of players can outperform any of the players in that group - notably the effect is stronger when the model is trained on non-expert chess players, where the errors might be expected to be uncorrelated.

(2) Specialization. We can see clear evidence of specialization in LLMs: they will answer questions about fish like an icthyologist, and questions about Ukraine like a Ukrainian. There is a nice discussion of this with many examples by ryxcommar:

“When you ask ChatGPT a more intelligent question, you get a more intelligent answer. Just like how you ask ChatGPT a more Spanish question, you get a more Spanish answer.

This works very well as long as the people who talk most about a topic tend to be the people who are most knowledgeable about that topic. If the reverse was true then AI would perform worse than the average person. In some cases it does seem to be true that the people who are most talkative are the least accurate, e.g. for conspiracy theories, politically partisan issues, or pseudosciences.¹ Thus we can predict that asking an LLM about these issues will tend to give low-quality answers, and indeed if you ask GPT-4 “what are some characteristics of Virgoes?” it will give a quite factual answer listing the traits of Virgoes.²

¹ Analogically, in music or visual art, there might be some genres where the people who create artworks are uniquely bad at it, and so in these genres imitative AI would learn to make artworks worse than the average human would make.

² As of Oct 2023.

The same point applies for general supervised learning: suppose we train an image model to recognize tumors, and the training set includes examples from different radiologists, each who is an expert in their area (e.g. a pediatric radiologist labels the scans from children, a vetenarian radiologist labels the scans from animals), then the trained model could outperform any single radiologist.

(3) Interpolation. I have not yet come up with a crisp question which an LLM can accurately answer but no human can, however there are tasks which LLMs can perform which it likely no human can perform without help:

Recent LLMs (e.g. GPT, Bard) can transpose styles very easily, e.g. writing a Shakespearean sonnet about a particular episode of a particular television show, which arguably cannot be done by any human being.³
Armengol-Estapé, Gibert Bonet, and Melero (2021) show that GPT-3 does fairly well at answering questions and producing text in Catalan, despite Catalan constituting only 35M words in the training set (0.02% of the total), implying that it can answer questions for which the answer is known by no Catalan speaker. In principle language models could translate between a pair of languages for which there exists no common speaker but I do not know of any explicit confirmation of this.

³ Thanks to Giorgio Martini for this point.

⁴ thanks to Rob Donnelly for first suggesting this point to me.

(4) Priors. When making extrapolation from the same set of data humans and computers will given different answers because they have different priors, and in some cases computer priors might give a better fit.⁴

In fact there is a literature from the 1950s showing that computers with linear regression models do better than humans in learning to predict from novel datasets, and indeed better than experts in making clinical judgments using a small number of cues (Camerer and Johnson (1991)). This is surprising because it took an additional 70 years for computers to catch up with human ability in making many other judgments. I think the key difference is that the 1950s results apply to low-dimensional cases (), while only recently have we taught computers to deal with high-dimensional data ().

Unfortunately I don’t know of any clear-cut examples which show an LLM outperforming a human because of superior priors. Here is a hypothetical: suppose the training data of an LLM included a large set of scientific observations, and those observations contain some underlying pattern which was not recognized by any contemporary scientist. If you ask the LLM about the existence of patterns then it should answer like a scientist and say there is no known pattern. However if you ask the LLM to predict new observations then those predictions may obey the pattern, as a consequence the LLM could be used to systematically map out prediction, which may make it easier to identify the pattern. Thus you could imagine an LLM would correctly predict the position of the stars.

There is an interesting sublety in how the AI ought to be prompted to elicit superhuman knowledge. Consider these two prompts:

“Q: where will Venus be on June the 16th? ___”
“On June the 16th Venus was observed to be ___”

If the AI learned superior priors to the human then we should expect it to answer these prompts differently: it would answer prompt #1 using the human model, and answer prompt #2 using the true model.

(5) Tacit Knowledge.

The relative success of machine learning over symbolic AI has often been connected to the importance of tacit human knowledge. I have written a lot about the importance of tacit knowledge in human decision-making, especially Cunningham (2015) and this post, see also Cunningham and De Quidt (2022). In memory a related phenomenon is the “recognition recall” gap: people are significantly better at recognizing whether they saw a word before than in recalling that word (MacDougall (1904)).

Stiennon et al. (2022) train models to summarize text, based not on human summaries but on human evaluations of summaries. The model produces summaries that are preferred to human-produced summaries 70% of the time, i.e. superhuman production by training on human feedback. However this isn’t quite an apples-to-apples comparison because it’s unclear what the goals were of the humans who produced the baseline summaries: the human raters had explicit rubrics, but the human summarizers weren’t explicitly incentivized on those rubrics (as far as I can tell).

In addition I think that “inversion of tacit knowledge” is a reasonable description of image synthesis by neural nets: models are first trained to recognize images given captions, and then a new image can be synthesized to match a given caption, e.g. through a diffusion algorithm. Here there’s a striking asymmetry: algorithms can approximately match average human performance in recognition, but they far outperform human performance in construction of new artefacts.

Evidence on Superhuman Performance

Timeline

The following table shows the year in which a computer (or mechanical device) could match performance with the best human:

arithmetic	1642
chess	1997
Jeopardy	2005
image recognition (ImageNet)	2015
handwriting recognition (MNIST)	2015
question answering (SQuAD1.1)	2019
difficult math questions (MATH)	2023
coding problems (MBPP)	(not yet)

⁵ Kiela et al. (2021) also say that “models that achieve super-human performance on benchmark tasks (according to the narrow criteria used to define human performance) nonetheless fail on simple challenge examples and falter in real-world scenarios.”

Computers have hit the ceiling on most benchmarks. Kiela et al. (2023) documents that most computer benchmarks have become “saturated,” i.e. computers get close-to-perfect performance, and that recently the speed of saturation has become quicker (see graph on right). They say identify only a single benchmark where performance is not close to the human baseline, and most of the models they discuss are imitation learning. As a consequence some work has moved to evaluating models against “adversarial” benchmarks where the problems are chosen specifically to fool computers (e.g. Dynabench, Kiela et al. (2021)).⁵

On some tasks human performance defines success. On some tasks human performance effectively is the ground truth, and so by definition computers could never beat humans. This is roughly true for text comprehension: a sentence has a given meaning if and only if the average person believes it has that meaning. When we observe computer outperformance on this type of benchmark it is because either (1) there is human variation and the computer output is more consistent; or (2) computers outperform amateur humans but the ground truth is expert humans.

Performance by Task

Arithmetic: computers passed humans 300 years ago. Machines have been used to do calculations since the 17th century, e.g. Pascal’s calculator from 1642.

Backgammon	1979
Chess	1997
Jeopardy	2005
Atari games	2013
Go	2016
Starcraft	2019

(source)

Playing games: computers passed humans over the last 45 years. See the table in the margin for games. I am not aware of any well-known games in which computers cannot reliably beat the best humans.

(source)

Image recognition: computers surpassed humans in the 2010s. With the qualifications above about the limitations of benchmark tasks.

Question answering: computers surpassed humans in the 2010s. With the qualifications above about the limitations of benchmark tasks.

Facial recognition: computers seem to be equivalent to experts. Towler et al. (2023) say “naturally skilled super-recognizers, trained forensic examiners and deep neural networks, … achiev[e] equivalent accuracy.”

Coding: computers still below expert. See the benchmarks on PapersWithCode, also a graph on OurWorldInData, specifically APPS and MBPP. The best-performing computers are still imperfect at solving these coding challengers (which presumably can be solved by an expert programmer), but progress is rapid.

Writing persuasive text: computer comparable to average human. A number of recent papers compare the persuasive power of LLM-generated text to human-generated text (Bai et al. (2023), Goldstein et al. (2023), Hackenburg and Margetts (2023), Matz et al. (2023), Palmer and Spirling (2023), Qin et al. (2023)). They all find that LLMs do relatively well, but none show clear signs of computer superiority.

Writing creative blurbs: computer comparable to average human. Koivisto and Grassini (2023) compared GPT4 to online recruited humans (£2 for a 13 minute task) in giving “creative” uses for everyday items. The prompt was to “come up with original and creative uses for an object”, objects were “rope”, “box”, “pencil” and “candle.” The responses were rated by humans for their “creativity” or “originality.” GPT-4 responses were perhaps 1SD above the average human score, but the difference was smaller when choosing just the best response for each user.

Summarizing text: computer beats average human. Two recent papers found that LLM-generated summaries, trained with feedback, were preferred by humans to human-generated summaries (Stiennon et al. (2022) using RLHF and Lee et al. (2023) using RLHF). However in both cases it wasn’t clear to me exactly how the human summarizers were incentivized, and whether they were trying to perform the same task as the LLMs.⁶

⁶ Lee et al. (2023) say “RLAIF summaries are preferred over the reference [human-written] summaries 79% of the time, and RLHF are preferred over the reference summaries 80% of the time.”

⁷ Hendrycks et al. (2021) says “We found that a computer science PhD student who does not especially like mathematics attained approximately 40% on MATH, while a three-time IMO gold medalist attained 90%”

Doing math problems: computer comparable to expert. The latest score on the MATH benchmark is 84%, compared to 90% by a three-time IMO gold medalist. The scores have been rising very rapidly so it seems likely that computers will soon surpass humans.⁷

Discussion

I will use “superhuman” to mean the AI can answer some question better than any human can. We can formalize “superhuman” ability in a variety of ways. The notation is introduced more fully below, but briefly represents a question, represents the correct answer, represents the computer’s answer, and represents the answer of human . We assume squared error loss, and the expected error can be interpreted as either over the universe of all questions, or some subset of questions:

Note that we cannot rank benchmark #2 and #3: the error of the best human could be either higher or lower than the error of the average human. The most interesting question is whether AI can exhibit “super strong” superhuman performance. It seems clear that imitative learning can easily lead to superhuman performance by all the other 3 definitions through (1) reducing noise, and (2) combining expertise. However super-strong superhuman performance would require either (1) interpolation, (2) superior priors, or (3) using tacit human knowledge.

Training models with custom-written answers is still imitative learning. Recent LLMs don’t train just on predicting existing text (books, internet, twitter) they also use datasets of instructions and responses generated by paid raters (Ouyang et al. (2022)). We can still call this imitation but it’s putting relatively more weight on imitating the responses of specific set of people, the paid raters. This fine-tuning significantly improves performance on most benchmarks but I think it also has costs: the model is now predicting output of a specific set of people (i.e. non-expert paid raters), and so conceivably will do less well at incorporating niche information available to an expert.

Training on human evaluations is using human tacit knowledge. Recent LLMs are not purely imitative, e.g. OpenAI’s GPT models are trained with human evaluation of their responses (called reinforcement learning with human feedback (RLHF)), and they find that it dramatically increases performance on instruction-following benchmarks.⁸ The key difference is that the goal now reflects how humans rate responses rather than how humans generate responses. In some domains the two functions might be identical but in others there’s a clear difference, I would argue that tacit knowledge is the core difference.

⁸ E.g. see OpenAI’s 2022 InstructGPT (Ouyang et al. (2022)). In fact they run reinforcement learning against a model trained to predict the human evaluation of outputs, and other papers run reinforcement learning against LLM-produced evaluations, RLAIF (Lee et al. (2023)).

Bowman (2023) on super-human performance by LLMs. I have found surprisingly little online or academic discussion about whether LLMs will hit a ceiling defined by human performance. A good paper by Bowman (2023) has a section titled “human performance on a task isn’t an upper bound on LLM performance.” He says LLMs can outperform humans for two reasons: (1) “they are trained on far more data than any human sees,” and (2) “they are often given additional training using reinforcement learning … which trains them to produce responses that humans find helpful without requiring humans to demonstrate such helpful behavior.” I think these correspond to two of the five reasons I identified (specialization and tacit knowledge).

For some tasks human-level performance is the ceiling by definition. The ground truth about language interpretation is humans interpretation, and so it is hard to see how a computer could exhibit superhuman performance (in the super-strong sense).⁹ In benchmarks for natural language understanding the labels are typically written by the human authors of the benchmark, so it would be impossible to observe superhuman performance (Tedeschi et al. (2023)). A similar point applies to content moderation the definition of ground truth is typically either majority-vote among paid human raters, or the reflective judgment of a senior human employee, thus a computer could only outperform in the weak senses above.¹⁰

⁹ There are exceptions but I don’t think they are quantitatively important. Consider the a sentence like “Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo.” This sentence has at least one well-defined meaning according to the typical rules of English but arguably no human would correctly identify that meaning unless specifically prompted. A computer trained only on human comprehension, i.e. data which did not contain such outlier sentences, could plausibly identify its meaning.

¹⁰ Many recent language models do outperform the average human baseline on language understanding tasks, but Tedeschi et al. (2023) argue that for a variety of reasons the strength of these results is significantly exaggerated.

Computers could outperform humans on recognition tasks, but we haven’t tested them yet. Most benchmarks for media recognition, e.g. testing for object detection in photos or speech recognition in audio, use human labels as the ground truth. However humans can be mistaken: they might think a photo has a dog in it when it does not or vice versa. Thus human judgment is not the ground truth. We could create a test set to measure superhuman performance either by (1) creating new media (e.g. taking new photos of dogs instead of using existing photos which humans identify as having a dog); (2) obfuscating existing media (e.g. blurring existing photos of dogs). By the argument in this note a classifiers trained only on human-provided labels could exhibit superhuman performance in such a test set, most plausibly through better priors: learning characteristic signs of dogs that humans do not.¹¹

¹¹ Defining the ground truth in a recognition task is somewhat complicated because a given arrangement of pixels is consistent with an infinite variety of objects having caused that arrangement. We talk about an image representing an object in the world only because we have strong priors about the world which allow us to make that inference. So the ground truth in a recognition task must be something like “in ordinary circumstances, what is the probability that these pixels would be caused by a scene with a dog in them.”

Self-play has helped performance in playing games. A common trick to teach computers to play games well is to have them play themselves (self-play), this has been used to get superhuman performance in Backgammon, Chess, Go, & Minecraft. However this is not imitation learning: here the computer is trained against a non-human ground truth, the computer directly observes whether they have won the game. However there are analogues of self-play in imitative models: (1) training models to generate images, and to discriminate between computer-generated and real images (generative adversarial nets, GAN); (2) training an LLM to produce text based on LLM-generated feedback (RLAIF).

The graphical model underplays the importance of model architecture. The graphical model shown above represents both questions and answers as unidimensional, and it makes it seem that a small sample is sufficient to get a reasonably good approximation of the true function. In reality the questions of interest are very high dimensional and most model architectures fail to generalize well at all. Neural nets, especially those with a transformer structure, have had remarkable success in fitting the data, leading to leaps in performance.

Venn diagram representation. This diagram shows an alternative way of representing some of the core claims: in general the questions answerable by an LLM will not be a subset of the questions answered in the training set, or even those answerable by the people who contributed to the training set. The Venn diagram’s disadvantage, relative to the visualizations above, is that it does not represent the mechanics of why the LLM can outperform humans, while the diagram above can be use to separately show five distinct reasons (averaging error, specialization, interpolation, different priors, and using tacit knowledge).

Imitation learning has problems in dynamic situations. The discussion in this note has been about a purely static problem of supplying answers to questions, but text generation can also be considered as a dynamic problem of sequentially generating tokens. A common observation regarding dynamic imitation learning is that pure prediction of expert behaviour (“behavioural cloning”) is not very robust, because the algorithm does not know what to do in situations not observed before (out of distribution), and this has been used to explain weaknesses in the behaviour of autoregressive generative text models.¹²

¹² Cundy and Ermon (2023) say “[the] simple behaviour cloning approach results in a compounding error problem, where the further the trained model gets from the typical expert states, the worse the model performs, incurring increasing error.” I also found these notes from Stanford’s CS273B useful.

Linear Model

Here I give a more formal model and derive some results. I wrote this model before coming up with the graphical argument above. There is a substantial overlap in implications, but I think there is some value in this linear model in the precision with which we define each quantity. The model has three steps:

Questions and answers. A question is defined by a set of binary attributes (), and the answer is a linear function of those attributes given some unobserved weights :

Human beliefs. After observing a set of question and their real-world answers the human will form beliefs about the weights . We can explicitly write the human posteriors if we assume their priors are Gaussian and i.i.d. ():¹³

¹³ I am assuming has zero-mean and is i.i.d. just to cut down on notation, the results all hold for the more general multivariate Normal case.

I give a derivation of the human posteriors below.

I will assume that the number of unobserved weights is large relative to the human’s experience (), so the human will gradually learn more about reality as she observes the answer to more questions. She will be able to perfectly answer any question she’s seen before, but will never learn the full set of weights.

Computer beliefs. Suppose that humans write down some set of questions, , and then record their best guesses at the answers. These could be questions that the humans already know the answer to (), or they could be new questions that they are guessing the answer to. We then use those questions and answers to train a computer, and the computer likewise assumes a linear model with i.i.d. Gaussian weights. Note that the computer is being trained to predict human responses, not to predict properties of the world. We can write the computer-estimated weights as follows:

Computer answers. Finally we can ask the computer a new question, , and observe its answer:

Model Implications

If one human records all their observations then the computer will perfectly imitate them.: Suppose that there is one human and they write down all of their observations, . Because the computer and human have the same priors, and observe the same data, then they will therefore end up with the same estimated weights (), and so the computer will answer every question exactly as the human does, though neither knows the truth ().

If humans are noisy then the computer will outperform them. Suppose humans report their answers with some i.i.d. noise . If the computer observes sufficiently many answers for each question then the noise will be washed out and they will outperform.

If humans record a subset of their observation then the computer will perform worse.: Suppose humans only write down some of their observations, i.e. is a row-wise subset of . Then computers and humans will give the same answers for any question in the training set, but outside of that set computers will generally do worse than humans. And so for any question the computer will do worse in expectation: Note that we are fixing the question and taking the expectation over all possible worlds, . I think you could probably rewrite this such that, in the world we are in, we should observe worse average performance across a set of questions, but I think you’d need to add some conditions to make sure that the questions are sufficiently independent (e.g. if there was a single weight which dominated all the other weights then the computer might beat the human by accident).
If there are two humans then the computer will outperform them both.: Suppose there are two humans who each observe answers to different question, and , and they both write them all down, so and . Now the computer has a strictly larger set of observations than either human, and so if we let represent the weights of human , then for any question we can write:

If there are multiple humans then the computer can answer question no human can answer. Suppose two humans observe the answers to the following questions:

The first human will learn the exact value of (), and the second human will learn the exact value of , but neither will learn both values, and so neither could predict the answer to this question with perfect confidence:

However if they both recorded their observations then the computer observes , and so the computer will be able to infer both and , and thus will be able to perfectly answer the question above. We can see this behaviour in LLMs: they sometimes combine a pair of facts or a pair of abilities which no single human has access to, e.g. when an LLM translates between a pair of languages for which there exists no human speaker of both.

If humans write outside their expertise then the computer will do worse. In the cases above we assumed that the two humans recorded only what they directly observed, . This means the computer essentially had a window directly to the world. However the humans could instead have written down their estimated answers to other questions, for which they have never observed the ground truth. Suppose both humans wrote down answers to every possible question, , then we could conjecture that the computer would learn the average of the two humans’ weights:¹⁴ Here the computer will do worse than the two humans on the original questions, and . The implication is that LLMs work so well only because people tend to write about what they know. Put another way, when an LLM answers a question, it will not predict the answer given by the average person, but will predict the answer given by people who are likely to answer that question in the real world. Luckily there tends to be a positive correlation between having knowledge about a domain, and writing about that domain.

¹⁴ We would have to augment the computer’s learning rule to allow for noise in answers - I need to confirm that the weighting will be exactly 1/2.

¹⁵ This is related to the “generator-discriminator” gap, but specific to knowledge rather than to logical implication.

If humans have tacit knowledge, then computers can outperform in choosing a question to maximize the answer. We can model tacit knowledge with two separate sets of human weights:¹⁵

When the human encounters a new question they will use their tacit knowledge to form an estimate of the answer, . But they have limited ability to introspect about that capacity, and so when asked how they make their judgments they can report only . For simplicity assume tacit knowledge is perfect (), and explicit knowledge is imperfect ().

The distinction becomes important when we want to create a new . Here it’s useful to interpret as an artefact, e.g. a text or image, and interpret as a property of that artefact, e.g. how persuasive is the text, or how attractive is the image. Suppose we want to choose to maximize . If we had perfect access to our beliefs this would be simple, however if we have access only to imperfect explicit knowledge , the artefact which maximizes that function will not generally be the one which maximizes . This represents an asymmetry in human cognition: we can recognize certain patterns (whether text is persuasive, whether a picture is pretty), without being able to produce those patterns.

Here the computer model is less constrained. Suppose the computer has observed sufficiently many questions such that they have perfectly learned human tacit knowledge, . If computation is costless we could query every single to find the highest . In the real-world we use a diffusion algorithm, or reinforcement learning against human or computer evaluation, to find an artefact with a high .

Derivation

Setup.

Written more compactly:

Human posteriors. Given you observe a subset of a set of multivariate normal variables there is a simple expression for your posteriors over the remaining unobserved variables (e.g. see here).

We can use the same formula to calculate computer beliefs.

Additional Observations

These are a few miscellaneous results additional results that helped me with intuition for the working of this model.

With one observation and two weights. Suppose , then we have:

We can normalize , then we have: Here we are dividing up responsibility for the answer () into the contributions of each component, nice and simple.

With two observations and one weight. Here we’re over-identified. With noise. Suppose we only observe the answers with random noise, then we have Compare to Bayesian linear regression. We can compare this result to Bayesian linear regression (e.g. Wikipedia):

I believe that these can be shown to be equivalent by the matrix inversion lemma, though I haven’t confirmed this. There is a concise proof in an online note from Utah State University.

Extension: quadratic forms. Instead of answers being linear in question-features () we could suppose they’re quadratic, , with a matrix having dimension . I am not sure whether we could still get an analytic solution for posteriors. One way to visualise is that each bit in adds an “L” (a row and a column) to the matrix, and is the sum of the cells where both the row and the column are activated.

Extension: binary answers. In some cases it is natural to think of the answer, , as binary instead of continuous. We might be able to reinterpret the model with representing the log-odds ratio of a binary outcome. Alternatively there might be a way of having a beta-binomial conjugate prior over the probability of .

Who’s closer

Suppose two people, and , have observed different training sets and , then we can characterize their expected error for the answer of a new question (where , ):

Setup. Let the true weights be and let the two individuals observe . Their posterior mean is
and their posterior covariance is

Expected error for a new question . Both people answer the fresh question by , while the truth is . Conditioning on the training set we therefore have

Isotropic prior. If we specialise to then
with the orthogonal projector
onto the row–span of . Hence

In words, the expected squared error is exactly the squared length of the component of that is orthogonal to the set of questions that person has already encountered. It is zero if is contained in the span of their past questions and grows with the distance of from that span.

Who’s closer? Person is expected to be more accurate than precisely when
Thus the tie-breaker between the two forecasters is which training set provides a better projection of the new question .

References

Armengol-Estapé, Jordi, Ona de Gibert Bonet, and Maite Melero. 2021. “On the Multilingual Capabilities of Very Large-Scale English Language Models.” https://arxiv.org/pdf/2108.13349.pdf.

Bai, Hui, Jan G Voelkel, johannes C Eichstaedt, and Robb Willer. 2023. “Artificial Intelligence Can Persuade Humans on Political Issues.” OSF Preprints. https://doi.org/10.31219/osf.io/stakv.

Bowman, Samuel R. 2023. “Eight Things to Know about Large Language Models.” arXiv Preprint arXiv:2304.00612. https://arxiv.org/pdf/2304.00612.pdf.

Camerer, Colin, and Eric J. Johnson. 1991. “The Process-Performance Paradox in Expert Judgment - How Can Experts Know so Much and Predict so Badly?” In. https://api.semanticscholar.org/CorpusID:67971809.

Cundy, Chris, and Stefano Ermon. 2023. “SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking.” arXiv Preprint arXiv:2306.05426. https://arxiv.org/pdf/2306.05426.pdf.

Cunningham, Tom. 2015. “Hierarchical Aggregation of Information and Decision-Making.” Unpublished Manuscript, Columbia University. https://www.semanticscholar.org/search?q=Hierarchical%20aggregation%20of%20information%20and%20decision-making.

Cunningham, Tom, and Jonathan De Quidt. 2022. “Implicit Preferences.” http://jondequidt.com/pdfs/paper_implicit.pdf.

Goldstein, Josh A, Jason Chao, Shelby Grossman, Alex Stamos, and Michael Tomz. 2023. “Can AI Write Persuasive Propaganda?” SocArXiv. https://doi.org/10.31235/osf.io/fp87b.

Hackenburg, Kobi, and Helen Margetts. 2023. “Evaluating the Persuasive Influence of Political Microtargeting with Large Language Models.” OSF Preprints. https://doi.org/10.31219/osf.io/wnt8b.

Hendrycks, Dan, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. “Measuring Mathematical Problem Solving with the Math Dataset.” arXiv Preprint arXiv:2103.03874. https://arxiv.org/pdf/2103.03874.pdf.

Kiela, Douwe, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, et al. 2021. “Dynabench: Rethinking Benchmarking in NLP.” arXiv Preprint arXiv:2104.14337. https://arxiv.org/pdf/2104.14337.pdf.

Kiela, Douwe, Tristan Thrush, Kawin Ethayarajh, and Amanpreet Singh. 2023. “Plotting Progress in AI.” Contextual AI Blog. https://contextual.ai/blog/plotting-progress-in-ai.

Koivisto, Mika, and Simone Grassini. 2023. “Best Humans Still Outperform Artificial Intelligence in a Creative Divergent Thinking Task.” Scientific Reports 13 (1): 13601. https://doi.org/10.1038/s41598-023-40858-3.

Lee, Harrison, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. 2023. “RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback.” arXiv Preprint arXiv:2309.00267. https://arxiv.org/pdf/2309.00267.pdf.

MacDougall, Robert. 1904. “Recognition and Recall.” The Journal of Philosophy, Psychology and Scientific Methods 1 (9): 229–33. https://doi.org/10.2307/2010991.

Matz, Sandra, Jake Teeny, Sumer S Vaid, Gabriella M Harari, and Moran Cerf. 2023. “The Potential of Generative AI for Personalized Persuasion at Scale.” PsyArXiv. https://doi.org/10.31234/osf.io/rn97c.

Ouyang, Long, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, et al. 2022. “Training Language Models to Follow Instructions with Human Feedback.” Advances in Neural Information Processing Systems 35: 27730–44. https://www.semanticscholar.org/paper/d766bffc357127e0dc86dd69561d5aeb520d6f4c.

Palmer, Alexis, and Arthur Spirling. 2023. “Large Language Models Can Argue in Convincing and Novel Ways about Politics: Evidence from Experiments and Human Judgement.” Working paper), Technical report. https://www.semanticscholar.org/paper/9000acdfc1ba46ff93ceb9f601ca7234f32d0f8b.

Qin, Zhen, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, et al. 2023. “Large Language Models Are Effective Text Rankers with Pairwise Ranking Prompting.” https://arxiv.org/pdf/2306.17563.pdf.

Stiennon, Nisan, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2022. “Learning to Summarize from Human Feedback.” https://arxiv.org/pdf/2009.01325.pdf.

Tedeschi, Simone, Johan Bos, Thierry Declerck, Jan Hajic, Daniel Hershcovich, Eduard H Hovy, Alexander Koller, et al. 2023. “What’s the Meaning of Superhuman Performance in Today’s NLU?” arXiv Preprint arXiv:2305.08414. https://arxiv.org/pdf/2305.08414.pdf.

Towler, Alice, James D. Dunn, Sergio Castro Martı́nez, Reuben Moreton, Fredrick Eklöf, Arnout Ruifrok, Richard I. Kemp, and David White. 2023. “Diverse Types of Expertise in Facial Recognition.” Scientific Reports 13 (1): 11396. https://doi.org/10.1038/s41598-023-28632-x.

Zhang, Edwin, Vincent Zhu, Naomi Saphra, Anat Kleiman, Benjamin L Edelman, Milind Tambe, Sham M Kakade, and Eran Malach. 2024. “Transcendence: Generative Models Can Outperform the Experts That Train Them.” arXiv Preprint arXiv:2406.11741. https://arxiv.org/pdf/2406.11741.pdf.

Citation

BibTeX citation:

@online{cunningham2023,
  author = {Cunningham, Tom},
  title = {An {AI} {Which} {Imitates} {Humans} {Can} {Beat} {Humans}},
  date = {2023-10-06},
  url = {tecunningham.github.io/posts/2023-09-05-model-of-ai-imitation.html},
  langid = {en}
}

For attribution, please cite this work as:

Cunningham, Tom. 2023. “An AI Which Imitates Humans Can Beat Humans.” October 6, 2023. tecunningham.github.io/posts/2023-09-05-model-of-ai-imitation.html.

Sushi-Roll Model of Online Media

Tom Cunningham, [Integrity Institute](https://integrityinstitute.org/) — Fri, 08 Sep 2023 07:00:00 GMT

A model of internet media: the platform chooses the composition, the user chooses the quantity. I think this is a nice crisp way of modeling how media platforms (FB, YouTube, TikTok) make their decisions about content: they chooses the mix of content, i.e. the shares of each type, and then their users choose the quantity. The platform is choosing the fillings for the sushi roll and the consumer is choosing how much to eat. Their decisions jointly determine the total amount of each ingredient consumed.

This gives a unified model of feed ranking inclusive of ad-load, revenue-sharing, producer-side effects, and advertiser demand elasticity.

We can break down four different ways in which increasing the share of a given content-type will affect total revenue:

Consumption incrementality. The effect on total impressions, AKA incrementality. In general platforms are always trying to find the types of content that increase total long-run consumption for a given user.
Price. The price received by the platform for each impression of content-type . For ads this is positive, but platforms also sometimes license content or pay a revenue-share, in which case the price is negative.
Price elasticity. The effect of quantity on price. E.g. when we increase the number of ad impressions shown this will lower the market price for all other ad-impressions, and so this is a reason to limit the quantity of ads.
Production elasticity. The effect on production by producers. Some producers will produce more content when they receive more impressions, and so this is an additional benefit of increasing their share of impressions.

An efficient mix of content will choose the shares such that each type of content has the same marginal value, i.e. for each the type four components of value all sum to the same number.

Related literature. There are some nice models of ad-media tradeoff in Anderson and Jullien (2015), but I believe they don’t consider the effects on production by producers, nor the tradeoff between different types of content (though it’s a long time since I read it).

Expressed formally: Suppose the platform chooses which represent the impression-shares of each type of content such that . User demand depends on the average quality of each type of content (), and they have diminishing returns in each type of content. The platform receives for showing an impression of type , but that price depends on the number of impressions-seen. We can write the maximization problem as:

The first order condition shows us the four components of value:

The final term, , is the Lagrangian, representing the average marginal value of the outside option, i.e. the other types of content that are being replaced. In some cases we can simplify this model and we get a closed-form solution for the optimal content composition.

What this model doesn’t include:

The effect of consumption on production. E.g. we sometimes want to reduce ad-load to increase production by users, or we want to show users friend posts because it increases production (“mimicry”). You could incorporate this by defining as the share shown from producer to consumer , and then each user’s production depends both on (1) how many impressions they get on their content; (2) composition of impressions that they give to other content.
Setting a price for production (revenue sharing). In this model the platform’s choice variables are just the quantities, . However for revenue-sharing it seems that the platform is setting a price, with the goal of increasing producer quality. I don’t think you can model this such that price is a function of quantity (as we do with ads). I think you need to keep track of two separate things: (1) what happens when I give this producer more distribution; (2) what happens when I pay this producer to produce.

More Details

We can walk through a series of models from simple to complicated, to build up to the full sushi-roll model:

Platform chooses share of ads. The platform chooses the share of impressions that are ads, and consumers choose how many total impressions to consume. If we assume the price of ads (CPMs) is fixed then the platform will set the ad-load to maximize the total number of ad-impressions. If the platform can influence the price of ads by their choice of quantity (i.e. they act as a monopolist) then the platform may choose to reduce ad-load to drive up CPMs.¹

In this model we’re letting the platform set the quantity of ads, but we would get the same result if the platform instead set the price of ads, e.g. they posted a specific CPM and advertisers can buy as much as they want.
Platforms chooses shares of organic content. Suppose ad-load is fixed but platforms can vary the shares of different types of organic content. Users’ consumption depends on the quality of the content but they also have a taste for variety (i.e. diminishing returns in each type of content). We then get a nice closed-form solution where the share of each type of content is increasing in its relative quality. On the margin the incrementality of each type of content will be zero: i.e. increasing the share of that type of content will have no effect on total impressions.
Platform choses shares of ads and organic content. Now lets treat both advertisers and organic producers as the same: each producer has a quality but they also will pay a certain price for impressions on their content. The platform takes those prices as given. We can then distinguish between three types of producer:
- Advertisers: : the producer will pay the platform per impression.
- Professional producers: : the producer asks to be paid per impression.
- Amateurs: : there is no monetary exchange, the content is in the public domain or generated by an ordinary user.
In equilibrium the share of impressions allocated to a given producer () will depend both on its quality (AKA incrementality) and the price that the producer sets. I don’t have a closed-form solution but we can derive a first-order condition that has a straight-forward interpretation.

This model is easy to state but I think is primarily applicable to small platforms where they take prices as given. E.g. suppose you run an app where you (1) license certain content, or use user-generated content; (2) run ads from various different ad networks, the ads vary in CPMs but they also vary in incrementality (i.e. how obnoxious they are to your userbase).

Because prices are taken as given, this model doesn’t help us calculate the optimal revenue share. It will tell us the optimal ad-load, but not taking into account the elasticity of supply from the advertiser.

Note that most platforms do not explicitly discriminate between advertisers based on their incrementality however they can implicitly discriminate by having a “quality score” or “organic bid”. This score is quite clearly designed to measure the incrementality of the advertisements, and so I think can be used to implement an efficient pricing scheme.
Platform choses shares of ads and organic content, prices endogenous. We can easily extend the model above to allow the price of each type of content to depend on the quantity shown (). For example the price of ads will depend on the number of ad impression shown (due to advertisers’ diminishing marginal returns from ads shown). This gives platforms a reason to restrict the quantity of ad-impressions.
Platform chooses both shares and prices. Up to this point the platform chose only the share of each type of content.

Now suppose the platform can set a price to pay producers (e.g. “revenue share”), and it’s a homogenous price. The price should roughly depend on (1) the producer’s elasticity of quality to price (i.e. their cost function), and (2) the incrementality of quality on the consumer side. We could set this up with a single price for all producers, or set a producer-specific price.

¹ In the profit-maximizing solution either the consumer-side or advertiser-side first-order-condition will be binding. It depends on the relative elasticity of the two sides of the platform.

Model 1: Platform Chooses Ad-Load

(see previous paper)

Model 2: Platform Chooses Organic Composition

We have a model where there are producers, the platform assigns to each producer a share of total content , with , and the consumer will choose how many total impressions to consume () based on the average quality, but with diminishing returns in each .

The platform wishes to maximize total impressions, . We want to solve for the resultant impression-share of each producer, i.e. as a function of the qualities and parameter . We get the following impression-maximizing shares:

Implication: impression-share will be proportional to quality. Interesting the elasticity will be increasing in quality: a 1% increase in quality will get a more than 1% increase in share of impressions, because .

Adding money. Suppose now that the platform gets paid for showing certain impressions. We can make various different assumption about the price paid:

Uniform homogenous price: the platform takes the price as given. This only makes sense if there are a subset of producers who are advertisers.
Each producer sets a payment rate per impression.
The platform chooses a single price for all producers to get extra impressions.

Model 3: Platform Chooses Composition, Prices Fixed

We have a model with a consumer, a platform, and a set of producers. The platform chooses the share of content from each producer, with . The consumer chooses the total amount of impressions they consume, , based on the mixture of content and the quality of each type of content . Finally producers can set a price for each impression that they receive from the consumer. A positive price means

If the platform simply wanted to maximize total impressions, , then they can derive the optimal impression-shares as follows:

However we want the platform to maximize profit, which we can write as follows:

I’m not sure if we can get a closed-form solution but we can at least get a first-order condition for each that tells us useful stuff about comparative statics:

Observations:

If producers offer more, increasing , then will go down until the marginal effect on total impression declines to balance the additional revenue.
If , meaning a producer charges for impressions, then they can still have a positive number of impressions if their effect on total impressions is higher than the average of other types of content. (We could have added an additional constraint that .)

Model 4: Platform Chooses Composition, Monopolist

Now we allow the price of each type of content to depend on the quantity used, e.g. the price of ads will be higher when the quantity of ad-impressions is smaller (monopolist in the ad market). Strictly we should write , but it’s somewhat easier to write and the answer should be similar for any type of content that is a small share.

There’s now one additional term in the first order condition:

The additional term represents the platform’s monopoly power with respect to the price paid. This has a natural interpretation for advertisers: showing fewer ads will drive up the price. For paid content-providers it could perhaps represent bulk discounts, I’m not sure whether this is a significant consideration.

Model 2 Derivation

This is derivation of model #2. (I had chatGPT help with this derivation, was very useful)

Setting up the Lagrangian. The objective is to maximize the total impressions, , subject to the constraint that the allocated shares of impressions sum to one. We start by writing the Lagrangian:

where is the Lagrange multiplier associated with the constraint.
Solving for the multiplier. To solve for the value of , we take the derivative of the Lagrangian with respect to and set it equal to zero:

Rearranging this equation yields:

Taking the sum of this expression over all producers and using the constraint that the shares of impressions must sum to one, we obtain:

Simplifying this equation gives:

Substituting this expression for back into the equation for results in:

This is our final expression for the share of impressions on each producer as a function of the exogenous qualities and .

Summary:

The Lagrangian: .

Expression for the multiplier:

The resultant expression for :

Slightly rearranged (by me):

References

Anderson, Simon P., and Bruno Jullien. 2015. “Chapter 2 - the Advertising-Financed Business Model in Two-Sided Media Markets.” In Handbook of Media Economics, edited by Simon P. Anderson, Joel Waldfogel, and David Strömberg, 1:41–90. Handbook of Media Economics. North-Holland. https://doi.org/https://doi.org/10.1016/B978-0-444-62721-6.00002-0.

How Much has Social Media affected Polarization?

Tom Cunningham, [Integrity Institute](https://integrityinstitute.org/) — Mon, 07 Aug 2023 07:00:00 GMT

TL;DR: The experiments run by Meta during the 2020 elections were not big enough to test the theory that social media has made a substantial contribution to polarization in the US. Nevertheless there are other reasons to doubt it.

Summary

Thanks to Dean Eckles, Solomon Messing, Jeff Allen, & Brandon Silverman for discussion which led to this post. I put together the spreadsheet summary of results with Dean and Solomon. See also a post by Dean.

¹ Allcott et al. (2019), see below for discussion of whether these standard deviations are comparable.

Three new experiments show that changing Facebook’s feed ranking algorithm for 1.5 months has an effect on affective polarization of less than 0.03 standard deviations. This is small compared to a growth of 1.1 standard deviations in nationwide affective polarization over the last 40 years.¹

Small effects in these experiments are consistent with large effects in aggregate. Guess et al. (2023b) says:

“these findings suggest that social media algorithms may not be the root cause of phenomena such as increasing political polarization.”

However they aggregate contribution of social media to polarization will differ from these experimental estimates in a number of ways: depth, breadth, duration, timing, category, and population. My rough attempts to account for these considerations make me think the aggregate effect is likely 10 or 20 times larger than the effects that would be measured in these experiments, and so small effects in these experiments are consistent with large effects on aggregate.

Put simply: these experiments measure the effect of reducing exposure of an individual user (not their friends and family) to political content on Facebook by 15% for 1.5 months, and occurred in a period after Facebook had already sharply reduced the amount of partisan content circulating. Thus we should expect them to measure only a small fraction of the cumulative impact of social media, and in fact these results are consistent with social media being entirely responsible for the growth of polarization in the US.

Nevertheless other evidence implies that social media has probably not made a huge contribution to US polarization. If we wish to evaluate the balance of evidence relating social media to polarization there are many other sources which are probably more informative than these experiments. I give a rough sketch below and it seems to me social media probably does not account for a majority share, mainly because (1) polarization had been growing for 20 years prior to social media’s introduction, and much of the growth since 2014 was in people without internet access; (2) a lot of partisan discourse continues to spread outside of social media, e.g. through cable TV and talk radio; (3) other countries do not show a similar increase in affective polarization.

Discussion of these results has been distressingly non-quantitative. The majority of discussion of these results (in papers, editorials, on Twitter) has been about whether these changes “have an effect” or “do not have an effect.” Interpreted sympathetically these statements are compressed ways of saying “an effect larger than 0.03 standard deviations.” However I think taking this shortcut so consistently has led to far too little time thinking about what we have learned from these experiments that we didn’t already know, and what is the balance of evidence regarding the effects of social media. I give a lot of examples below.

The Experiments

Last week’s papers reported the results of three experiments on Facebook’s News Feed. The experiments (Guess et al. (2023b), Guess et al. (2023a), Nyhan et al. (2023)) were run between September and December 2020, and half-way through participants were asked about their feelings towards members of their own party and the opposing party, e.g. “how warm do you feel about Republicans on a scale of 0-100?”² The answers were aggregated to make an index of “affective polarization”:

² Although the treatments ran for 3 months (24 Sep–23 Dec 2020), the survey responses were collected during the experiment and the average survey measure was measured after around 1.5 months of treatment: see Figure S2 in the Supplementary Appendix.

All three experiments found effects on polarization of less than 0.03 standard deviations (SDs). The 95% confidence intervals on affective polarization are approximately 0.03 SDs, and the effect-sizes are all smaller than that (i.e. they do not estimate a significant effect). Dean Eckles, Solomon Messing, and myself put together a spreadsheet summary of the results from all the experiments reported so far, along with other results from the literature on political effects of media.

They also measured effects on a number of other off-platform outcomes: removing reshares did lower news knowledge by 0.07 standard deviations, but all other outcomes (factual discernment, issue polarization, perceived legitimacy, self-reported turnout) were not significant, and had similar-sized confidence intervals.

Extrapolating to the Cumulative Effect of Social Media

Many people have interpreted these results as implying that social media has not had much effect on overall polarization. E.g. one of the experimental papers says:³

³ Guess et al. (2023b)

“these findings suggest that social media algorithms may not be the root cause of phenomena such as increasing political polarization.”

Here I try to extrapolate from these experiments to the long-run aggregate effect of social media. The comparison is between two extremes but there are a lot of other intermediate estimands that we could alternatively use, e.g. the effect of permanantly disabling just Facebook for everybody, or the effect of temporarily disabling all social networks for an individual user.

These are difficult judgment calls. I have tried my best to be neutral and discuss evidence on either side but it’s likely I’m forgetting some important considerations.

I work through six ways in which the experimental results will differ from the aggregate impact on social media:

Depth. Whether changing one feature or disabling the app entirely.
Breadth. Whether changing the experience for one user or for all users.
Duration. Whether changing the experience for 1.5 months or for the whole history of social media.
Timing. Whether changing the experience in Oct 2020, or the average effect over 2004-2020.
Category. Whether changing the experience just for Facebook or for all social media.
Population. Whether we are estimating the effect for all US adults or just Facebook users.

I try to give quantitative estimates for each of these six differences, and it makes me think that having tight confidence intervals on the effects of the experiments (plus or minus 0.03 SDs) is still consistent with the aggregate effect of social media being having an effect as large as 1 SD or more.⁴

⁴ I am considering the effect-size rather than the uncertainty, you could separately do a similar exercise to propagate up the uncertainty, and I think this would make the experiments seem even more under-powered to measure the aggregate effect.

(1) Depth: the experiments have small effects on exposure. Each of the experiments reported have effects on overall Facebook time-spent of less than 25%, and on exposure to political material of less than 15%. Thus the effect of complete withdrawal from Facebook seems likely to be at least 2X larger than measured by any of these experiments. The most natural causal path from Facebook use to polarization is exposure to partisan or misleading political media. An additional experiment was run which deactivated peoples’ accounts but the results from that experiment are not yet public (as of August 4).

effects on metric⁵	time-spent	political impression	cross-cutting impressions	untrustworthy impressions
baseline	?	14pp	21pp	3pp

- rank chronologically	-21%	+12%	-10%	+60%
- remove reshares	-5%	-14%	-3%	-32%
- downrank likeminded posts	-1%	-5%	+7%	?

⁵ source data.

(2) Breadth: Experiments exclude network effects. The effects of social media on polarization likely work not just through direct exposure but also downstream via peoples’ interactions with friends and families, in both online and offline conversations. Thus it seems likely the aggregate effect could be 2X or larger than the individual effect.

Causal evidence on peer effects on attitudes is notoriously difficult to find. It’s worth mentioning that (1) the simplest rational model of beliefs implies that peer effects are overwhelmingly strong⁶; (2) cross-sectional variation in attitudes typically shows large between-group variation relative to within-group, consistent with strong peer effects: e.g. those brought up catholic tend to be catholic, those brought up protestant tend to be protestant.

⁶ In the simplest model with common knowledge of rationality everyone will converge to the same belief, effectively pooling their information. Thus in a world of 7B people the weight each person gives to their personal experience would be 1/7B and the remainder is peer effects. The calculation becomes slightly more complicated when there are common sources of information.

⁷ The polarization survey measures were collected in wave 3, 4, and 5. From eyeballing Figure S3 (and assuming the response rate declines over time) it appears the average response would be collected around 1.5 months after the treatment began: .

⁸ The only dynamic effects discussion I could find was wave-by-wave results for survey questions in the supplementary appendices. I think it would be useful to show the dynamic effects for on-platform behavior because (1) there is good reason to expect the cumulative treatment effects will dramatically change over time, (2) the experiments are sufficiently well-powered that dynamic effects should be easy to observe.

(3) Duration: the experiments only measure short-run effects. These experiments measured the effect of a News Feed change on polarization after around 1.5 months, while most American adults have been using Facebook for perhaps 10 years.⁷ It is hard to judge how quickly we should expect polarization attitudes to respond to treatment, and I have not found useful academic literature. The national polarization trends documented in Allcott et al. (2019) seem fairly stable despite a volatile news cycle suggesting attitudes change relatively slowly. If the half-life of adjustment was 1.5 months (which seems quite short to me) then the effects measured in these experiments would be half of the long-run effect. It seems likely that the effects of exposure do not decay at a constant rate: there is a short-run component that decays quickly (the effect of salience), and a long-run component that decays slowly.⁸

Guess et al. (2023b) notes this limitation:

“It is possible that such downstream effects require a more sustained intervention period … although our approximately 3-month study had a much longer duration than that of most experimental research in political communication.”

Although if I understand correctly this is slightly misleading: the outcome variables were measured after 1.5 months exposure, not 3 months.

(4) Timing: Experiments were run during the lead-up to the 2020 election. The experiments ran between September and November 2020. If we compare this to the average experience on Facebook over the previous decade there are reasons to expect relatively smaller effects on polarization:

Following the 2016 elections facebook invested very heavily in integrity systems reducing prevalence of many types of bad content by factors of between 2X and 10X, especially misinformation and hyperpartisan political content. In May 2020 Guy Rosen claimed that Facebook had made “a number of important steps to reduce the amount of content that could drive polarization on our platform” over the prior years.
Allcott et al. (2019) estimates that exposure to misinformation on Facebook (measured by data on engagement with domains known to host misinformation) grew over 2015 and 2016, roughly doubling, then fell over 2017 and 2018, roughly halving. The data ends at the end of 2018 but I believe the trend would continue downward.
Meta’s Community Standards reports show a decline in prevalence of most types of harmful content by a factor of between 2 and 5 over roughly 2017 to 2022 (see chart here).
Prior to and during the 2020 election Facebook implemented a series of extra “break the glass” measures with the effect of suppressing extreme or fringe political content.
During election seasons there tends to be significantly more political content circulating. This might mean that Facebook would have a relatively larger impact on polarization in this period. However if the influence of social media on polarization depends on the share of exposure to partisan or polarizing content (rather than the level) then the effect would be the same in election season as outside election season.

In fact 2020 Facebook has further reduced the prevalence of political and fringe content since 2020:

The share of politics on News Feed was reduced by 50%.⁹
Prevalence of hate-speech fell by factor of 5 between 2020 and 2022 (from 0.1% to 0.02%). (ref).
Engagement on US right-wing politics pages has fallen by factor of 4 from 2021-2022 (ref).
Prevalence of engagement-bait among the top 20 most-viewed posts went from 100% to 5% between 2021Q3 to 2022Q3 (ref).

⁹ The WSJ reported that in late 2021 “Mr. Zuckerberg and the board chose the most drastic [option], instructing the company to demote posts on “sensitive” [(politics and health)] topics as much as possible in the newsfeed that greets users when they open the app”, and that in 2022 “politics accounts for less than 3% of total content views in users’ newsfeed, down from 6% around the time of the 2020 election.” The article reports that these experiments reduced daily visitation (daily active users) by 0.2%.

(5) Category: The experiments affected only Facebook, but in 2020 Facebook probably accounted for around 1/4 of all partisan political content that people are exposed to on social media. If we include YouTube, TikTok, Instagram, Twitter, Snapchat, Reddit, and the long tail of niche social networks. In contrast, if we are estimating the cumulative effect (2004-2020) then Facebook would likely comprise a significantly larger share of exposure political content.

(6) Population: The experiments measure outcomes only on Facebook users. I believe that the “population average treatment effects” reported in the papers are weighted to match the Facebook-using population, not the voting population. This would be a reason for the experimental effect-size to be larger than the aggregate effect. I would guess around 2/3 of the US adult population is active once/month on Facebook, and so the aggregate effect-size could be smaller by that factor.¹⁰

¹⁰ The supplement to the Science paper mentions Facebook had 231 million monthly active users.

Putting it all together. If factors 1-5 each contributed a 2X amplification, as well as factor 6 contributing a 2/3 shrinkage, then the cumulative effect of social media on polarization until 2020 would be 20X larger than the experimentally-measured effect, i.e. effective confidence intervals would be 0.6 SDs instead of 0.03 SDs. In other words these experiments would not be sufficiently well-powered to rule out social media being responsible for the entire growth of polarization since 2014.

Note: adjusting for standard deviation size. The papers all report effect sizes on affective polarization in units of standard deviations. I wasn’t sure whether these are standard-deviations of the cross-sectional variance, or the residual variance after controlling for pre-treatment values. If the latter then I would guess they are perhaps half the size of the cross-section SD, based on my professional experience (and Supplementary Appendix page S-140). If correct this would halve the estimated effect-size when expressed in cross-sectional standard deviations, i.e. it would close the gap by a factor of 2.

Other Evidence on Media and Polarization

Here is a rough sketch of the evidence of related to affective polarization. I do not consider myself an expert on this literature and I would love corrections or additions. On balance this evidence seems to imply that social media hasn’t been the primary contributor to US affective polarization, but I think a thorough analysis of this evidence would be really valuable.

News Sources: From Pew data I would guess social media is around 25% of all exposure to political news, probably a higher share of exposure to partisan political news. Cable TV and political talk radio probably account for similar shares of overall exposure to partisan media. This seems the strongest evidence that social media is not the primary driver of affective polarization.
Professional Opinion: The political science literature talks about “the paradox of minimal effects” and the economics-of-media literature generally seems to have a consensus that most persuasive effects of media are small. However this might just apply to marginal effects.
Other Experiments: Allcott et al. (2020) is often interpreted as finding an effect on affective polarization but it does not (see below). Broockman and Kalla (2022) finds a null effect. I don’t know of other good experiments on affective polarization.
National Trends: In the US affective polarization steadily grew 1978-2020, for a total of 1.1 SD over 40 years. Other countries do not show a consistent trend, and there is no clear connection with internet access or online news consumption.
Demographic Trends: Over 1996-2012 affective polarization grew the most in groups who have not increased their internet access. I’m not aware of more recent data.
Natural Experiments: Some papers find that roll-out of mobile internet is associated with increased support for populist parties, to a degree that mobile internet could account for perhaps 1/3 of their total support

Discussion

Trends in affective polarization. Boxell et al. (2022) document affective polarization across a dozen countries, 1978-2020:

In the US affective polarization index increased from around 25 to 50, “an increase of 1.08 standard deviations as measured in the 1978 distribution.” (I’m not sure if the SD increased).
Across the world there’s no clear trend: some countries increased, other countries decreased. This weakens the simple argument that polarization has increased at the same time as social media use.
In the US the trend seems to be almost entirely due to increasing negative feelings about the opposing party:

The US timeseries can be seen online from the ANES.

Growth in populist support. Across the world there has been a substantial growth in populist governments. Guriev et al. (2021) and Manacorda et al. (2023) both argue from natural variation in mobile internet expansion that the internet has caused perhaps 1/3 to 1/2 of the increase in populist support in Europe.

Observational data finds that much of the growth in polarization in the US was among people who were not online. Boxell et al. (2017) say

“the growth in polarization in recent years [1996-2012] is largest for the demographic groups least likely to use the internet and social media”

Content on Meta platforms. Guess et al. (2023b) has data from the control group in their 2020 experiments:

Share of Impressions	Facebook	Instagram
Political content	14%	5%
Political news content	6%	-
Content from untrustworthy sources	3%	1%
Uncivil content	3%	2%

Pew 2022 has data on where people get their news from:

	pct adults regularly get news from
television	65%
news websites	63%
search	60%
social media	50%
radio	47%
print	33%
podcasts	23%

Radio show popularity. Around half of the top 20 most-listened radio shows in the US are conservative talk, with around 90M weekly listeners (this is double-counting overlapping users). Data from 2021.

Television. Fox News is Cable TV’s most-watched network with around 5M regular viewers. (source from 2016).

Time spent on social media. Statista: Average time-spent 150 minutes/day/person on social networks

The academic literature has identified other possible causes of polarization. Some potential causes: southern realignment, 1968 changes to the primary system, the Obama presidency, the tea party movement (though each of these could be in part proximal causes). Martin & Yurcoglu (2017) argue that a large part of recent growth is due to cable news: > “the cable news channels can explain an increase in political polarization of similar size to that observed in the US population over [2000-2008]. … In absolute terms, however, this increase is fairly small.”

See also Haidt and Bail’s long document Social Media and Political Dysfunction: A Collaborative Review

Does Allcott et al. (2020) find that Facebook use increases polarization? This paper reports on an experiment paying people to stop using Facebook for a month. They find an effect of -0.16 SDs ( 0.08) on a measure they describe as “political polarization,” however there are some subtleties:

Unlike the questions used in typical population surveys the questions were explicitly about their feelings during the period of the experiment, e.g. “Thinking back over the last 4 weeks, how warm or cold did you feel towards the parties and the president on the feeling thermometer?”
Polarization is measured by a composite of different measures. By far the largest effect was on the “congenial news exposure” question: “over the last 4 weeks how often did you see news that made you better understand the point of view of the Democrat (Republican) party?” The score was the difference between the answer for their own party vs the other-side party. It seems to me that it’s not surprising that deactivating Facebook would affect one’s exposure to such news, but that this wouldn’t normally be called a measure of “polarization” in the literature. The paper mentions in a footnote that “the effect on the political polarization index is robust to excluding each of the seven individual component variables,” but it turns out that removing “congenial news exposure” halves the effect-size and shifts the p-value from 0.00 to 0.09 (i.e. from very significant to non-significant). I’m not sure I would describe this as evidence of robustness.
The paper finds no significant effect on their two “affective polarization” measures (-0.08 0.08 SD, and 0 0.04 SD), however the Meta2020 papers which cite Allcott et al. (2020) seem to treat it as finding that Facebook has a positive effect on “polarization” without noting that it has a null effect on affective polarization.

Literature Review

Zhuravskaya et al. (2020) review literature on the political effects of the internet and social media. Regarding politics they say:

“The spread of the internet and social media has contributed, at least in part, to the electoral success of populists in Europe and to reduced political support for the ruling parties in immature democracies and semiautocratic regimes. There is also evidence that social media can be used to mobilize voters.”

Regarding polarization they say:

“the available evidence so far is not conclusive about whether social media increases political polarization.”
Guriev et al. (2021) uses 3G roll-out worldwide 2008 to 2017, they find a 40pp increase in mobile internet use causes:

“reduced the confidence in the national government of the region’s population by 2.5 percentage points (from the mean level of 51%), and increased the perception that the government is corrupt by 1.4 percentage points (from the mean of 77%).

They estimate within Europe that a 50pp increase in 3G access caused a 5% drop in support for incumbents, a 5% increase in support for right-populist parties, and a 4% increase for left-populist parties.
Melnikov (2021) finds that 3G roll-out in the US (1) increased the rates of self-identified ; (2) increased polarization in voting “the vote share of Republican candidates increases by 4.5 percentage points in Republican-leaning counties and decreases by 2.6 percentage points in Democratic-leaning counties.” The paper says “3G network coverage can account for 11.3% of the increase in polarization in political views, 37.7% of the increase in polarization in voting behavior, and, on average, 34.8% of the increase in polarization in policy preferences.” The paper calculates a “persuasion rate” of around 10, meaning I think that around 10% of people who are exposed to 3G internet are persuaded.
Manacorda et al. (2023) study roll-out of mobile internet across Europe 2007-2017, they estimate: > “between one third and one half of the remarkable success of communitarian [populist] parties, which roughly doubled their support over the period, can be ascribed to enhanced access to mobile Internet technology.”

Quantitative vs Qualitative Description of Results

Throughout these papers and in the public discussion the findings have been described in qualitative terms: i.e. either as “positive,” “negative,” or “neutral.” Implicitly these terms are referring to whether the results are statistically-significant (p<0.05), which depends on whether the effect-size is bigger than the confidence interval. These statements only make sense given some implicit understanding of how broad the confidence intervals are, yet I do not think that implicit understanding exists: I’m fairly confident that most people reading these statements (and many people making them) do not know quantitatively what the thresholds are.¹¹

¹¹ It’s worth stating that all of these treaments will have some non-zero effect, so it’s never literally correct to say “this treatement has no effect on polarization,” it can only be understood as a roundabout way of saying “this treatment has a small effect” for some definition of “small”.

Titles and abstracts used qualitative descriptions. The titles and abstracts all used qualitative language, e.g. “did not reduce” or “did not significantly affect” or “had no measurable effects.” None of the abstracts of the papers gave information on the size of the effects that were ruled out.
Hypotheses used qualitative descriptions. The pre-analysis plans contained a series of hypotheses, e.g.:

H1: Decreased exposure to content shared by like-minded friends, Pages, and groups decreases affective polarization.

H1: Reverse chronological feed will reduce polarization and negative perceptions of out-groups.

The terms “decrease” and “reduce” are presumably implicitly referring to the width of the confidence intervals, but I could find no discussion of how much .
Public discussion used qualitative descriptions. Almost all discussion in editorials and on Twitter described the results in qualitative terms, whether there was an effect or not, not in quantitative terms.
Elicitation of priors used qualitative descriptions. I was at an SSRC conference a few days before the results were released and there was a poll taken to predict the results. For “polarization” the options were (as I recall) “no effect”, “small increase”, “substantial increase”, etc., where I believe “increase” was intended to be interpreted as “statistically significant increase.” However as I recall we were not told the width of the confidence intervals when asked to make predictions. I think this is a bad way of eliciting priors: whether something is significant depends on the width of the confidence intervals as much as the effect-size. Thus an equivalent way of phrasing the question would be “do you think these experiments are sufficiently powered?”¹²
Power calculations used qualitative descriptions.

Guess et al. (2023b) said that there was sufficient power to detect “small” effects, without explaining why they regarded 0.03 SD as small.¹³ The supplement and pre-analysis plan do not mention “power” or seem to discuss the quantitative interpretation of these effect-sizes.

They do cite a previous paper:

“In all cases, we could rule out effect sizes smaller than those found in previous research [citation to Allcott 2020]

However I beleive this is a misinterpretation: Allcott et al. (2020) does test for affective polarization but they find a non-significant effect. As discussed above that paper reports a significant effect for “polarization” but the significance is due solely to the response to asking people about “congenial news exposure” over the last 4 weeks, which I think is quite different from polarization.

The Supplementary Appendix to Guess et al. (2023b) says the sample size was chosen to detect an effect size of 1.5 percentage points in vote choice (p. S-139), however it is not clear why this effect size was chosen.

Guess et al. (2023a) says explicitly that they (or someone) expected a significant effect:

“Contrary to expectations, the treatment does not significantly affect political polarization or any measure of individual-level political attitudes.”

However I did not find a discussion of why they expected an effect of that size.

¹² A similar phenomenon occurs in forecasting: if someone asks you a question for which you miss crucial context like “what is the chance of the Grockles winning the Kaplooey cup?” then you can give a good answer but it will be based on your judgment of the person asking the question, not your judgment about the substance of the question itself.

¹³ “The large samples … allowed for adequate statistical power to detect small effects (for example, for affective polarization, we were powered to detect population average treatment effects with Cohen’s d = 0.032 or larger for both Facebook and Instagram).”

Good quantitative work. Some of the authors of these 2020 papers have written other papers which I think use a much more useful approach: they use observational data, are quite focussed on quantitative outcomes, and they perform back-of-the-envelope calculations to reconcile evidence from different sources, e.g. Boxell et al. (2022), Boxell et al. (2017), Allcott and Gentzkow (2017).

References

Allcott, H., Braghieri, L., Eichmeyer, S., Gentzkow, M., 2020. The welfare effects of social media. American Economic Review 110, 629–676.

Allcott, H., Gentzkow, M., 2017. Social media and fake news in the 2016 election. Journal of economic perspectives 31, 211–236.

Allcott, H., Gentzkow, M., Yu, C., 2019. Trends in the diffusion of misinformation on social media. Research & Politics 6, 2053168019848554.

Boxell, L., Gentzkow, M., Shapiro, J.M., 2022. Cross-Country Trends in Affective Polarization. The Review of Economics and Statistics 1–60. https://doi.org/10.1162/rest_a_01160

Boxell, L., Gentzkow, M., Shapiro, J.M., 2017. Greater internet use is not associated with faster growth in political polarization among US demographic groups. Proceedings of the National Academy of Sciences 114, 10612–10617.

Broockman, D., Kalla, J., 2022. Consuming cross-cutting media causes learning and moderates attitudes: A field experiment with fox news viewers. https://doi.org/10.31219/osf.io/jrw26

Guess, A.M., Malhotra, N., Pan, J., Barberá, P., Allcott, H., Brown, T., Crespo-Tenorio, A., Dimmery, D., Freelon, D., Gentzkow, M., 2023a. Reshares on social media amplify political news but do not detectably affect beliefs or opinions. Science 381, 404–408.

Guess, A.M., Malhotra, N., Pan, J., Barberá, P., Allcott, H., Brown, T., Crespo-Tenorio, A., Dimmery, D., Freelon, D., Gentzkow, M., González-Bailón, S., Kennedy, E., Kim, Y.M., Lazer, D., Moehler, D., Nyhan, B., Rivera, C.V., Settle, J., Thomas, D.R., Thorson, E., Tromble, R., Wilkins, A., Wojcieszak, M., Xiong, B., Jonge, C.K. de, Franco, A., Mason, W., Stroud, N.J., Tucker, J.A., 2023b. How do social media feed algorithms affect attitudes and behavior in an election campaign? Science 381, 398–404. https://doi.org/10.1126/science.abp9364

Guriev, S., Melnikov, N., Zhuravskaya, E., 2021. 3g internet and confidence in government. The Quarterly Journal of Economics 136, 2533–2613.

Manacorda, M., Tabellini, G.E., Tesei, A., 2023. Mobile internet and the rise of communitarian politics. Centre for Economic Policy Research.

Melnikov, N., 2021. Mobile internet and political polarization. Available at SSRN 3937760.

Nyhan, B., Settle, J., Thorson, E., Wojcieszak, M., Barberá, P., Chen, A.Y., Allcott, H., Brown, T., Crespo-Tenorio, A., Dimmery, D., Freelon, D., Gentzkow, M., González-Bailón, S., Guess, A.M., Kennedy, E., Kim, Y.M., Lazer, D., Malhotra, N., Moehler, D., Pan, J., Thomas, D.R., Tromble, R., Rivera, C.V., Wilkins, A., Xiong, B., Jonge, C.K. de, Franco, A., Mason, W., Stroud, N.J., Tucker, J.A., 2023. Like-minded sources on facebook are prevalent but not polarizing. Nature 620, 137–144. https://doi.org/10.1038/s41586-023-06297-w

Zhuravskaya, E., Petrova, M., Enikolopov, R., 2020. Political effects of the internet and social media. Annual review of economics 12, 415–438.