This material was first presented at MIT CODE 2021. Thanks to Sean Taylor among others for comments.

Thinking about tradeoffs? draw an ellipse. When making a tradeoff between two outcomes, $X$ and $Y$, it’s useful to sketch out what the tradeoff looks like, and an ellipse is often a good first-order approximation. The ellipse helps visualize the most interesting parameter: the tightness, i.e. how much the rate of tradeoff between $X$ and $Y$ varies as you increase $X$.

In addition we can show that if the Pareto frontier is formed by the sum of vectors, and the vectors are drawn from a joint Normal distribution, then the expected frontier will be exactly an ellipse.

Concrete Applications:

Choosing launch criteria? draw an ellipse. Suppose you have a set of features each of which has some metric impact, $\Delta X$ and $\Delta Y$. If we assume that the effects are additive then we can construct a Pareto frontier, i.e. a set of all the aggregate effects on $\Delta X$ and $\Delta Y$ achievable by selection from the set of features. The frontier will typically look like an ellipse. You can prove that the Pareto frontier will be exactly an ellipse if the set of experiment-effects have a joint Normal distribution.
Choosing ranking weights? draw an ellipse. Suppose you are ranking items for a user using a set of features e.g. p(Like), p(Comment), etc. It is useful to sketch out the Pareto frontier, i.e. the set of outcomes achievable by different ranking algorithms. If the outcomes are additively separable functions of the item-features, and if the joint distribution of features is Normal, then the Pareto frontier will be an ellipse.
Allocating headcount? draw an ellipse. When you shuffle headcount around a company it’s hard to precisely measure the impact on different goals, however I have found it useful to sketch ellipses to make explicit the tradeoffs you face. This is particularly useful for visualizing the differences between within-team vs between-team reallocation of effort.

Tight and Loose Tradeoffs

Suppose we care about two metrics, $X$ and $Y$. E.g. suppose we care about DAU and time-spent, or revenue and retention, or engagement and misinformation.

It is useful to draw a Pareto frontier. A Pareto frontier will show the set of achievable outcomes for X and Y, to make the tradeoff precise. If we have a well-defined objective function then we can visually represent the optimal choice where the indifference curve is tangent to the Pareto frontier.

If your frontier is “tight” then there is not much tradeoff. The first figure shows a tight frontier, meaning that there is not much tradeoff available between X and Y. With a tight tradeoff it doesn’t matter whether we maximize X or Y or a weighted average, we’ll end up in roughly the same place anyway. Suppose we are choosing among experiments: if we observe a high positive correlation between $\Delta X$ and $\Delta Y$ then the choice of shipping criteria is relatively unimportant, most criteria would select the same experiments anyway. Suppose instead we are calibrating a recommender system: if we observe a high positive correlation between predictions of the two outcomes then the choice of weights is relatively unimportant, we would end up showing the same items anyway.

If your frontier is “loose” then there is a lot of tradeoff. The second figure shows a loose tradeoff: in this case the outcome does depend substantially on the relative weight we put on $X$ and $Y$.

Ellipses for Experiments

Suppose we have a set of experiments. Each experiment has some impact on two metrics, $\Delta X$ and $\Delta Y$. We visualize such a set of experiments at right.

If the set of features is separable, meaning that the impact of each feature is independent of what other features are launched, then a natural question will be the shape of the Pareto frontier formed by all possible combination of experiments.

If the distribution of experiments is mean zero and joint Normal then the Pareto frontier will be an ellipse, and it will have exactly the shape of an isovalue of the density of experiments. Thus knowing the variance and covariance of experiment results allows us to characterize the nature of the Pareto frontier we face.

Ellipses for Ranking

Suppose we are choosing a fixed set of items to show to a user, based on two metrics $x_1$ and $x_2$. E.g. pLike and pComment, or pDAU and quality etc. A natural question will be the shape of the Pareto frontier formed by alternative selections of items.

The Pareto frontier will be an ellipse. We show below that, if the predictions are well calibrated, the outcomes are independent (i.e. additive), and the distribution of prediction obeys a joint Normal distribution, then the Pareto frontier will be an ellipse and it will have exactly the shape of an isovalue of the density of predictions. Thus knowing the variance and covariance of predictions allows us to exactly characterize the nature of the aggregate tradeoffs we face.

Ellipses for Company Strategy

Tradeoffs are looser higher in the decision hierarchy. Suppose a company cares about two outcomes, $X$ and $Y$. Many different people will be making tradeoff decisions between X and Y, we can distinguish between four objectives used at different levels in the company hierarchy: \[\substack{\text{company objective}\\\text{(choose headcount)}} > \substack{\text{team objective}\\\text{(choose projects)}} > \substack{\text{shipping objective}\\\text{(choose experiments)}} > \substack{\text{algorithm objective}\\\text{(choose items)}} \]

We can think of each successive level as holding more variables fixed, and so we expect the Pareto frontiers to become successively tighter (Le Chatelier principle). We thus expect the tradeoff to be loosest at the level of overall company objectives, where we reallocate headcount. For this reason we should expect that, if the company as a whole pivots form metric $X$ to metric $Y$, the principal effect will be a reallocation of effort between products rather than reallocation within products.

We now walk through some of the different levels of optimization:

Different product areas have different Pareto frontiers. Typically two different product areas will have substantially different ability to affect different metrics, and we will often observe a situation like that shown on the right: team A’s choices primarily affect metric $X$, team B’s choices primarily affect metric $Y$.

We can also draw a combined Pareto frontier. Here we add up the Pareto frontiers of team A and B. In this case the combined frontier is somewhat tight, because the two constituent frontiers are tight. Neither individual Pareto frontier shows a substantial effect from changing weights (if we restrict weights to be positive), and so accordingly the combined Pareto frontier shows little response to a change in weights.

Note that I have drawn the frontier only approximately, the frontier achieved by combining two ellipses does not have a simple representation. When the two constituent frontiers are straight lines then the combination will be a parallelogram. (Note also that when the two frontiers are circles then the combination will be a circle).

Greater investment will shift Pareto frontiers out. Here we visualize reallocating employees from team B (the frontier shifts in) to team A (the frontier shifts out).

A combined company Pareto frontier will be loose. Here the green curve represents all the possible outcomes as you shift resources between team A and B: we have now turned a tight tradeoff into a loose tradeoff. In this case this represents that a change in company objectives will be reflected mainly in reallocation of effort between teams rather than within teams.

Appendix: Model for Normal Distributions

Suppose we have a set of items with, $x_1$ and $x_2$, distributed Normally: \[\binom{x_1}{x_2}\sim N\left(\binom{0}{0}, \begin{pmatrix}\sigma_1^2 & \rho\sigma_1\sigma_2 \\ \rho\sigma_1\sigma_2 & \sigma_2^2\end{pmatrix}\right).\]

We additionally let each item have a score, $v$, which is simply a weighted sum of the two characteristics (normalizing the weight on the first characteristic to be 1): \[v=x_1 + wx_2.\]

We can write the covariance between the characteristics and the score as follows:

\[Cov\begin{bmatrix}x_1\\x_2\\v\end{bmatrix}= \begin{bmatrix} \sigma^2_1 & \sigma_1\sigma_2\rho & \sigma_1^2+ w\rho\sigma_1\sigma_2 \\ \sigma_1\sigma_2\rho & \sigma_2^2 & \rho\sigma_1\sigma_2+w\sigma_2^2 \\ \sigma_1^2+w\rho\sigma_1\sigma_2 & \rho\sigma_1\sigma_2+w\sigma_2^2 & \sigma_1^2+w^2\sigma_2^2 + 2\rho w\sigma_1\sigma_2 \\ \end{bmatrix}\]

We wish to know the total number of actions of each type, $X_1$ and $X_2$, for a given score threshold $\bar{v}$:

\[\begin{aligned} X_1 &=P(v\geq \bar{v})E[x_1|v\geq \bar{v}] \\ X_2 &=P(v\geq \bar{v})E[x_2|v\geq \bar{v}]. \end{aligned}\]

We first calculate the conditional expectations:

\[\begin{aligned} E[x_1|v\geq \bar{v}] =& \sigma_1 \frac{Cov(x_1,v)}{\sqrt{Var(x_1)Var(v)}} \frac{\phi(\frac{\bar{v}}{\sqrt{Var(v)}})}{\Phi(\frac{\bar{v}}{\sqrt{Var(v)}})} \\ =& \sigma_1 \frac{\sigma_1^2+w\rho\sigma_1\sigma_2} {\sqrt{\sigma_1^2(\sigma_1^2+w^2\sigma_2^2 + 2\rho w\sigma_1\sigma_2)}} \frac{\phi(\frac{\bar{v}}{\sqrt{Var(v)}})}{\Phi(\frac{\bar{v}}{\sqrt{Var(v)}})}\\ =& \frac{\sigma_1^2+w\rho\sigma_1\sigma_2} {\sqrt{\sigma_1^2+w^2\sigma_2^2 + 2\rho w\sigma_1\sigma_2}} \frac{\phi(\frac{\bar{v}}{\sqrt{Var(v)}})}{\Phi(\frac{\bar{v}}{\sqrt{Var(v)}})} \end{aligned}\]

Next we will assume that the expected quantity of items is fixed. This implies that both both $P(v\geq \bar{v})$ and $\frac{\bar{v}}{\sqrt{Var(v)}}$ will be constant, and we will define: \[\begin{aligned} \gamma\equiv &\frac{\phi\left(\frac{\bar{v}}{\sqrt{Var(v)}}\right)} {\Phi\left(\frac{\bar{v}}{\sqrt{Var(v)}}\right)}P(v\geq \bar{v}) \\ X_1 =& \frac{\sigma_1^2+w\rho\sigma_1\sigma_2} {\sqrt{\sigma_1^2+w^2\sigma_2^2 + 2\rho w\sigma_1\sigma_2}}\gamma \\ X_2 =& \frac{w\sigma_2^2+\rho\sigma_1\sigma_2} {\sqrt{\sigma_1^2+w^2\sigma_2^2 + 2\rho w\sigma_1\sigma_2}}\gamma \end{aligned}\]

We thus have expressions for $X_1$ and $X_2$ as a function of the relative weight $w$. We wish to rearrange these to express $X_1$ directly in terms of $X_2$. To help we turn to Mathematica, with the following input:¹

¹ see notebook

F1[w_,p_,s1_,s2_,g_]:=g(s1^2+w p s1 s2 )/Sqrt[s1^2+w^2 s2^2+2w p s1 s2]
F2[w_,p_,s1_,s2_,g_]:=g(w s2^2 +p s1 s2)/Sqrt[s1^2+w^2 s2^2+2w p s1 s2]
Solve[{X1==F1[w,p,s1,s2,g],X2==F2[w,p,s1,s2,g]}, {X1,w}]
Simplify[First[%1]]

This returns a large expression:

\[\begin{aligned} X_1(X_2) &= \frac{ \gamma^2 \rho \sigma_1 \sigma_2^3 X_2 - p \sigma_1 \sigma_2 X_2^3 + \gamma^3 \sigma_2^4 \sqrt{\frac{-\gamma^2 (-1 + p^2) \sigma_1^2 \sigma_2^2}{\gamma^2 \sigma_2^2 - X_2^2}} - \gamma \sigma_2^2 X_2^2 \sqrt{-\frac{\gamma^2 (-1 + p^2) \sigma_1^2 \sigma_2^2}{\gamma^2 \sigma_2^2 - X_2^2}} + X_2 \sqrt{(-1 + p^2) \sigma_1^2 \sigma_2^2 X_2^2 (-\gamma^2 \sigma_2^2 + X_2^2)} }{\gamma^2 \sigma_2^4 - \sigma_2^2 X_2^2}\end{aligned}\]

We can however substantially simplify this: \[\begin{aligned} X_1 &= \frac{ \sigma_1\sigma_2X_2 (\gamma^2 \rho \sigma_2^2 - p X_2^2) + \gamma^2\sigma_2^2(\gamma^2 \sigma_2^2- X_2^2) \sqrt{-\frac{(-1 + p^2) \sigma_1^2 \sigma_2^2}{\gamma^2 \sigma_2^2 - X_2^2}} - X_2^2\sigma_1\sigma_2 \sqrt{(p^2-1) (\gamma^2 \sigma_2^2-X_2^2)} }{\gamma^2 \sigma_2^4 - \sigma_2^2 X_2^2} \\ &= \frac{ \sigma_1\sigma_2X_2 p(\gamma^2 \sigma_2^2 - X_2^2) + \gamma \sigma_2^3\sigma_1 \gamma \sqrt{(p^2-1)(\gamma^2 \sigma_2^2 - X_2^2)} - X_2^2\sigma_1\sigma_2 \sqrt{(p^2-1) (\gamma^2 \sigma_2^2-X_2^2)} }{\sigma_2^2(\gamma^2 \sigma_2^2 - X_2^2)} \\ &= \frac{\sigma_1}{\sigma_2}X_2p + \frac{ \sigma_1\sigma_2(\gamma^2\sigma_2^2 - X_2^2 ) \sqrt{(p^2-1) (X_2^2-\gamma^2 \sigma_2^2)} }{\sigma_2^2(\gamma^2 \sigma_2^2 - X_2^2)} \\ &= X_2 \rho \frac{\sigma_1}{\sigma_2} +\frac{\sigma_1}{\sigma_2}\sqrt{(p^2-1) (X_2^2-\gamma^2 \sigma_2^2)}. \end{aligned}\]

We now wish to show that this curve is equal to an isovalue of the joint distribution of $x_1$ and $x_2$ (illustrated at right). We can write the isovalue of the joint Normal distribution of $(x_1,x_2)$ for any given $k$ as follows:²

² From Bertsekas and Tsitsiklis (2002) “Introduction to Probability”, Section 4.7

\[k = \frac{x_1^2}{\sigma_1^2}+\frac{x_2^2}{\sigma_2^2}-2\rho\frac{x_1x_2}{\sigma_1\sigma_2}.\]

Solving this quadratic we can write: \[\begin{aligned} x_1 &= x_2 \rho \frac{\sigma_1}{\sigma_2} \pm \frac{\sigma_1}{\sigma_2}\sqrt{-x_2^2+x_2^2\rho^2+k\sigma_2^2} \\ &= x_2 \rho \frac{\sigma_1}{\sigma_2} \pm \frac{\sigma_1}{\sigma_2}\sqrt{k\sigma_2^2-(1-\rho^2)x_2^2}. \end{aligned}\]

We can see that this will be identical to the relationship between $X_1$ and $X_2$ above when $k=\frac{\sigma_2^2}{\sigma_1^2}(\rho^2-1)\gamma^2$.

Appendix: Simulations for Non-Normal Distributions

In this section I compare Pareto frontiers generated from different distribution of $(X,Y)$ from different joint distribution, and then draw the Pareto frontiers. There are a few points of interest:

Other distributions apart from the Normal do not have the property that the Pareto frontier is equal to an isovalue of the joint density.
In my examples the Pareto frontier of many other distributions does look roughly elliptical though not precisely an ellipse. This makes me more comfortable to use an ellipse as a first-order approximation of a Pareto frontier.

These simulations show something slightly different from what is proved in the prior section. These simulations show that as the number of experiments is large ($N\rightarrow\infty$) then the Pareto frontier begins to resemble an ellipse. What was proved in the previous section is that for any given $N$ the expected Pareto frontier will be an ellipse, where we calculate the expected value of $X$ and $Y$ for a given rate of tradeoff.

In each case the left-hand plot shows the raw distribution, the right-hand plot shows the Pareto frontier.

Joint normal with positive correlation, small $N$:

Joint normal with positive correlation, $N$=100:

Joint t Distribution with 3 degrees of freedom, $N$=100.

Independent Laplace, $N$=100:

Common Laplace factor with independent Gaussian noise, $N$=100:

Independent uniform, $N$=100:

Common uniform factor plus independent uniform noise, $N$=100:

--- title: Thinking About Tradeoffs? Draw an Ellipse author: Tom Cunningham, OpenAI. execute: echo: false cache: true # caches chunk output warning: false error: false date: 2023-10-25 updated: 2024-05-25 engine: knitr reference-location: margin figure-location: margin format: html: toc: true toc-depth: 2 toc-location: left code-tools: source: true --- <style> h1 { border-bottom: 4px solid black; } h2 { border-bottom: 1px solid #ccc;} .reveal section p { display: inline-block; font-size: 2em; #line-height: 1.2em; #vertical-align: top; } </style>  ::: {.column-margin} This material was first presented at MIT CODE 2021. Thanks to [Sean Taylor](https://www.linkedin.com/in/seanjtaylor/) among others for comments. :::  **Thinking about tradeoffs? draw an ellipse.** When making a tradeoff between two outcomes, $X$ and $Y$, it's useful to sketch out what the tradeoff looks like, and an ellipse is often a good first-order approximation. The ellipse helps visualize the most interesting parameter: the *tightness*, i.e. how much the rate of tradeoff between $X$ and $Y$ varies as you increase $X$. In addition we can show that if the Pareto frontier is formed by the sum of vectors, and the vectors are drawn from a joint Normal distribution, then the expected frontier will be exactly an ellipse. **Concrete Applications:** 1. **Choosing launch criteria? draw an ellipse.** Suppose you have a set of features each of which has some metric impact, $\Delta X$ and $\Delta Y$. If we assume that the effects are additive then we can construct a Pareto frontier, i.e. a set of all the aggregate effects on $\Delta X$ and $\Delta Y$ achievable by selection from the set of features. The frontier will typically look like an ellipse. You can prove that the Pareto frontier will be exactly an ellipse if the set of experiment-effects have a joint Normal distribution. 2. **Choosing ranking weights? draw an ellipse.** Suppose you are ranking items for a user using a set of features e.g. p(Like), p(Comment), etc. It is useful to sketch out the Pareto frontier, i.e. the set of outcomes achievable by different ranking algorithms. If the outcomes are additively separable functions of the item-features, and if the joint distribution of features is Normal, then the Pareto frontier will be an ellipse. 3. **Allocating headcount? draw an ellipse.** When you shuffle headcount around a company it's hard to precisely measure the impact on different goals, however I have found it useful to sketch ellipses to make explicit the tradeoffs you face. This is particularly useful for visualizing the differences between within-team vs between-team reallocation of effort. # Tight and Loose Tradeoffs ```{tikz} #| column: margin \begin{tikzpicture}[scale=3] \draw[<->] (0,1.1) -- node[rotate=90,above] {$\Delta$Y} (0,-.1); \draw[<->] (-.1,0) -- node[below] {$\Delta$X}(1.1,0); \begin{scope} \clip(0,0) rectangle (1,1); \draw[color=red, line width=2] (.5,.5) circle[x radius=.6, y radius=.2, rotate=45]; \end{scope} \draw[dashed] (0,.95) -- (1,.95);% node[right]{max Y}; \draw[dashed] (.94,0) -- (.94,1);% node[above,align=center]{max X}; \draw[dashed] (.75,1.1) -- (1.1,.75);% node[anchor=south west,align=center]{max X+Y}; \node at (.5,1.2) {Tight Tradeoff}; \end{tikzpicture} ``` ```{tikz} #| column: margin \begin{tikzpicture}[scale=3] \draw[<->] (0,1.1) -- node[rotate=90,above] {$\Delta$Y} (0,-.1); \draw[<->] (-.1,0) -- node[below] {$\Delta$X}(1.1,0); \begin{scope} \clip(0,0) rectangle (1,1); \draw[color=red, line width=2] (.5,.5) circle[x radius=.2, y radius=.6, rotate=45]; \end{scope} \draw[dashed] (0,.95) -- (1,.95);% node[right]{max Y}; \draw[dashed] (.95,0) -- (.95,1);% node[above,align=center]{max X}; \draw[dashed] (1,.3) -- (.3,1); %node[anchor=south west,align=center]{max X+Y}; \node at (.5,1.2) {Loose Tradeoff}; \end{tikzpicture} ``` **Suppose we care about two metrics, $X$ and $Y$.** E.g. suppose we care about DAU and time-spent, or revenue and retention, or engagement and misinformation. **It is useful to draw a Pareto frontier.** A Pareto frontier will show the set of achievable outcomes for X and Y, to make the tradeoff precise. If we have a well-defined objective function then we can visually represent the optimal choice where the indifference curve is tangent to the Pareto frontier. **If your frontier is "tight" then there is not much tradeoff.** The first figure shows a tight frontier, meaning that there is not much tradeoff available between X and Y. With a tight tradeoff it doesn't matter whether we maximize X or Y or a weighted average, we'll end up in roughly the same place anyway. Suppose we are choosing among experiments: if we observe a high positive correlation between $\Delta X$ and $\Delta Y$ then the choice of shipping criteria is relatively unimportant, most criteria would select the same experiments anyway. Suppose instead we are calibrating a recommender system: if we observe a high positive correlation between predictions of the two outcomes then the choice of weights is relatively unimportant, we would end up showing the same items anyway. **If your frontier is "loose" then there is a lot of tradeoff.** The second figure shows a loose tradeoff: in this case the outcome does depend substantially on the relative weight we put on $X$ and $Y$. # Ellipses for Experiments ```{tikz} #| column: margin \begin{tikzpicture}[scale=1] % ------------ second panel (draw this first because of "clip" below) \draw[<->] (2,0) -- (4,0) node[below]{$\Delta$X}; \draw[<->] (3,-1) -- (3,1) node[left]{$\Delta$Y}; \draw[color=red, line width=2] (3,0) circle[x radius=.9, y radius=.5, rotate=45]; \node at (3,-1.1) {Pareto frontier}; % ------------ first panel \draw[<->] (-1,0) -- (1,0) node[below]{$\Delta$X}; \draw[<->] (0,-1) -- (0,1) node[left]{$\Delta$Y}; \node at (0,-1.1) {experiments}; \clip (0,0) circle[x radius=.4, y radius=.2, rotate=45]; \pgfmathsetseed{24122015} \foreach \p in {1,...,200} { \fill (rand,rand) circle (0.02); } \end{tikzpicture} ``` **Suppose we have a set of experiments.** Each experiment has some impact on two metrics, $\Delta X$ and $\Delta Y$. We visualize such a set of experiments at right. If the set of features is *separable*, meaning that the impact of each feature is independent of what other features are launched, then a natural question will be the shape of the Pareto frontier formed by all possible combination of experiments. If the distribution of experiments is mean zero and joint Normal then the Pareto frontier will be an *ellipse*, and it will have exactly the shape of an isovalue of the density of experiments. Thus knowing the variance and covariance of experiment results allows us to characterize the nature of the Pareto frontier we face. # Ellipses for Ranking ```{tikz} #| column: margin \tikzset{ partial ellipse/.style args={#1:#2:#3}{insert path={+ (#1:#3) arc (#1:#2:#3)} } } \begin{tikzpicture}[scale=2.2] % ------------ first panel \draw[->] (0,1) node[above] {$x_2$} -- (0,0) -- (1,0) node[right] {$x_1$}; %\draw[dashed,color=blue, line width=2] (.2,.95)-- (.95,.2) % node[below,align=left] {$w_1x_1+w_2x_2=v^*$}; \draw[rotate around={45:(.5,.5)},fill=black,opacity=.1] (.5,.5) ellipse (.4 and .5); \draw[rotate around={45:(.5,.5)},fill=black,opacity=.1] (.5,.5) ellipse (.32 and .4); \draw[rotate around={45:(.5,.5)},fill=black,opacity=.1] (.5,.5) ellipse (.24 and .3); \node[align=center] at (.6,-.3) {Joint distribution\\of predictions.}; % ------------- second panel \draw[<->] (1.5,1) node[above] {$X_2$} --(1.5,0) --(2.5,0) node[right]{$X_1$}; %\draw[rotate around={45:(1.5+1,1)},line width=2,color=pink] (1.5+1,1) % [partial ellipse=150:210:.5 and .6]; \draw[rotate around={45:(1.5+.5,.5)},line width=2,gray,dashed] (1.5+.5,.5) ellipse (.2 and .25); \draw[rotate around={45:(1.5+.5,.5)},line width=2] (1.5+.5,.5) [partial ellipse=-55:55:.2 and .25]; \fill (1.5+.64,.64) circle[radius=0.5pt]; \node[align=center] at (2.1,-.3) {Pareto frontier\\of outcomes}; \end{tikzpicture} ``` **Suppose we are choosing a fixed set of items to show to a user, based on two metrics $x_1$ and $x_2$.** E.g. `pLike` and `pComment`, or `pDAU` and `quality` etc. A natural question will be the shape of the Pareto frontier formed by alternative selections of items. **The Pareto frontier will be an ellipse.** We show below that, if the predictions are well calibrated, the outcomes are independent (i.e. additive), and the distribution of prediction obeys a joint Normal distribution, then the Pareto frontier will be an ellipse and it will have exactly the shape of an isovalue of the density of predictions. Thus knowing the variance and covariance of predictions allows us to exactly characterize the nature of the aggregate tradeoffs we face. # Ellipses for Company Strategy **Tradeoffs are looser higher in the decision hierarchy.** Suppose a company cares about two outcomes, $X$ and $Y$. Many different people will be making tradeoff decisions between X and Y, we can distinguish between four objectives used at different levels in the company hierarchy: $$\substack{\text{company objective}\\\text{(choose headcount)}} > \substack{\text{team objective}\\\text{(choose projects)}} > \substack{\text{shipping objective}\\\text{(choose experiments)}} > \substack{\text{algorithm objective}\\\text{(choose items)}} $$ We can think of each successive level as holding more variables fixed, and so we expect the Pareto frontiers to become successively tighter (Le Chatelier principle). We thus expect the tradeoff to be loosest at the level of overall company objectives, where we reallocate headcount. For this reason we should expect that, if the company as a whole pivots form metric $X$ to metric $Y$, the principal effect will be a reallocation of effort *between* products rather than reallocation *within* products. We now walk through some of the different levels of optimization: ```{tikz} #| column: margin \begin{tikzpicture}[scale=3] \draw (0,1) -- node[rotate=90,above] {DAU} (0,0) -- node[below] {time spent}(1,0) -- (1,1) -- (0,1); \begin{scope} \clip(0,0) rectangle (1,1); \draw[color=red, line width=2] (0,0) circle[x radius=.7, y radius=.05, rotate=80]; \draw[color=blue, line width=2] (0,0) circle[x radius=.7, y radius=.05, rotate=10]; \end{scope} \node[blue] at (.6,.22) {engagement}; \node[red] at (.3,.8) {growth}; \end{tikzpicture} ``` **Different product areas have different Pareto frontiers.** Typically two different product areas will have substantially different ability to affect different metrics, and we will often observe a situation like that shown on the right: team A's choices primarily affect metric $X$, team B's choices primarily affect metric $Y$. ```{tikz} #| column: margin \begin{tikzpicture}[scale=3] \draw (0,1) -- node[rotate=90,above] {DAU} (0,0) -- node[below] {time spent}(1,0) -- (1,1) -- (0,1); \begin{scope} \clip(0,0) rectangle (1,1); \draw[line width=2] plot[smooth] coordinates {(0,.5) (.1,.8) (.9,.9) (.8,.1) (.5,0)}; \node at (.6,.8) {combined}; \draw[color=red, line width=2, opacity=.3] (0,0) circle[x radius=.7, y radius=.05, rotate=80]; \draw[color=blue, line width=2, opacity=.3] (0,0) circle[x radius=.7, y radius=.05, rotate=10]; \end{scope} \end{tikzpicture} ``` **We can also draw a *combined* Pareto frontier.** Here we add up the Pareto frontiers of team A and B. In this case the combined frontier is somewhat tight, because the two constituent frontiers are tight. Neither individual Pareto frontier shows a substantial effect from changing weights (if we restrict weights to be positive), and so accordingly the combined Pareto frontier shows little response to a change in weights. Note that I have drawn the frontier only approximately, the frontier achieved by combining two ellipses does not have a simple representation. When the two constituent frontiers are straight lines then the combination will be a parallelogram. (Note also that when the two frontiers are circles then the combination will be a circle).  ```{tikz} #| column: margin \begin{tikzpicture}[scale=3] \draw (0,1) -- node[rotate=90,above] {DAU} (0,0) -- node[below] {time spent}(1,0) -- (1,1) -- (0,1); \begin{scope} \clip(0,0) rectangle (1,1); \draw[color=red, dashed, line width=2] (0,0) circle[x radius=.7, y radius=.1, rotate=80]; \draw[color=blue, dashed, line width=2] (0,0) circle[x radius=.7, y radius=.1, rotate=10]; \draw[color=red, line width=2] (0,0) circle[x radius=.5, y radius=.1, rotate=80]; \draw[color=blue, line width=2] (0,0) circle[x radius=.9, y radius=.1, rotate=10]; \node[blue] at (.6,.25) {engagement}; \node[red] at (.3,.8) {growth}; \end{scope} \end{tikzpicture} ``` **Greater investment will shift Pareto frontiers out.** Here we visualize reallocating employees from team B (the frontier shifts in) to team A (the frontier shifts out). ```{tikz} #| column: margin \begin{tikzpicture}[scale=3] \draw (0,1) -- node[rotate=90,above] {DAU} (0,0) -- node[below] {time spent}(1,0) -- (1,1) -- (0,1); \draw[line width=2] plot[smooth] coordinates {(0,.3) (.1,.6) (.7,.7) (.6,.1) (.3,0)}; \begin{scope} \clip(0,0) rectangle (1,1); \draw[color=green!50!black, line width=2] (.2,.2) circle[x radius=.72, y radius=.72, rotate=45]; \end{scope} \node[green!50!black] at (.73,.9) {company}; \end{tikzpicture} ``` **A combined company Pareto frontier will be loose.** Here the green curve represents all the possible outcomes as you shift resources between team A and B: we have now turned a tight tradeoff into a loose tradeoff. In this case this represents that a change in company objectives will be reflected mainly in reallocation of effort *between* teams rather than *within* teams. # Appendix: Model for Normal Distributions Suppose we have a set of items with, $x_1$ and $x_2$, distributed Normally: $$\binom{x_1}{x_2}\sim N\left(\binom{0}{0}, \begin{pmatrix}\sigma_1^2 & \rho\sigma_1\sigma_2 \\ \rho\sigma_1\sigma_2 & \sigma_2^2\end{pmatrix}\right).$$ We additionally let each item have a *score*, $v$, which is simply a weighted sum of the two characteristics (normalizing the weight on the first characteristic to be 1): $$v=x_1 + wx_2.$$ We can write the covariance between the characteristics and the score as follows: $$Cov\begin{bmatrix}x_1\\x_2\\v\end{bmatrix}= \begin{bmatrix} \sigma^2_1 & \sigma_1\sigma_2\rho & \sigma_1^2+ w\rho\sigma_1\sigma_2 \\ \sigma_1\sigma_2\rho & \sigma_2^2 & \rho\sigma_1\sigma_2+w\sigma_2^2 \\ \sigma_1^2+w\rho\sigma_1\sigma_2 & \rho\sigma_1\sigma_2+w\sigma_2^2 & \sigma_1^2+w^2\sigma_2^2 + 2\rho w\sigma_1\sigma_2 \\ \end{bmatrix}$$ We wish to know the total number of actions of each type, $X_1$ and $X_2$, for a given score threshold $\bar{v}$: $$\begin{aligned} X_1 &=P(v\geq \bar{v})E[x_1|v\geq \bar{v}] \\ X_2 &=P(v\geq \bar{v})E[x_2|v\geq \bar{v}]. \end{aligned}$$ We first calculate the conditional expectations: $$\begin{aligned} E[x_1|v\geq \bar{v}] =& \sigma_1 \frac{Cov(x_1,v)}{\sqrt{Var(x_1)Var(v)}} \frac{\phi(\frac{\bar{v}}{\sqrt{Var(v)}})}{\Phi(\frac{\bar{v}}{\sqrt{Var(v)}})} \\ =& \sigma_1 \frac{\sigma_1^2+w\rho\sigma_1\sigma_2} {\sqrt{\sigma_1^2(\sigma_1^2+w^2\sigma_2^2 + 2\rho w\sigma_1\sigma_2)}} \frac{\phi(\frac{\bar{v}}{\sqrt{Var(v)}})}{\Phi(\frac{\bar{v}}{\sqrt{Var(v)}})}\\ =& \frac{\sigma_1^2+w\rho\sigma_1\sigma_2} {\sqrt{\sigma_1^2+w^2\sigma_2^2 + 2\rho w\sigma_1\sigma_2}} \frac{\phi(\frac{\bar{v}}{\sqrt{Var(v)}})}{\Phi(\frac{\bar{v}}{\sqrt{Var(v)}})} \end{aligned}$$ Next we will assume that the expected quantity of items is fixed. This implies that both both $P(v\geq \bar{v})$ and $\frac{\bar{v}}{\sqrt{Var(v)}}$ will be constant, and we will define: $$\begin{aligned} \gamma\equiv &\frac{\phi\left(\frac{\bar{v}}{\sqrt{Var(v)}}\right)} {\Phi\left(\frac{\bar{v}}{\sqrt{Var(v)}}\right)}P(v\geq \bar{v}) \\ X_1 =& \frac{\sigma_1^2+w\rho\sigma_1\sigma_2} {\sqrt{\sigma_1^2+w^2\sigma_2^2 + 2\rho w\sigma_1\sigma_2}}\gamma \\ X_2 =& \frac{w\sigma_2^2+\rho\sigma_1\sigma_2} {\sqrt{\sigma_1^2+w^2\sigma_2^2 + 2\rho w\sigma_1\sigma_2}}\gamma \end{aligned}$$ We thus have expressions for $X_1$ and $X_2$ as a function of the relative weight $w$. We wish to rearrange these to express $X_1$ directly in terms of $X_2$. To help we turn to Mathematica, with the following input:^[[see notebook](https://www.wolframcloud.com/env/tomcunningham/Ellipses.nb)] ``` F1[w_,p_,s1_,s2_,g_]:=g(s1^2+w p s1 s2 )/Sqrt[s1^2+w^2 s2^2+2w p s1 s2] F2[w_,p_,s1_,s2_,g_]:=g(w s2^2 +p s1 s2)/Sqrt[s1^2+w^2 s2^2+2w p s1 s2] Solve[{X1==F1[w,p,s1,s2,g],X2==F2[w,p,s1,s2,g]}, {X1,w}] Simplify[First[%1]] ``` This returns a large expression: $$\begin{aligned} X_1(X_2) &= \frac{ \gamma^2 \rho \sigma_1 \sigma_2^3 X_2 - p \sigma_1 \sigma_2 X_2^3 + \gamma^3 \sigma_2^4 \sqrt{\frac{-\gamma^2 (-1 + p^2) \sigma_1^2 \sigma_2^2}{\gamma^2 \sigma_2^2 - X_2^2}} - \gamma \sigma_2^2 X_2^2 \sqrt{-\frac{\gamma^2 (-1 + p^2) \sigma_1^2 \sigma_2^2}{\gamma^2 \sigma_2^2 - X_2^2}} + X_2 \sqrt{(-1 + p^2) \sigma_1^2 \sigma_2^2 X_2^2 (-\gamma^2 \sigma_2^2 + X_2^2)} }{\gamma^2 \sigma_2^4 - \sigma_2^2 X_2^2}\end{aligned}$$ We can however substantially simplify this: $$\begin{aligned} X_1 &= \frac{ \sigma_1\sigma_2X_2 (\gamma^2 \rho \sigma_2^2 - p X_2^2) + \gamma^2\sigma_2^2(\gamma^2 \sigma_2^2- X_2^2) \sqrt{-\frac{(-1 + p^2) \sigma_1^2 \sigma_2^2}{\gamma^2 \sigma_2^2 - X_2^2}} - X_2^2\sigma_1\sigma_2 \sqrt{(p^2-1) (\gamma^2 \sigma_2^2-X_2^2)} }{\gamma^2 \sigma_2^4 - \sigma_2^2 X_2^2} \\ &= \frac{ \sigma_1\sigma_2X_2 p(\gamma^2 \sigma_2^2 - X_2^2) + \gamma \sigma_2^3\sigma_1 \gamma \sqrt{(p^2-1)(\gamma^2 \sigma_2^2 - X_2^2)} - X_2^2\sigma_1\sigma_2 \sqrt{(p^2-1) (\gamma^2 \sigma_2^2-X_2^2)} }{\sigma_2^2(\gamma^2 \sigma_2^2 - X_2^2)} \\ &= \frac{\sigma_1}{\sigma_2}X_2p + \frac{ \sigma_1\sigma_2(\gamma^2\sigma_2^2 - X_2^2 ) \sqrt{(p^2-1) (X_2^2-\gamma^2 \sigma_2^2)} }{\sigma_2^2(\gamma^2 \sigma_2^2 - X_2^2)} \\ &= X_2 \rho \frac{\sigma_1}{\sigma_2} +\frac{\sigma_1}{\sigma_2}\sqrt{(p^2-1) (X_2^2-\gamma^2 \sigma_2^2)}. \end{aligned}$$ ```{tikz} #| column: margin \begin{tikzpicture}[scale=2] \draw[<->] (0,2)node[above,align=center]{$x_2$}--(0,0)-- (2,0) node[right]{$x_1$}; \def\si{1} \def\sj{1} \def\rho{.8} \def\min{0} \def\max{1} % top half of ellipse \draw [domain=\min:\max, variable=\x] plot ({\x}, {\si*sqrt((1-\rho^2)*(1-\x^2/\sj^2))+\rho*\si*\x/\sj}); % bottom half of ellipse \draw [domain=.5:\max, variable=\x] plot ({\x}, {-\si*sqrt((1-\rho^2)*(1-\x^2/\sj^2))+\rho*\si*\x/\sj}); \end{tikzpicture} ``` We now wish to show that this curve is equal to an isovalue of the joint distribution of $x_1$ and $x_2$ (illustrated at right). We can write the isovalue of the joint Normal distribution of $(x_1,x_2)$ for any given $k$ as follows:^[From Bertsekas and Tsitsiklis (2002) "Introduction to Probability", [Section 4.7](http://athenasc.com/Bivariate-Normal.pdf)] $$k = \frac{x_1^2}{\sigma_1^2}+\frac{x_2^2}{\sigma_2^2}-2\rho\frac{x_1x_2}{\sigma_1\sigma_2}.$$ Solving this quadratic we can write: $$\begin{aligned} x_1 &= x_2 \rho \frac{\sigma_1}{\sigma_2} \pm \frac{\sigma_1}{\sigma_2}\sqrt{-x_2^2+x_2^2\rho^2+k\sigma_2^2} \\ &= x_2 \rho \frac{\sigma_1}{\sigma_2} \pm \frac{\sigma_1}{\sigma_2}\sqrt{k\sigma_2^2-(1-\rho^2)x_2^2}. \end{aligned}$$ We can see that this will be identical to the relationship between $X_1$ and $X_2$ above when $k=\frac{\sigma_2^2}{\sigma_1^2}(\rho^2-1)\gamma^2$.  # Appendix: Simulations for Non-Normal Distributions In this section I compare Pareto frontiers generated from different distribution of $(X,Y)$ from different joint distribution, and then draw the Pareto frontiers. There are a few points of interest: 1. Other distributions apart from the Normal do not have the property that the Pareto frontier is equal to an isovalue of the joint density. 2. In my examples the Pareto frontier of many other distributions does look roughly *elliptical* though not precisely an ellipse. This makes me more comfortable to use an ellipse as a first-order approximation of a Pareto frontier. 3. These simulations show something slightly different from what is proved in the prior section. These simulations show that as the number of experiments is large ($N\rightarrow\infty$) then the Pareto frontier begins to resemble an ellipse. What was proved in the previous section is that for any given $N$ the *expected* Pareto frontier will be an ellipse, where we calculate the expected value of $X$ and $Y$ for a given rate of tradeoff. In each case the left-hand plot shows the raw distribution, the right-hand plot shows the Pareto frontier. ```{r} library(MASS) library(tidyverse) library(ggforce) library(ExtDist) # extra distributions library(cowplot) # for combining two plots # pareto frontier from selecting top N slope_steps <- 100 pareto_topn <- function(points, num_sample) { pareto <- data.frame(x = as.numeric(), y = as.numeric()) for (slope in 1:slope_steps) { points <- points %>% mutate( # score = x + y * -log(slope / slope_steps), score = x + y * tan(2 * 3.1415 * slope / slope_steps), rank = rank(desc(score)) ) stats <- points %>% summarize( avg_x = sum(x[rank < num_sample]) / num_sample, avg_y = sum(y[rank < num_sample]) / num_sample ) pareto[nrow(pareto) + 1, ] <- list(x = stats$avg_x, y = stats$avg_y) } ggplot(pareto, aes(x, y)) + geom_point(data = points, alpha = .1) + geom_density_2d(data = points) + geom_point(data = pareto, color = "red", alpha = 1) + coord_fixed() + xlim(-5, 5) + ylim(-5, 5) } # pareto_topn(points, 1000) # pass this function a set of points and it'll draw a pareto frontier along each axis pareto_from_points <- function(points) { slope_steps <- 100 pareto <- data.frame(x = as.numeric(), y = as.numeric()) for (slope in 1:slope_steps) { # loop through slopes slope_ratio <- slope / slope_steps points <- points %>% mutate( # given an angle between 0 and 2pi, unit vector will be (cos(theta),sin(theta)). # can get score by dot product with the unit vector score = x * cos(slope_ratio * 3.1415 * 2) + y * sin(slope_ratio * 3.1415 * 2) ) stats <- points %>% summarize( avg_x = sum(x[score > 0]), avg_y = sum(y[score > 0]) ) pareto[nrow(pareto) + 1, ] <- list(x = stats$avg_x, y = stats$avg_y) } return(pareto) } plot_pareto= function(points,pareto){ p <- ggplot(points, aes(x, y)) + geom_hline(yintercept = 0) + geom_vline(xintercept = 0) + geom_point() + coord_fixed() q <- ggplot(pareto, aes(x, y)) + geom_hline(yintercept = 0) + geom_vline(xintercept = 0) + # geom_point(data = points)+ # geom_density_2d(data = points) + geom_point(data = pareto, color = "red", alpha = 1) + geom_polygon(data = pareto, alpha = 0.1) + # fill coord_fixed() plot_grid(p, q) } ``` **Joint normal with positive correlation, $N=10$:** ```{r} Sigma <- matrix(c(1, 1, 1, 4), 2, 2) # 2x2 covariance matrix points <- data.frame(mvrnorm(n = 10, rep(0, 2), Sigma)) %>% rename("x" = 1, "y" = 2) plot_pareto(points,pareto_from_points(points)) ``` **Joint normal with positive correlation, $N$=100:** ```{r} Sigma <- matrix(c(1, 1, 1, 4), 2, 2) # 2x2 covariance matrix points <- data.frame(mvrnorm(n = 100, rep(0, 2), Sigma)) %>% rename("x" = 1, "y" = 2) plot_pareto(points,pareto_from_points(points)) ``` **Joint *t* Distribution with 3 degrees of freedom, $N$=100.** ```{r} library(mvtnorm) Sigma <- matrix(c(1, 1, 1, 4), 2, 2) # 2x2 covariance matrix points <- data.frame(rmvt(n=100,Sigma,df=3)) %>% rename("x" = 1, "y" = 2) plot_pareto(points,pareto_from_points(points)) ``` **Independent Laplace, $N$=100:** ```{r} points <- tibble(x = rLaplace(100), y = rLaplace(100)) plot_pareto(points,pareto_from_points(points)) ``` **Common Laplace factor with independent Gaussian noise, $N$=100:** ```{r} z <- rLaplace(100, mu = 0, b = 3) Sigma <- matrix(c(1, 0, 0, 1), 2, 2) # 2x2 covariance matrix points <- data.frame(mvrnorm(n = 100, rep(0, 2), Sigma)) %>% rename("x" = 1, "y" = 2) %>% cbind(z) %>% mutate(x = z + 2 * x, y = z + 2 * y) plot_pareto(points,pareto_from_points(points)) ``` **Independent uniform, $N$=100:** ```{r} points <- tibble(x = runif(100) - .5, y = runif(100) - .5) plot_pareto(points,pareto_from_points(points)) ``` **Common uniform factor plus independent uniform noise, $N$=100:** ```{r} points <- tibble(x = runif(100) - .5, y = runif(100) - .5, z = runif(100) - .5) %>% mutate( x = x + .5 * z, y = y + .5 * z ) plot_pareto(points,pareto_from_points(points)) ```