Non-Additive Axiologies in Large Worlds

4.1. Convergence

First, though, let us make the notion of ‘convergence’ more precise. Informally, we say that one axiology, $\mathcal{A}$, converges with another, ${\mathcal{A}}^{\prime }$, if the verdicts of $\mathcal{A}$ approximate the verdicts of ${\mathcal{A}}^{\prime }$ to arbitrary precision, as the size of the background population increases. In spelling this out, we will restrict attention to background populations of a given type, for example, all those having a certain average level of welfare. Here is the basic formal definition.

Ordinal Convergence

Axiology $\mathcal{A}$ converges ordinally with ${\mathcal{A}}^{\prime }$ relative to background populations of type $T$ if and only if, for any populations $X$ and $Y$, if $Z$ is a sufficiently large population of type $T$, then

$X+Z{\succ }_{{\mathcal{A}}^{\prime }}Y+Z\Rightarrow X+Z{\succ }_{\mathcal{A}}Y+Z.$

Of course, if ${\mathcal{A}}^{\prime }$ is additive, the last implication is equivalent to

We can, in other words, compare $X+Z$ and $Y+Z$ with respect to $\mathcal{A}$ by comparing $X$ and $Y$ with respect to ${\mathcal{A}}^{\prime }$—if we know that $Z$ is a sufficiently large population of the right type.

Note two ways in which this notion of convergence is fairly weak. First, what it means for $Z$ to be ‘sufficiently large’ can depend on $X$ and $Y$. Second, the displayed implications need not be a biconditional; thus, when ${\mathcal{A}}^{\prime }$ does not have a strict preference between $X+Z$ and $Y+Z$ (e.g., when it is indifferent between them), convergence with ${\mathcal{A}}^{\prime }$ does not imply anything about how $\mathcal{A}$ ranks those two populations.¹² Because of this, every axiology converges with the trivial axiology according to which no population is better than any other. Of course, such a result is uninformative, and we are only interested in convergence with more discriminating axiologies. Specifically, we will only ever consider axiologies that satisfy the Pareto principle (which we discuss in Section 5.1).

Ordinal convergence is ‘ordinal’ because it only concerns the way in which the two axiologies rank populations. As we noted in Section 2, one could interpret the value function used to define an axiology as conveying ‘cardinal’ information about the relative values of different populations. There is, correspondingly, a different notion of convergence that we will call cardinal convergence. Specifically, if $V_{\mathcal{A}}$ is the value function for $\mathcal{A}$, then one could interpret ratios like

as measuring how much better $X_1$ is than $Y_1$, compared to how much better $X_2$ is than $Y_2$.¹³ Cardinal convergence occurs when two axiologies agree about these cardinal facts to arbitrary precision as the background population becomes large.

Cardinal Convergence

Axiology $\mathcal{A}$ (with value function $V_{\mathcal{A}}$) converges cardinally with ${\mathcal{A}}^{\prime }$ (with value function $V_{{\mathcal{A}}^{\prime }}$) relative to background populations of type $T$ if and only if, for any four populations $X_1,\hspace{0.17em}{Y}_1,\hspace{0.17em}{X}_2,\hspace{0.17em}{Y}_2$ and any margin of error $ϵ>0$, if Z is a sufficiently large population of type $T$, then

$\frac{V_{\mathcal{A}}(X_1+Z)-V_{\mathcal{A}}(Y_1+Z)}{V_{\mathcal{A}}(X_2+Z)-V_{\mathcal{A}}(Y_2+Z)}\hspace{1em}\text{is}\hspace{0.17em}\text{within}\hspace{0.17em}ϵ\hspace{0.17em}\text{of}\hspace{1em}\frac{V_{{\mathcal{A}}^{\prime }}(X_1+Z)-V_{{\mathcal{A}}^{\prime }}(Y_1+Z)}{V_{{\mathcal{A}}^{\prime }}(X_2+Z)-V_{{\mathcal{A}}^{\prime }}(Y_2+Z)}$

(assuming the denominator $V_{{\mathcal{A}}^{\prime }}(X_2+Z)-V_{{\mathcal{A}}^{\prime }}(Y_2+Z)\ne 0$).

As we have defined it, cardinal convergence does not quite imply ordinal convergence. Thus our main results will assert both ordinal and cardinal convergence. And we will often just speak of ‘convergence’ to cover both kinds.

4.2. Average Utilitarianism

Average utilitarianism identifies the value of a population with its average welfare level.¹⁴

Average Utilitarianism $(\text{AU})$

$V_{\text{AU}}(X)=\overline{X}={\displaystyle \sum _{w\in \mathcal{W}}\frac{X(w)}{|X|}}w.$

Our first result describes the behavior of AU as the size of the background population tends to infinity.

Theorem 1. Average utilitarianism converges ordinally and cardinally to $C{L}_c$, relative to background populations with average welfare $c$. In fact, for any populations $X,\hspace{0.17em}Y,\hspace{0.17em}Z$, if $\overline{Z}=c$ and

then $V_{{\text{CL}}_c}(X)>V_{{\text{CL}}_c}(Y)\Rightarrow V_{\text{AU}}(X+Z)>V_{\text{AU}}(Y+Z)$.

Proofs of all theorems are given in the appendix.

Discussion of the normative implications of this and other results is deferred to the second half of the paper (§§6–9).

4.3. ‘Variable Value’ Views

Some philosophers have sought an intermediate position between total and average utilitarianism, acknowledging that increasing the size of a population (without changing its average welfare) can count as an improvement, but holding that additional lives have diminishing marginal value. The most widely discussed version of this approach is the variable value view.¹⁵ It is useful to distinguish two types of this view, the second more general than the first.

Variable Value I $(\text{VV}1)$

$V_{\text{VV1}}(X)=\overline{X}g(|X|)$

where $g:{\mathbb{Z}}_{+}\to {ℝ}_{+}$ is a non-zero function that is weakly increasing, concave, and bounded above.

Recall that the total welfare of a population $X$ is equal to $\overline{X}|X|$; roughly speaking, $\text{VV}1$ says that changes in the second factor, the size of $X$, are less important when $X$ is already large. The next view also gives varying marginal importance to average welfare:

Variable Value II $(\text{VV}2)$

$V_{\text{VV2}}(X)=f(\overline{X})g(|X|)$

where $f:ℝ\to ℝ$ is differentiable and strictly increasing, and $g:\mathbb{N}\to {ℝ}_{+}$ is a non-zero function that is weakly increasing, concave, and bounded above.

Sloganistically, variable value views can be ‘totalist for small populations’ (where $g$ may be nearly linear), but must become ‘averagist for large populations’ (as $g$ approaches its upper bound). It is therefore not entirely surprising that, in the large-background-population limit, $\text{VV}1$ and $\text{VV}2$ display the same behavior as AU, converging with a critical-level view with the critical level given by the average welfare of the background population.

Theorem 2. Variable value views converge ordinally and cardinally to ${\text{CL}}_c$, relative to background populations with average welfare $c$.

For the broad class of variable value views, we cannot give the sort of threshold for $|Z|$ that we gave for $\text{AU}$, above which the ranking of $X+Z$ and $Y+Z$ must agree with the ranking given by ${\text{CL}}_{\overline{Z}}$. For instance, because $g$ can be any non-zero function that is weakly increasing, concave, and bounded above, variable value views can remain in arbitrarily close agreement with totalism for arbitrarily large populations, so if $\text{TU}$ prefers one population to another, there will always be some variable value theory that agrees. In the case of $\text{VV}1$, we can say that if both $\text{TU}$ and $\text{AU}$ prefer $X$ to $Y$, then all $\text{VV}1$ views will as well (see Proposition 1 in appendix B), and so whenever $\text{TU}$ and ${\text{CL}}_{\overline{Z}}$ have the same strict preference between $X$ and $Y$, the threshold given in Theorem 1 holds for $\text{VV}1$ as well. For $\text{VV}2$, we cannot even say this much.¹⁶

5.1. Two-Factor Egalitarianism

Among two-factor egalitarian theories, there is another important choice point between ‘totalist’ and ‘averagist’ views.

Totalist Two-Factor Egalitarianism

$V(X)=\text{Tot}(X)-I(X)|X|$

where $I$ is some measure of inequality in $X$.

Averagist Two-Factor Egalitarianism

$V(X)=\overline{X}-I(X)$

where $I$ is some measure of inequality in $X$.¹⁷

Here, in each case, the second term of the value function can be thought of as a penalty representing the badness of inequality. Such a penalty could have any number of forms, but for the purposes of illustration we stipulate that $I(X)$ depends only on the distribution of $X$, where this can be understood formally as the function $X/|X|:\mathcal{W}\to ℝ$ giving the proportion of the population in $X$ having each welfare level. The degree of inequality is indeed plausibly a matter of the distribution in this sense, and the badness of inequality is then plausibly a function of the degree of inequality and the size of the population. The more substantial assumption is that the badness of inequality either scales linearly with the size of the population (for the totalist version of the view) or does not depend on population size (for the averagist version).

Now, we want to know what these theories do as $|Z|\to \infty$. In the last section, we had to hold one feature of $Z$ constant as $|Z|\to \infty$, namely, $\overline{Z}$. Egalitarian theories, however, are potentially sensitive to the whole distribution of welfare levels in the population, and so to obtain limit results it is useful to hold fixed the whole distribution of welfare in the background population, that is, $D\hspace{0.17em}:=Z/|Z|$.

We’ll state the general result, explain some of the terminology it uses, and then give some examples.

Theorem 3. Suppose $V$ is a value function of the form $V(X)=\text{Tot}(X)-I(X)|X|$, or else $V(X)=\overline{X}-I(X)$, where $I$ is a differentiable function of the distribution of $X$. Then the axiology $\mathcal{A}$ represented by $V$ converges ordinally and cardinally with an additive axiology, relative to background populations with any fixed distribution $D$; specifically, it converges with the additive axiology with weighting function given by¹⁸

If the Pareto principle holds with respect to $\mathcal{A}$, then $f$ is weakly increasing, and if Pigou-Dalton transfers are weak improvements, then $f$ is concave.

A few points in the theorem require further explanation. Informally, the function $I$ is differentiable if $I(X)$ varies smoothly with $X$; we will give the formal definition when it comes to the proof (see Remark 1 in the appendix), but at any rate all proposed measures of inequality that we’re aware of are differentiable, including the two we discuss below. The Pareto principle holds that increasing anyone’s welfare increases the value of the population. This principle clearly holds for prioritarian views (because the priority-weighting $f$ is strictly increasing), but it need not in principle hold for egalitarian views: conceptually, increasing someone’s wellbeing might contribute so much to inequality as to be on net a bad thing. Still, the Pareto principle is generally held to be a desideratum for egalitarian views. Finally, a Pigou-Dalton transfer is a total-preserving transfer of welfare from a better-off person to a worse-off person that keeps the first person better-off than the second. The condition that Pigou-Dalton transfers are at least weak improvements (they do not make things worse) is often understood as a minimal requirement for egalitarianism.

To illustrate Theorem 3, let’s consider two more specific families of egalitarian axiologies that instantiate the schemata of totalist and averagist two-factor egalitarianism respectively.

For the first, we’ll use a measure of inequality based on the mean absolute difference ($\text{MD}$) of welfare, defined for any population $X$ as follows:

$\text{MD}(X)$ represents the average welfare inequality between any two individuals in $X.\text{MD}(X)|X|$, which scales with population size, can be understood as taking each individual’s average welfare inequality with the members of $X$ (including herself), then summing across individuals. Consider, then, the following totalist two-factor view:

Mean Absolute Difference Total Egalitarianism $(\text{MDT})$

$V_{\text{MDT}}(X)=\text{Tot}(X)-\alpha \text{MD}(X)|X|$

where $\alpha \in (0,\hspace{0.17em}1/2)$ is a constant that determines the relative importance of inequality.¹⁹

Second, consider the following averagist two-factor view, which identifies overall value with a quasi-arithmetic mean of welfare:²⁰

Quasi-Arithmetic Average Egalitarianism $(\text{QAA})$

$V_{\text{QAA}}(X)=\text{QAM}(X)=g^{-1}({\displaystyle \sum _{w\in \mathcal{W}}\frac{X(w)}{|X|}}g(w))$

for some strictly increasing, concave function $g:\mathcal{W}\to ℝ$.

Implicitly, the measure of inequality in $\text{QAA}\hspace{0.17em}\hspace{0.17em}\text{is}\hspace{0.17em}\hspace{0.17em}I(X)=\overline{X}-\text{QAM}(X)$, which one can show is a positive function, weakly decreasing under Pigou-Dalton transfers. In the limiting case where $g$ is linear, $\text{QAM}(X)=\overline{X}$.

Theorem 4. $\text{MDT}$ converges ordinally and cardinally to $\text{PR}$, relative to background populations with a given distribution $D$. Specifically, ${\text{MDT}}_{\alpha }$ converges with ${\text{PR}}_f$, the prioritarian axiology whose weighting function is

Here $\text{MD}(w,\hspace{0.17em}D):={\sum }_{x\in \mathcal{W}}D(x)|x-w|$ is the average distance between $w$ and the welfare levels occurring in $D$.

Theorem 5. $\text{QAA}$ converges ordinally and cardinally to $\text{PR}$, relative to background populations with a given distribution $D$. Specifically, ${\text{QAA}}_g$ converges with ${\text{PR}}_f$, the prioritarian axiology whose weighting function is

5.2. Rank Discounting

Another family of population axiologies that is often taken to reflect egalitarian motivations is rank-discounted utilitarianism (RDU). The essential idea of rank-discounting is to give different weights to marginal changes in the welfare of different individuals, not based on their absolute welfare level (as prioritarianism does), but rather based on their welfare rank within the population. One potential motivation for RDU over two-factor views is that, because we are simply applying different positive weights to the marginal welfare of each individual, we clearly avoid any charge of ‘leveling down’: unlike on two-factor views, there is nothing even pro tanto good about reducing the welfare of a better-off individual—it is simply less bad than reducing the welfare of a worse-off individual.²¹

Versions of rank-discounted utilitarianism have been discussed and advocated under various names in both philosophy and economics, for example, by Asheim and Zuber (2014) and Buchak (2017). In these contexts, the RDU value function is generally taken to have the following form:

where $X_k$ denotes the welfare of the $k$th worst off welfare subject in $X$, and $f:\mathbb{N}\to ℝ$ is a positive but weakly decreasing function.²²

However, these discussions often assume a context of fixed population size, and there are different ways one might extend the formula when the size is not fixed.

We will consider the most obvious approach, simply taking equation (2) as a definition regardless of the size of X.²³ A view of this type, explicitly designed for a variable-population context, is set out in Asheim and Zuber (2014). Simplifying slightly to set aside features irrelevant for our purposes, their view is as follows:

Geometric Rank-Discounted Utilitarianism $(\text{GRD})$

$V_{\text{GRD}}(X)={\displaystyle \sum _{k=1}^{|X|}{\beta }^k}{X}_k$

for some $\beta \in (0,\hspace{0.17em}1)$.

Here, the rank-weighting function is $f(k)={\beta }^k$. In general, since $f$ is assumed to be weakly decreasing and positive, $f(k)$ must asymptotically approach some limit $L$ as $k$ increases. For $\text{GRD},\hspace{0.17em}L=0$. But a simpler situation arises when $L>0$ (so that $f$ is bounded away from zero):

Bounded Rank-Discounted Utilitarianism $(\text{BRD})$

$V_{\text{BRD}}(X)={\displaystyle \sum _{k=1}^{|X|}f}(k)X_k$

for some weakly decreasing, positive function $f:ℝ\to ℝ$ that is eventually convex²⁴ with asymptote $L>0$.

We will state formal results about both $\text{GRD}$ and $\text{BRD}$ in Appendix A; they involve a slightly more restricted notion of convergence than we have considered so far. The case of $\text{BRD}$ is relatively simple: it converges with total utilitarianism. This is because, when the background population is very large, each life in the foreground population with welfare level $w$ contributes approximately $Lw$ to the overall value of the population (at least assuming that $w$ is higher than some level in the background population). So the overall contribution of the foreground population is approximately equal to its total welfare times $L$.

When, as in $\text{GRD}$, the asymptote of the weighting function $f$ is at $L=0$, the situation is subtler and appears to depend on the exact rate at which $f$ decays. We will consider only $\text{GRD}$, as it is the best-motivated example in the literature. Uniquely among the axiologies we consider, $\text{GRD}$ does not converge with an additive, Paretian axiology on any interesting range of populations. Roughly speaking, this is because, as the background population gets larger, the weight given to the best-off individual in $X$ becomes arbitrarily small relative to the weight given to the worst-off—smaller than the relative weight given by any particular additive, Paretian axiology.

Nonetheless, it turns out that GRD does converge with a separable, Paretian axiology, which we call critical-level leximin. This is an extreme form of prioritarianism in which infinite priority is always given to the less well-off. We’ll explain this carefully in Appendix A, but perhaps the most important take-away is that (because critical-level leximin is so extreme) GRD leads to some very strange and counterintuitive results when the background population is sufficiently large.

For example, tiny benefits to worse-off individuals will often be preferred over astronomical benefits to even slightly better-off individuals; moreover, adding an individual to the population with anything less than the maximum welfare level in the background population will often make things worse overall.²⁵ In fact, $\text{GRD}$ implies what we might call the ‘Snobbish Conclusion’:

Snobbish Conclusion

In some circumstances, given a very high welfare level $w_1$ just slightly below the best in the background population, and an even higher welfare level $w_2$ greater than any in the background population, adding even one life at $w_1$ makes things so much worse that it cannot be compensated by any number of lives at $w_2$.

This seems crazy to us. We could just about understand the Snobbish Conclusion in the context of an anti-natalist view, according to which adding lives invariably has negative value; but, according to $\text{GRD}$, there are many possible background populations (for instance, any in which the highest welfare level is less than $w_1$) to which the addition described above would constitute an improvement. We could also understand the view that adding good lives can make things worse if it lowers average welfare or increases inequality (e.g., as measured by mean absolute difference or standard deviation). But, again, that’s not what’s going on here. Instead, $\text{GRD}$ implies that adding excellent lives makes things worse if the number of even slightly better lives already in existence happens to be sufficiently great, regardless of the other facts about the distribution. In some cases, it makes things so much worse that it cannot be compensated by adding any number of even better lives.

To sum up, many forms of egalitarianism, including many forms of rank-discounted utilitarianism, converge with interesting additive axiologies. Geometric Rank-Discounted Utilitarianism provides one counterexample, although it does converge with an interesting separable axiology. Moreover, our general methodology of thinking about large background populations draws out some features that make $\text{GRD}$ seem especially implausible.

6.1. Population Size

We will make two claims about the size of the background populations that are relevant to real-world choices, with different degrees of confidence.

First, with high confidence, these populations are much larger (at least multiple orders of magnitude) than the present human population. Concretely, while there are fewer than ${10}^{10}$ humans alive today, we conservatively estimate that there have been at least ${10}^{17}$ welfare subjects in Earth’s past, with estimates of ${10}^{20}$ or more being plausible. Informally, this suggests that our limit results should at least be relevant when comparing options that only affect present and near-future humans (though a background population of this size can also substantially affect the evaluation of choices affecting the far future, as we will see in §7).

Second, with much lower confidence, real-world background populations may well be much larger (again, by multiple orders of magnitude) than the entire population in our future light cone. If this is right then our limit results are likely to be relevant to essentially any real-world choice.

Let’s start by establishing the first claim, which only requires us to consider past welfare subjects on Earth. Estimates of the number of human beings who have ever lived are on the order of ${10}^{11}$ (Kaneda & Haub 2018), already an order of magnitude larger than the present human population. But past welfare subjects include a vast number of non-human animals, and especially wild animals over many millions of years. There are today at least ${10}^{11}$ wild mammals; for vertebrates in general, the number is far higher, with a conservative lower bound of ${10}^{13}$ (dominated by fish).²⁶ Prehistoric wild animal populations were presumably similarly large or larger, given the significant decline in wild animal populations as a result of human encroachment.²⁷ Inferring the total number of animals from the number alive at a given time requires assumptions about mortality rates. We will use a very conservative estimate of 0.1 deaths per individual per year in wild animal populations (roughly corresponding to a life expectancy of 10 years). The actual rates are almost certainly much higher for most species (especially given high infant mortality), implying larger total past populations. Being extremely conservative, then, we find that there have been at least $6.6\hspace{0.17em}\times \hspace{0.17em}{10}^{17}$ mammals since the extinction of the dinosaurs 66 million years ago.²⁸ This gives our basic lower bound for the size of the background population. If we less conservatively allow that all vertebrates are welfare subjects, then a similar calculation gives a lower bound of $5\hspace{0.17em}\times \hspace{0.17em}{10}^{20}$ individuals over the last 500 million years. And of course some invertebrates may be welfare subjects too.

While these background populations are large compared to the present population, they may not be large compared to the entire affectable future population. If our civilization survives for a very long time, the number of future individuals might be truly astronomical, and actions that affect the long-term future (for instance by causing or preventing existential catastrophes) might affect this entire population. If we can sustain just the size of the present human population until the Earth becomes uninhabitable, this would yield a future population size on the order of ${10}^{17}$, even ignoring non-humans.²⁹ This is roughly on a par with our lower-bound estimate of the number of past animals on Earth (though still much smaller than the more generous estimate that includes all vertebrates). Less conservatively, if humanity someday settles the cosmos and creates digital minds on a mass scale, far larger populations become possible—for instance, Bostrom (2013) estimates that such an interstellar civilization could support 10⁵⁴ subjective life-years of human-like experience, perhaps in the form of 10⁵² lives with a subjective duration of 100 years each. Any plausible estimate of past animals on Earth will pale in comparison with these numbers.

There may nevertheless be unaffectable background populations even larger than these astronomical potential future populations. The crucial point is that the universe as a whole appears to be at least 100 times larger, and perhaps vastly larger, than the accessible universe (the portion of the universe that it is possible in principle for us to reach).³⁰ So, if life arises independently in many places, we would expect at least 99% of it to be outside the accessible universe and thus necessarily part of the background population. Similarly, if the universe contains many spacefaring civilizations, at least 99% of them should be inaccessible. However large the population of future human-originating civilization, this background population (consisting of many similar civilizations) will be orders of magnitude larger. But, of course, the hypothesis of extraterrestrial civilizations is entirely speculative, and deserves significantly less confidence than our lower-bound estimate of $6.6\hspace{0.17em}\times \hspace{0.17em}{10}^{17}$ for the size of the background population.³¹ We next consider two common objections to this lower-bound estimate.

6.2. The Distribution of Welfare

What about the distribution of welfare in the background population? Anything we say about this will of course be enormously speculative. However, since it is—according to non-additive views!—an important topic, it seems worth making a few brief remarks.

With respect to average welfare in the background population, two hypotheses seem particularly plausible.

Hypothesis 1

The background population consists mainly of small animals (whether terrestrial or extraterrestrial). Most of these animals have short natural lifespans, with high rates of infant mortality, so the average welfare level of the background population is likely close to zero. If the capacity for welfare scales with brain size or something similar, this would reinforce the same conclusion. Moreover, it seems plausible that average welfare in these populations will be negative, at least on a hedonic view of welfare (Horta 2010; Ng 1995). These assumptions imply, for instance, that $\text{AU,}\hspace{0.17em}\hspace{0.17em}\text{VV1}$ and $\text{VV2}$ converge with a version of $\text{CL}$ with a slightly negative critical level.

Hypothesis 2

The background population consists mainly of the members of advanced alien civilizations, with astronomically large population sizes driven by space settlement or other technological advances. Under this hypothesis, given the limits of our present knowledge, all bets are off: average welfare in the background population could be very high (Ord 2020: 235–39), very low (Sotala & Gloor 2017), or anything in between.

With respect to the distribution of welfare more generally, we have even less to say. There is clearly a wide range of welfare levels in the background population, leading to significant inequality within specific groups.³⁷ However, it could still turn out that the background population as a whole is dominated by welfare subjects who lead fairly similar lives—for example, by small animals who almost always experience lifetime welfare close to 0, or by members of a highly egalitarian alien civilization. This would lead to a low level of inequality, at least by standard measures.

7.1. Making the Question Precise: Existential Risk

We almost never face a choice between a certainty of catastrophe and a certainty of non-catastrophe. So, we suggest, the best way to understand the tradeoff between (i) and (ii) is in terms of the sacrifice the current generation might make in order to reduce existential risk, that is, the probability of existential catastrophe.⁴¹

To make this precise, let $Z$ denote a background population that includes, as usual, any past welfare subjects, as well as any present or future ones who will be unaffected by the choice. Let $F$ denote the future population that will exist only if we avoid existential catastrophe. Let $C$ denote the current generation; more specifically, let $C_w$ be a version of the current generation with average welfare $w$ (and fixed size $|C|$). The following risky prospect then represents a $p$ probability of existential catastrophe:

$P:C_w+Z\hspace{0.17em}\text{with}\hspace{0.17em}\text{probability}\hspace{0.17em}p;\hspace{0.17em}F+C_w+Z\hspace{0.17em}\text{otherwise}.$

From this baseline we can consider reducing the probability $p$ of catastrophe by an infinitesimal amount $\delta p$ while also decreasing the average welfare $w$ of the current generation by $\delta w$ to obtain a new prospect

$P^{\prime }:C_{w–\delta w}+Z\text{ with probability }p–\delta p; F+C_{w–\delta w}+Z\text{ otherwise}.$

Suppose we do this in such a way that $P$ and $P^{\prime }$ are equally good prospects. Then the ratio $\delta w/\delta p$ is an ‘exchange rate’ telling us how to weigh small changes in the welfare $w$ of the current generation against small changes in the probability $p$ of catastrophe. The higher the exchange rate, the greater the sacrifice that would be compensated by a marginal reduction in risk. So, formally, our question becomes:

Question 1. How important is existential catastrophe as measured by the exchange rate $\delta w/\delta p$? In particular, how does it depend on the relative sizes of $C,\hspace{0.17em}F$, and $Z$?

Unfortunately, one cannot read the answer to Question 1 directly off of our limit results, which, after all, say nothing about probabilities. The plan for the rest of the section is to explain why our limit results are nonetheless relevant, and then to give a concrete analysis using the estimates for population sizes that we developed in Section 6.

7.2. Expected Value and Cardinal Convergence

To address Question 1, we must adopt some rule for evaluating risky prospects like $P$. When—as in all our examples—an axiology is defined using a value function, the most obvious rule is to rank prospects by their expected value. We will assume that this is the appropriate rule for both $\text{AU}$ and for its limiting axiology ${\text{CL}}_{\overline{Z}}$. Let us call the extended theories $\mathbb{E}\text{AU}$ and $\mathbb{E}{\text{CL}}_{\overline{Z}}$, respectively, where the $\mathbb{E}$ stands for expected.⁴²

What justifies this assumption? When it comes to critical level views, there are foundational arguments supporting the use of expected value (see, e.g., Blackorby et al. 2005). For $\text{AU}$ we have less to go on, but maximizing expected average welfare seems like a natural default, and there is no alternative for $\text{AU}$ (or for other non-additive axiologies) that has achieved anything like widespread acceptance. Moreover, the use of expected value is closely connected to the idea that the value function $V_{\text{AU}}$ represents cardinal facts about value. If $V_{\text{AU}}(X)-V_{\text{AU}}(Y)$ is a measure of how much better it is to get $X$ instead of $Y$, we should expect $p\times (V_{\text{AU}}(X)-V_{\text{AU}}(Y))$ to be at least a rough measure of how much better it is to get $X$ instead of $Y$ with probability $p$. The use of expected value is a systematic development of this idea.

This connection explains why our results about cardinal convergence are, after all, relevant to Question 1. We can interpret $P$ and $P^{\prime }$ as involving three scenarios: (1) with probability $p-\delta p$, the catastrophe happens no matter which option is chosen; (2) with probability $\delta p$, choosing $P^{\prime }$ would successfully prevent the catastrophe; (3) with probability $1-p$, the catastrophe fails to happen no matter what. In each of these scenarios, $P$ and $P^{\prime }$ lead to different outcomes, and expected value theory effectively tells us to weigh up how much better or worse the outcome of $P$ would be than the outcome of $P^{\prime }$ in each scenario, using the probabilities as weights. When the background population is extremely large, Theorem 1 says that $V_{\text{AU}}$ and $V_{{\text{CL}}_{\overline{\text{Z}}}}$ will agree to high precision about the relative sizes of these differences in value. They will thus tend to agree about which option has higher expected value. And in particular, they will answer Question 1 in approximately the same way.

Here is the general theoretical lesson, stated somewhat informally. Suppose that axiology $\mathcal{A}$ converges cardinally with ${\mathcal{A}}^{\prime }$ relative to background populations of type $T$. For any two prospects, each involving finitely many possible outcomes, if there is certain to be a sufficiently large background population of type $T$, then $\mathcal{A}$ and ${\mathcal{A}}^{\prime }$ will agree about which prospect has higher expected value.

7.3. Analysis

With this set-up in hand, we now compare the answers to Question 1 given by $\mathbb{E}\text{AU}$ and by $\mathbb{E}{\text{CL}}_{\overline{Z}}$. We will give a general qualitative analysis of how the answers depend on the population sizes, illustrated numerically using some of our estimates from Section 6.

Let’s first consider the case of ${\text{CL}}_{\overline{Z}}$. Again, we will assume that the prospect $P$ is evaluated using its expected value $\mathbb{E}{V}_{{\text{CL}}_{\overline{\text{Z}}}}(P)$. By definition,

We can think of this expected value as a function of $p$ and $w$, while holding all other parameters fixed. Then the exchange rate is given by a ratio of derivatives:

(For example, if the expected value decreases rapidly as $p$ increases, but increases slowly as $w$ increases, then existential catastrophe is relatively important.) As is easy to deduce,

As one would anticipate, this quantity is positive (so some sacrifice is warranted) only if $\overline{F}$ is above the critical level $\overline{Z}$, and the importance of existential catastrophe scales linearly with $|F|$.

By contrast, using the value function $V_{\text{AU}}$ instead of $V_{{\text{CL}}_{\overline{\text{Z}}}}$, one finds

This expression is unattractive, but informative, and it simplifies greatly if we make further assumptions about population sizes. Consider the following three cases:

Case 1: $|\mathbf{\text{F}}|\gg \hspace{0.17em}|\mathbf{\text{C}}|\gg \hspace{0.17em}|\mathbf{\text{Z}}|$. In this case, with a small background population, $\delta w/\delta p$ is approximately $\frac{1}{p}(\overline{F}-w)$.⁴³ So it is roughly independent of the population sizes, and (in particular) does not scale with $|F|$. Moreover, in this approximation, reducing existential risk is only worthwhile if the future population would have average welfare higher than the current generation.

Case 2: $|\mathbf{\text{F}}|\gg \hspace{0.17em}|\mathbf{\text{Z}}|\gg \hspace{0.17em}|\mathbf{\text{C}}|$. In this case, where $Z$ is intermediate in size between $F$ and $C,\hspace{0.17em}\hspace{0.17em}\delta w/\delta p$ is approximately $\frac{1}{p}\frac{|Z|}{|C|}(\overline{F}-\overline{Z})$. In one way, this approximation agrees with ${\text{CL}}_{\overline{Z}}$: it is worth reducing existential risk insofar as $\overline{F}>\overline{Z}$. Moreover, $\delta w/\delta p$ will tend to be very large, proportional to $|Z|/|C|$. So, in this regime, existential catastrophe may be very important, but its importance is still insensitive to the size of $F$.

Case 3: $|\mathbf{\text{Z}}|\gg \hspace{0.17em}|\mathbf{\text{F}}|\gg \hspace{0.17em}|\mathbf{\text{C}}|$. In this case, where the background population is much larger than any of the potential foreground populations, $\delta w/\delta p$ is approximately $\frac{|F|}{|C|}(\overline{F}-\overline{Z})$. This is exactly the value we found using ${\text{CL}}_{\overline{Z}}$. In particular, for both $\text{AU}$ and ${\text{CL}}_{\overline{Z}}$, the importance of existential catastrophe scales with $|F|$ in this regime.

The most basic qualitative point to take away from this analysis is that $\delta w/\delta p$ increases without bound as we increase both $|F|$ and $|Z|$. The fact that possible future and actual background populations are both extremely large suggests that $\delta w/\delta p$ is likely to be large (thus favoring existential risk minimization) for a robust range of the other parameters.

7.4. Numerical Illustration

We now illustrate the preceding qualitative points using plausible numerical estimates of the various population sizes. The results are summarized in Table 1.

For the sizes of the foreground populations, we will suppose that $|C|={10}^{10}$ and $|F|={10}^{17}$. The former is a realistic estimate of the size of the present and near-future human population; the latter is a rough estimate of the potential size of the future human-originating population, supposing that we maintain current population sizes for as long as possible on Earth (see §6.1).

For the size of the background population, we will consider three values, which correspond exactly to Cases 1–3 above. First, just for illustration, we consider $|Z|=0$ (the case of no background population). Second, more realistically, $|Z|={10}^{13}$, a rounding-down of our most conservative estimate of the number of past mammals, weighted by lifespan and cortical neuron count, from §6.1.1. Finally, a somewhat less conservative estimate of $|Z|={10}^{20}$. This last value corresponds to our lower bound for the number of vertebrates in Earth’s past; alternatively, it could correspond to a background population dominated by 1000 alien civilizations, of the same scale that our civilization will achieve if we avoid existential catastrophe.

In terms of average welfare, we have less to go on, but the specific values are also less important for our present purposes. We will assume $\overline{Z}=0$ (plausible for the case where $Z$ consists mainly of wild animals, somewhat less plausible for the case where it consists mainly of advanced civilizations). If the current generation has positive average welfare, we can then choose units so that $\overline{C_w}=w=1$. And finally, for simplicity let us suppose $\overline{F}=2$ (ensuring that reducing existential risk will be worth some sacrifice in all three cases). Finally, we take $p=0.5$.

Table 1 gives the values of $\delta p/\delta w$ according to $\mathbb{E}\text{AU}$ and $\mathbb{E}{\text{CL}}_{\overline{Z}}$, under these assumptions, for all three background population sizes. Comparing the values in the third and fourth columns, we see that in this example, with three- or four-order-of-magnitude differences in the population sizes of $C,\hspace{0.17em}F$, and $Z$, the approximations used in the last subsection are accurate to at least the third significant figure. In particular, in the third case, where $|Z|\gg \hspace{0.17em}|F|\gg \hspace{0.17em}|C|,\mathbb{E}\text{AU}$ agrees with $\mathbb{E}{\text{CL}}_{\overline{Z}}$ to the third significant figure—preferring even very small reductions in the probability of existential catastrophe over a fairly substantial increase in the welfare of the current generation.

7.5. Conclusions

We have used the standard value functions defining $\text{AU}$ and ${\text{CL}}_{\overline{Z}}$ to analyze the expected value of reducing existential risk. This analysis yields the following conclusions: (1) When the background population $Z$ is small or non-existent, the importance of avoiding existential catastrophe according to $\text{AU}$ is approximately independent of population size, depending only on the average welfares of the potential foreground populations. It is therefore unlikely to be astronomically large. (2) When—as suggested by our most conservative estimates—$Z$ is much larger than the current generation but still much smaller than the potential future population, the importance of avoiding existential catastrophe according to $\text{AU}$ approximately scales with $|Z|$, and may therefore be extremely large, while still falling well short of its importance according to ${\text{CL}}_{\overline{Z}}$. (3) Finally, if the background population is much larger even than the potential future population $F$ (as it would be, for instance, if it includes many advanced civilizations elsewhere in the universe), $\text{AU}$ agrees closely with ${\text{CL}}_{\overline{Z}}$ about the importance of avoiding existential catastrophe, treating it as approximately linear in $|F|$.

8.1. Repugnant and Sadistic Addition

The Repugnant Conclusion, recall, is the conclusion (implied by TU among other axiologies) that for any two positive welfare levels , for any population in which everyone has welfare $w_2$, there is a better population in which everyone has welfare $w_1$. Avoidance of the Repugnant Conclusion is often seen as a significant desideratum in population axiology. But additive axiologies, as we have defined them, can avoid the Repugnant Conclusion only at the cost of implying the Strong Sadistic Conclusion: for any negative welfare level $w$, for any population in which everyone has welfare $w$, there is a worse population in which everyone has positive welfare (Arrhenius 2000).⁴⁴

One of the motivations for population axiologies with an ‘averagist’ flavor (like $\text{AU}$ and $\text{VV}$) is to avoid these unappealing consequences of additivity. These views do not imply either the Repugnant Conclusion or the Strong Sadistic Conclusion. But a straightforward implication of our results is that they imply both of the following, closely related conclusions.

Repugnant Addition. For any positive welfare levels and any population $X$ in which everyone has welfare $w_2$, there is a population $Y$ in which everyone has welfare $w_1$ and a population $Z$ such that $Y+Z\succ X+Z$.

Strong Sadistic Addition. For any negative welfare level $w$ and any population $X$ in which everyone has welfare $w$, there is a population $Y$ in which everyone has positive welfare and a population $Z$ such that $X+Z\succ Y+Z$.

Informally, where the Repugnant Conclusion says that for any imaginable utopia, there’s a better population in which everyone’s life is barely worth living, Repugnant Addition says that it’s sometimes better to add the latter population to a preexisting population. And likewise, where the Strong Sadistic Conclusion says that for any imaginable dystopia, there’s a worse population in which everyone’s life is worth living (if only barely), Strong Sadistic Addition says that it’s sometimes worse to add the latter population to a preexisting population.

A non-additive view will imply both of these conclusions if it can converge with an additive view with either a positive or a negative critical level. This includes $\text{AU,}\hspace{0.17em}\hspace{0.17em}\text{VV1,}\hspace{0.17em}\hspace{0.17em}\text{VV2}$, and all natural versions of two-factor egalitarianism.⁴⁵ AU yields Repugnant Addition, for instance, when there is a large background population with average welfare 0, and yields Strong Sadistic Addition when there is a large background population with positive average welfare.⁴⁶ We find these implications nearly as counterintuitive as the original Repugnant and Strong Sadistic Conclusions, though your mileage may vary. And non-additive views that imply both Repugnant Addition and Strong Sadistic Addition are, in one respect, worse off than any additive view, which will only imply one of the Repugnant and Strong Sadistic Conclusions.

These observations are not particularly original. Spears and Budolfson (2021) point out the difficulty of avoiding Repugnant Addition, for a broader range of axiologies than we have considered in this paper. And Franz and Spears (2020) show that, under modest assumptions, any view that rejects Mere Addition (including AU and similar views) will imply a weaker version of what we have called Strong Sadistic Addition. But our results provide a systematic and illuminating explanation for the difficulty of avoiding these unpalatable conclusions.

8.2. Exploitability

Our results also show that agents whose choices are guided by non-additive axiologies are vulnerable to a particular kind of exploitation. Suppose, for instance, that we in the Milky Way are all average utilitarians, while the inhabitants of the Andromeda Galaxy are all total utilitarians. And suppose that, the distance between the galaxies being what it is, we can communicate with each other but cannot otherwise interact. Being total utilitarians, the Andromedans would prefer that we act in ways that maximize total welfare in the Milky Way. To bring that about, they might create an astronomical number of welfare subjects with welfare very close to zero—for instance, very small, short-lived animals with mostly bland experiences—and send us evidence that they have done so. We in the Milky Way would then make all our choices under the awareness of a large background population whose average welfare is close to zero. The Andromedans could thus ‘force’ us to behave like de facto total utilitarians, doing the work of total utilitarianism on their behalf.

Agents who accept additive (or, more generally, separable) axiologies, on the other hand, are immune from this sort of exploitation. Thus non-additivists are at a practical disadvantage in strategic interactions with additivists. The incentive for others to exploit in this way also makes non-additive views potentially self-defeating: adopting and acting on such a view can incentivize other agents to act in ways that make things worse by its lights. For instance, the existence of average utilitarians incentivizes total utilitarians to add individuals with welfare 0 to the population, which makes things worse from the average utilitarian’s perspective if the average welfare of the preexisting population is positive.

We ourselves do not see this vulnerability as a particularly weighty reason to reject non-additive views—after all, nearly every agent is vulnerable to some forms of exploitation. (On the vulnerabilities of total utilitarians, for instance, see Gustafsson 2022a.) But it is certainly an unwelcome feature, and others may see it as a more severe drawback.