AI existential risk probabilities are (still) too unreliable to inform policy
打开互动全文版（逐段中英对照 + 针对本篇提问）→本文认为，通常被称为 p（doom）的 AI 存在性风险量化估计过于不可靠，无法作为公共政策的正当依据。作者指出，此类概率并非源自经过验证的模型或方法，而是以量化的外衣包装模糊的直觉与恐惧。他们考察了概率估计的三种可用依据——归纳、演绎与主观——并表明三者均无法为 AI 存在性风险预测提供基础：人类灭绝缺乏合适的参考类别，技术进步与失控缺乏可靠的演绎模型，而主观估计的差异可达数个数量级。尽管作者并不反对将 AI 存在性风险预测作为学术活动或用于私人决策，但他们主张，政策制定者应认识到这些数字缺乏正当、高昂且分布不均的政策措施所需的正当性依据。结论是，政府在限制 AI 开发或开放发布时，不应将具体的 p（doom）数值视为权威证据。
This essay argues that quantitative estimates of AI existential risk, commonly termed p(doom), are too unreliable to serve as a legitimate basis for public policy. The authors contend that such probabilities are not derived from validated models or methods but instead launder vague intuitions and fears through a facade of quantification. They examine the three available justifications for probability estimates—inductive, deductive, and subjective—and show that none can ground AI x-risk forecasts: there is no suitable reference class for human extinction, no reliable deductive model of technological progress and loss of control, and subjective estimates vary by orders of magnitude. While the authors do not oppose AI x-risk forecasting as an academic activity or for private decision-making, they argue that policymakers should recognize these figures lack the justification required for legitimate, costly, and unevenly distributed policy measures. The conclusion is that governments should not treat specific p(doom) numbers as authoritative evidence when restricting AI development or open releases.
两年多前，我们曾对 p（doom）进行过详细的解构，并指出其主要作用是通过量化的外表来洗白模糊的直觉和恐惧。如今，随着 p（doom）言论前所未有地推动公共讨论和政策关注，这篇文章重新具有现实意义。毫不意外的是，当今对 AI 存在性风险的概率估计并不比 2024 年的更严谨。因此重新发布此文。
Over two years ago we wrote a detailed deconstruction of p(doom) and argued that its primary effect is to launder vague intuitions and fears through a facade of quantification. The essay has fresh relevance today as p(doom) rhetoric is driving public discourse and policy attention to an unprecedented degree. Unsurprisingly, today’s probability estimates of AI existential risk are no more rigorous than the ones from 2024. Hence this repost.
我们需要明确我们**没有**在说什么。
We should be clear about what we’re not saying.
* 我们并不是说预测者（或其他试图量化 AI 存在性风险的人）有意误导。但不幸的是，我们认为其效果仍然是误导性的。 * 一般来说，对预测进行量化是好事。事实上，Arvind 一直担任预测研究所纵向专家 AI 小组的顾问，他认为这项工作极具价值。正如文章所明确指出的，构建有用的定量模型不可行是 AI 存在性风险预测所特有的问题，而非一般预测的问题。 * 我们此处的目标并不是试图推动 AI 安全倡导者停止用概率进行交流（即使尝试了也未必能成功）。相反，我们发表此文是希望那些接收这些概率的人——尤其是政策制定者——认识到，与我们遇到的几乎所有其他定量预测不同，这些概率并非来自经过验证的模型或方法。
* We aren’t saying forecasters (or others who attempt to quantify AI x-risk) intend to mislead. Unfortunately, we think the effect is misleading nonetheless. * In general, being quantitative about forecasts is good. In fact, Arvind has been serving as an advisor to the Forecasting Research Institute’s Longitudinal Expert AI Panel, an effort he thinks is extremely valuable. As the essay makes clear, the infeasibility of building useful quantitative models is specific to forecasts of AI x-risk, not forecasting in general. * Our goal here is not to try to push AI safety advocates to stop communicating in terms of probabilities (not that we’d succeed even if we tried). Rather, our hope in publishing this is for those at the receiving end of these probabilities — especially policymakers — to recognize that unlike almost every other quantitative forecast that we encounter, they are not the result of a validated model or method.
Over two years ago we wrote a detailed deconstruction of_p(doom)_ and argued that its primary effect is to launder vague intuitions and fears through a facade of quantification. The essay has fresh relevance today as _p(doom)_ rhetoric is driving public discourse and policy attention to an unprecedented degree. Unsurprisingly, today’s probability estimates of AI existential risk are no more rigorous than the ones from 2024. Hence this repost.
We should be clear about what we’re _not_ saying.
尽管如此，我们认为整个 _p(doom)_ 文化象征着一种特定的 AI 安全观，这种观点以存在性风险和超级智能为中心，实际上不利于更广义的安全概念。几天后我们将发表一篇后续文章，阐述这一论点并倡导一场“大帐篷”式的安全运动。
That said, we think the whole _p(doom)_ culture is emblematic of a particular view of AI safety that centers existential risk and superintelligence in a way that is actively counterproductive to a broader conception of safety. In a few days we'll publish a follow-up essay making that argument and advocating for a "big tent" safety movement.
鉴于研究人员之间缺乏共识，政府应多大程度上重视 AI 带来的存在性风险威胁？一方面，存在性风险（x-risks）必然带有一定推测性：等到有具体证据时，可能为时已晚。另一方面，政府必须分清轻重缓急——毕竟，他们并不太担心外星入侵带来的存在性风险。
How seriously should governments take the threat of existential risk from AI, given the lack of consensus among researchers? On the one hand, existential risks (x-risks) are necessarily somewhat speculative: by the time there is concrete evidence, it may be too late. On the other hand, governments must prioritize — after all, they don’t worry too much about x-risk from alien invasions.
AI 安全社区严重依赖预测 AI 导致人类灭绝的概率（在特定时间范围内），以便为决策和政策提供依据。例如，几十年内 10% 的估计显然足以使该问题成为社会的首要任务。
The AI safety community relies heavily on forecasting the probability of human extinction due to AI (in a given timeframe) in order to inform decision making and policy. An estimate of 10% over a few decades, for example, would obviously be high enough for the issue to be a top priority for society.
我们的核心主张是，AI 存在性风险预测过于不可靠，无法用于政策，而且实际上极具误导性。
Our central claim is that AI x-risk forecasts are far too unreliable to be useful for policy, and in fact highly misleading.
如果我们俩预测外星人在未来十年内登陆地球的概率是 80%，你会认真对待这个可能性吗？当然不会。你会要求看我们的证据。尽管这看起来显而易见，但在 AI 存在性风险的辩论中，似乎已经忘记了概率本身并不具有权威性。概率_通常_是从某种有根据的方法中推导出来的，因此我们有一种强烈的认知偏差，认为量化的风险估计比定性的风险估计更有效。但概率也可能仅仅是猜测。在整篇文章中（以及在更广泛的 AI 存在性风险辩论中）请记住这一点。
If the two of us predicted an 80% probability of aliens landing on earth in the next ten years, would you take this possibility seriously? Of course not. You would ask to see our evidence. As obvious as this may seem, it seems to have been forgotten in the AI x-risk debate that probabilities carry no authority by themselves. Probabilities are _usually_ derived from some grounded method, so we have a strong cognitive bias to view quantified risk estimates as more valid than qualitative ones. But it is possible for probabilities to be nothing more than guesses. Keep this in mind throughout this essay (and more broadly in the AI x-risk debate).
如果我们预测肯塔基德比大赛的赔率，我们不必给你理由——你可以接受或不接受。但如果政策制定者根据预测者提出的概率采取行动，他们最好能够向公众解释这些概率（并且该解释必须反过来来自预测者）。正当性对于政府的合法性和权力的行使至关重要。自由民主的一个核心原则是，国家不应基于合理的人可以拒绝的有争议的信念来限制人们的自由。
If we predicted odds for the Kentucky Derby, we don’t have to give you a reason — you can take it or leave it. But if a policymaker takes actions based on probabilities put forth by a forecaster, they had better be able to explain those probabilities to the public (and that explanation must in turn come from the forecaster). Justification is essential to legitimacy of government and the exercise of power. A core principle of liberal democracy is that the state should not limit people’s freedom based on controversial beliefs that reasonable people can reject.
当考虑的政策成本高昂时，解释尤为重要，而当这些成本在利益相关者之间分布不均时，就更是如此。一个很好的例子是限制 AI 模型的开放发布。政府能否说服那些将从开放模型中受益的人和公司，让他们因为一个推测性的未来风险而做出这种牺牲？
Explanation is especially important when the policies being considered are costly, and even more so when those costs are unevenly distributed among stakeholders. A good example is restricting open releases of AI models. Can governments convince people and companies who stand to benefit from open models that they should make this sacrifice because of a speculative future risk?
本文的主要目的是分析政策辩论中引用的任何具体的存在性风险概率估计是否有任何正当理由。我们不反对将 AI 存在性风险预测作为一种学术活动，预测可能对公司和其他私人决策者有帮助。我们只质疑其在公共政策背景下的使用。
The main aim of this essay is analyzing whether there is any justification for any of the specific x-risk probability estimates that have been cited in the policy debate. We have no objection to AI x-risk forecasting as an academic activity, and forecasts may be helpful to companies and other private decision makers. We only question its use in the context of public policy.
基本上只有三种已知的方法可以让预测者试图说服怀疑者：归纳法、演绎法和主观概率估计。我们将在以下各节中逐一考虑。所有三种方法都要求双方就世界的某些基本假设达成一致（这些假设本身无法被证明）。这三种方法的区别在于概率估计从该组假设中得出的经验和逻辑方式。
There are basically only three known ways by which a forecaster can try to convince a skeptic: inductive, deductive, and subjective probability estimation. We consider each of these in the following sections. All three require both parties to agree on some basic assumptions about the world (which cannot themselves be proven). The three approaches differ in terms of the empirical and logical ways in which the probability estimate follows from that set of assumptions.
大多数风险估计是归纳性的：它们基于过去的观察。例如，保险公司根据过去类似驾驶员的事故数据来预测个人发生车祸的风险。用于概率估计的观察集合称为参考类。对于汽车保险，一个合适的参考类可能是居住在同一城市的驾驶员集合。如果分析师拥有关于该个体的更多信息，例如年龄或驾驶的汽车类型，参考类可以进一步细化。
Most risk estimates are inductive: they are based on past observations. For example, insurers base their predictions of an individual's car accident risk on data from past accidents about similar drivers. The set of observations used for probability estimation is called a reference class. A suitable reference class for car insurance might be the set of drivers who live in the same city. If the analyst has more information about the individual, such as their age or the type of car they drive, the reference class can be further refined.
对于人工智能的存在性风险，不存在参考类，因为它是独一无二的事件。需要明确的是，这是一个程度问题，而非种类问题。从来不存在一个明确的“正确”参考类可供使用，实践中参考类的选择取决于分析师的直觉。
For existential risk from AI, there is no reference class, as it is an event like no other. To be clear, this is a matter of degree, not kind. There is never a clear "correct" reference class to use, and the choice of a reference class in practice comes down to the analyst's intuition.
预测的准确性取决于被预测事件的生成过程与参考类中事件的生成过程之间的相似程度，这可以看作一个谱系。对于预测物理系统（如抛硬币）的结果，过去的经验是非常可靠的指南。其次，对于车祸，风险估计可能因所使用的历史数据集而相差约 20%——这对保险公司来说已经足够好了。
The accuracy of the forecasts depends on the degree of similarity between the process that generates the event being forecast and the process that generated the events in the reference class, which can be seen as a spectrum. For predicting the outcome of a physical system such as a coin toss, past experience is a highly reliable guide. Next, for car accidents, risk estimates might vary by, say, 20% based on the past dataset used — good enough for insurance companies.
在这个谱系上更远的是地缘政治事件，其中参考类的选择变得更加模糊。预测专家 Philip Tetlock 解释说：“希腊退出欧元区可能看起来是独特的，因为截至 2015 年还没有国家退出欧元区，但它也可以被视为一个广泛比较类（如谈判失败）的另一个实例，或者一个更窄的类（如民族国家退出国际协议），或者更窄的类（如强制货币兑换）。”他接着辩护说，即使是像苏联解体或阿拉伯之春这样看似黑天鹅的事件，也可以建模为参考类的成员，并且归纳推理甚至对这种事件也是有用的。
Further along the spectrum are geopolitical events, where the choice of reference class gets even fuzzier. Forecasting expert Philip Tetlock explains: "Grexit may have looked sui generis, because no country had exited the Eurozone as of 2015, but it could also be viewed as just another instance of a broad comparison class, such as negotiation failures, or of a narrower class, such as a nation-states withdrawing from international agreements or, narrower still, of forced currency conversions." He goes on to defend the idea that even seeming Black Swan events like the collapse of the USSR or the Arab Spring can be modeled as members of reference classes, and that inductive reasoning is useful even for this kind of event.
在 Tetlock 的谱系中，这些事件代表了独特性的“顶峰”。就地缘政治事件而言，这可能是真的。但即使是这些事件，也远不如人工智能导致的灭绝那样独特。只需看看为人工智能存在性风险寻找参考类的尝试：动物灭绝（作为人类灭绝的参考类）、过去的全球转型如工业革命（作为人工智能导致的社会经济转型的参考类），或导致大规模死亡的事故（作为导致全球灾难的事故的参考类）。现实一点吧。这些都没有告诉我们任何关于开发超级智能人工智能或失去对此类人工智能控制的可能性，而这些是人工智能存在性风险预测中不确定性的核心来源。
In Tetlock's spectrum, these events represent the "peak" of uniqueness. When it comes to geopolitical events, that might be true. But even those events are far less unique than extinction from AI. Just look at the attempts to find reference classes for AI x-risk: animal extinction (as a reference class for human extinction), past global transformations such as the industrial revolution (as a reference class for socioeconomic transformation from AI), or accidents causing mass deaths (as a reference class for accidents causing global catastrophe). Let's get real. None of those tell us anything about the possibility of developing superintelligent AI or losing control over such AI, which are the central sources of uncertainty for AI x-risk forecasting.
总而言之，AI 导致的人类灭绝是一种与过去发生过的任何事情都相去甚远的结果，以至于我们无法用归纳方法来“预测”其概率。当然，我们可以从过去的技术突破以及过去的灾难性事件中获得定性洞见，但 AI 风险足够不同，以至于定量估计缺乏在政策制定中取得正当性所需的那种依据。
To summarize, human extinction due to AI is an outcome so far removed from anything that has happened in the past that we cannot use inductive methods to "predict" the odds. Of course, we can get qualitative insights from past technical breakthroughs as well as past catastrophic events, but AI risk is sufficiently different that quantitative estimates lack the kind of justification needed for legitimacy in policymaking.
在柯南·道尔的《六座拿破仑半身像》中——剧透预警！——夏洛克·福尔摩斯在开始监视前宣布，抓住嫌疑人的概率正好是三分之二。这似乎令人困惑——任何与人类行为相关的事情，怎么能被赋予数学上精确的概率呢？
In Conan Doyle’s _The Adventure of the Six Napoleons_ — spoiler alert! — Sherlock Holmes announces before embarking on a stakeout that the probability of catching the suspect is exactly two-thirds. This seems bewildering — how can anything related to human behavior be ascribed a mathematically precise probability?
事实证明，福尔摩斯已经推断出了导致嫌疑人看似古怪的观察行为的一系列潜在事件：嫌疑人正在有条不紊地寻找一颗已知藏在伦敦及其周边不同人拥有的六座拿破仑半身像之一的宝石。细节并不太重要，但关键在于，嫌疑人和侦探都不知道它藏在六座半身像中的哪一座里，而关于嫌疑人行为的其他一切（被假定为）完全可预测。因此，不确定性可以被精确量化。
It turns out that Holmes has deduced the underlying series of events that gave rise to the suspect’s seemingly erratic observed behavior: the suspect is methodically searching for a jewel that is known to be hidden inside one of six busts of Napoleon owned by different people in and around London. The details aren’t too important, but the key is that neither the suspect nor the detectives know _which_ of the six busts it is in, and everything else about the suspect’s behavior is (assumed to be) entirely predictable. Hence the precisely quantifiable uncertainty.
关键在于，如果我们有一个可以依赖的世界模型，我们就可以通过逻辑演绎来估计风险，甚至不需要依赖过去的观察。当然，在虚构场景之外，世界并不那么整洁，尤其是当我们想要预测遥远的未来时。
The point is that if we have a model of the world that we can rely upon, we can estimate risk through logical deduction, even without relying on past observations. Of course, outside of fictional scenarios, the world isn’t so neat, especially when we want to project far into the future.
当涉及存在性风险（x-risk）时，有一个有趣的例外，即小行星撞击，它不符合我们没有演绎模型的普遍规律。归纳和演绎风险估计的结合确实允许我们估计 x-risk 的概率，但仅仅是因为我们讨论的是一个纯物理系统。让我们花点时间回顾一下这是如何运作的，因为重要的是要认识到这些方法不能推广到其他类型的 x-risk。
When it comes to x-risk, there is an interesting exception to the general rule that we don’t have deductive models — asteroid impact. A combination of inductive and deductive risk estimation does allow us to estimate the probability of x-risk, only because we’re talking about a purely physical system. Let’s take a minute to review how this works, because it’s important to recognize that the methods are not generalizable to other types of x-risk.
关键在于能够对小行星的大小（更准确地说，撞击能量）与撞击频率之间的关系进行建模。由于我们已经观测到了数千次小撞击，我们可以外推以推断从未被直接观测到的大撞击的频率。我们还可以估计会导致全球灾难的阈值。1
The key is being able to model the relationship between the size of the asteroid (more precisely, the energy of impact) and the frequency of impact. Since we have observed thousands of small impacts, we can extrapolate to infer the frequency of large impacts that have never been directly observed. We can also estimate the threshold that would cause global catastrophe.1
图：关于小行星撞击的数据（左侧图示）可外推至灭绝级撞击（右侧）。
Figure: data on small asteroid impacts (illustrated on the left) can be extrapolated to extinction-level impacts (right).
对于 AI，未知因素涉及技术进步与治理，而非某个物理系统，因此如何对其进行数学建模并不清楚。尽管如此，人们仍做过尝试。例如，为预测一个假想 AGI 的算力需求，若干研究假设 AI 系统所需的计算量大致相当于人脑，并进一步对人脑所需的计算量作出假设。这些假设远比小行星建模中所涉及的假设更为薄弱，而且这一切甚至都没有触及失控问题。
With AI, the unknowns relate to technological progress and governance rather than a physical system, so it isn't clear how to model it mathematically. Still, people have tried. For example, in order to predict the computational requirements of a hypothetical AGI, several works assume that an AI system would require roughly as many computations as the human brain, and further make assumptions about the number of computations required by the human brain. These assumptions are far more tenuous than those involved in asteroid modeling, and none of this even addresses the loss-of-control question.
在没有参考类别或可靠理论的情况下，预测必然是“主观概率”，即基于预测者判断的猜测。不出所料，这些猜测会相差数个数量级。
Without the reference classes or grounded theories, forecasts are necessarily "subjective probabilities", that is, guesses based on the forecaster's judgment. Unsurprisingly, these vary by orders of magnitude.
主观概率估计并不能绕开这样一个需求：概率估计必须具有归纳或演绎的基础。它仅仅免除了预测者_解释_其估计的必要性。由于人类解释自身直觉推理（无论是归纳、演绎还是二者的结合）的能力有限，解释可能很困难。本质上，它允许预测者说：“尽管我没有展示我的方法，但你可以因为我的过往记录而信任这个估计”（我们在下一节解释为何即便这一点在 AI 存在性风险预测中也会失效）。但归根结底，在既无归纳基础又无演绎基础的情况下，预测者所能做的只是编造一个数字，而这些编造出来的数字五花八门。
Subjective probability estimation does not get around the need for having either an inductive or a deductive basis for probability estimates. It merely avoids the need for the forecaster to _explain_ their estimate. Explanation can be hard due to humans' limited ability to explain our intuitive reasoning, whether inductive, deductive, or a combination thereof. Essentially, it allows the forecaster to say: "even though I haven't shown my methods, you can trust this estimate because of my track record" (we explain in the next section why even this breaks down for AI x-risk forecasting). But ultimately, lacking either an inductive or a deductive basis, all that forecasters can do is to make up a number, and those made-up numbers are all over the place.
以预测研究所于 2022 年底开展的“存在性风险说服锦标赛”（XPT）为例，我们认为这是迄今为止最精细、执行得最好的存在性风险预测活动。它涉及多组预测者，包括 AI 专家和预测专家（图中称为“超级预测者”）。对于 AI 专家，对 2100 年前 AI 灭绝风险估计的高端（第 75 百分位）为 12%，中位估计为 3%，低端（第 25 百分位）为 0.25%。对于预测专家，即便是高端（第 75 百分位）也仅为 1%，中位数仅为 0.38%，而低端（第 25 百分位）在图上与零在视觉上无法区分。换言之，AI 专家预测的第 75 百分位与超级预测者预测的第 25 百分位至少相差 100 倍。
Consider the Existential Risk Persuasion Tournament (XPT) conducted by the Forecasting Research Institute in late 2022, which we think is the most elaborate and well-executed x-risk forecasting exercise conducted to date. It involved various groups of forecasters, including AI experts and forecasting experts ("superforecasters" in the figure). For AI experts, the high end (75th percentile) of estimates for AI extinction risk by 2100 is 12%, the median estimate is 3%, and the low end (25th percentile) is 0.25%. For forecasting experts, even the high end (75th percentile) is only 1%, the median is a mere 0.38%, and the low end (25th percentile) is visually indistinguishable from zero on the graph. In other words, the 75th percentile AI expert forecast and the 25th percentile superforecaster forecast differ by at least a factor of 100.
所有这些估计都来自在该主题上具有深厚专业知识、并参加了长达数月、试图相互说服的锦标赛的人！如果这里的预测范围还不够极端，请记住，这整个活动是由一个群体在一个时间点开展的。如果今天重复该锦标赛、如果问题的表述方式不同，等等，我们可能会得到不同的数字。
All of these estimates are from people who have deep expertise on the topic and participated in a months-long tournament where they tried to persuade each other! If this range of forecasts here isn't extreme enough, keep in mind that this whole exercise was conducted by one group at one point in time. We might get different numbers if the tournament were repeated today, if the questions were framed differently, etc.
最能说明问题的是预测者提供的理由，报告中对这些理由做了详尽阐述。他们并未使用定量模型，尤其是在思考在开发出强大 AI 的条件下出现坏结果的概率时。在很大程度上，预测者所做的推测与普通人在讨论超级智能 AI 时所做的推测属于同一类。也许 AI 会通过对系统操作员的超人说服力接管关键系统。也许 AI 会试图降低全球温度，因为这有助于计算机运行得更快，却意外地消灭人类。或者也许 AI 会去太空而非地球寻求资源，因此我们不必那么担心。这类推测本身没有错。但我们应该清楚，在 AI 存在性风险问题上，预测者并没有借助任何特殊知识、证据或模型来使其直觉比你我或任何其他人的直觉更可信。
What's most telling is to look at the rationales that forecasters provided, which are extensively detailed in the report. They aren't using quantitative models, especially when thinking about the likelihood of bad outcomes conditional on developing powerful AI. For the most part, forecasters are engaging in the same kind of speculation that everyday people do when they discuss superintelligent AI. Maybe AI will take over critical systems through superhuman persuasion of system operators. Maybe AI will seek to lower global temperatures because it helps computers run faster, and accidentally wipe out humanity. Or maybe AI will seek resources in space rather than Earth, so we don't need to be as worried. There's nothing wrong with such speculation. But we should be clear that when it comes to AI x-risk, forecasters aren't drawing on any special knowledge, evidence, or models that make their hunches more credible than yours or ours or anyone else's.
“超级预测者”一词源自菲利普·泰特洛克（Philip Tetlock）为期 20 年的预测研究（他也是 XPT 的组织者之一）。超级预测者往往接受过改进预测方法的训练，例如整合多样信息和最小化心理偏差。这些方法已被证明在地缘政治等领域有效。但如果没有多少有用的证据可供参考，再多的训练也无法产生良好的预测。
The term superforecasting comes from Philip Tetlock's 20-year study of forecasting (he was also one of the organizers of the XPT). Superforecasters tend to be trained in methods to improve forecasts such as by integrating diverse information and by minimizing psychological biases. These methods have been shown to be effective in domains such as geopolitics. But no amount of training will lead to good forecasts if there isn't much useful evidence to draw from.
即使预测者拥有可信的定量模型（他们并没有），他们也必须考虑“未知的未知”，即模型本身可能出错的可能性。正如 x-risk 哲学家尼克·博斯特罗姆（Nick Bostrom）所解释的那样：“我们对风险的一阶评估的不确定性和易错性本身，就是我们必须纳入全盘考虑的概率分配中的因素。在低概率、高后果的风险中——尤其是那些涉及理解不足的自然现象、复杂的社会动态或新技术，或由于其他原因难以评估的风险——这一因素往往占主导地位。”
Even if forecasters had credible quantitative models (they don't), they must account for "unknown unknowns", that is, the possibility that the model itself might be wrong. As noted x-risk philosopher Nick Bostrom explains: "The uncertainty and error-proneness of our first-order assessments of risk is itself something we must factor into our all-things-considered probability assignments. This factor often dominates in low-probability, high-consequence risks — especially those involving poorly understood natural phenomena, complex social dynamics, or new technology, or that are difficult to assess for other reasons."
这是一个合理的观点，AI x-risk 预测者确实非常担心风险评估中的不确定性。但这样做的一个后果是，对于那些遵循这一原则的人来说，预测_必然_是猜测，而不是模型的输出——毕竟，没有模型可以用来估计模型本身出错的可能性，或者如果模型出错风险会是什么。
This is a reasonable perspective, and AI x-risk forecasters do worry a lot about uncertainty in risk assessment. But one consequence of this is that for those who follow this principle, forecasts are _guaranteed_ to be guesses rather than the output of a model — after all, no model can be used to estimate the probability that the model itself is wrong, or what the risk would be if the model were wrong.
回顾一下，主观的 AI 风险预测相差几个数量级。但如果我们能衡量预测者的过往记录，也许就能借此判断哪些预测者值得信赖。与之前两种证明风险估计合理性的方法（归纳和演绎）不同，预测者不必解释其估计，而是基于其在预测过去**其他**结果方面所展示的技能来证明其合理性。
To recap, subjective AI-risk forecasts vary by orders of magnitude. But if we can measure forecasters’ track records, maybe we can use that to figure out which forecasters to trust. In contrast to the previous two approaches for justifying risk estimates (inductive and deductive), the forecaster doesn’t have to explain their estimate, but instead justifies it based on their demonstrated skill at predicting _other_ outcomes in the past.
这在地缘政治事件领域已被证明是非常有价值的，预测社区在技能衡量上投入了大量精力。评估预测技能的方法有很多，例如校准、Brier 分数、对数分数，或预测竞赛网站 Metaculus 上使用的 Peer 分数。
This has proved to be invaluable in the domain of geopolitical events, and the forecasting community spends a lot of effort on skill measurement. Many ways to evaluate forecasting skill exist, such as calibration, the Brier score, the logarithmic score, or the Peer score used on the forecasting competition website Metaculus.
但无论使用哪种方法，在存在性风险方面，评估主观概率的预测技能存在许多障碍：缺乏参考类、低基础率和长时间跨度。让我们逐一来看。
But regardless of which method is used, when it comes to existential risk, there are many barriers to assessing forecast skill for subjective probabilities: the lack of a reference class, the low base rate, and the long time horizon. Let’s look at each of these in turn.
正如参考类问题困扰预测者一样，它也影响评估者。让我们回到外星人登陆的例子。考虑一位在预测选举方面被证明高度准确的预测者。假设这位预测者在没有任何证据的情况下宣布，外星人将在一年内登陆地球。尽管该预测者展示了技能，但这不会促使我们更新对外星人登陆的信念，因为它与选举预测太不相似，我们并不预期该预测者的技能能够泛化。同样，AI 存在性风险与任何被预测过的过去事件都如此不同，以至于没有证据表明任何预测者具备估计 AI 存在性风险的技能。
Just as the reference class problem plagues the forecaster, it also affects the evaluator. Let’s return to the alien landing example. Consider a forecaster who has proved highly accurate at calling elections. Suppose this forecaster announces, without any evidence, that aliens will land on Earth within a year. Despite the forecaster’s demonstrated skill, this would not cause us to update our beliefs about an alien landing, because it is too dissimilar to election forecasting and we do not expect the forecaster’s skill to generalize. Similarly, AI x-risk is so dissimilar to any past events that have been forecast that there is no evidence of any forecaster’s skill at estimating AI x-risk.
即使我们以某种方式消除了参考类问题，其他问题仍然存在——尤其是灭绝风险是“尾部风险”这一事实，即由罕见事件导致的风险。假设预测者 A 说 AI 存在性风险的概率是 1%，预测者 B 说是百万分之一。我们应该对哪个预测更有信心？我们可以查看他们的过往记录。假设我们发现预测者 A（将 AI 存在性风险的概率定为 1%）有更好的过往记录。这仍然不意味着我们应该对 A 的预测更有信心，因为**技能评估对尾部风险的高估不敏感**。换句话说，A 的总分更高可能是因为 A 在发生概率较大的日常事件上比 B 校准得稍好，但往往将罕见的尾部风险（例如概率为百万分之一的事件）高估几个数量级。没有评分规则能充分惩罚这种类型的校准错误。
Even if we somehow do away with the reference class problem, other problems remain — notably, the fact that extinction risks are “tail risks”, or risks that result from rare events. Suppose forecaster A says the probability of AI x-risk is 1%, and forecaster B says it is 1 in a million. Which forecast should we have more confidence in? We could look at their track records. Say we find that forecaster A (who has assigned a 1% probability to AI x-risk) has a better track record. It still doesn’t mean we should have more confidence in A’s forecast, because _skill evaluations are insensitive to overestimation of tail risks_. In other words, it could be that A scores higher overall because A is slightly better calibrated than B when it comes to everyday events that have a substantial probability of occurring, but tends to massively overestimate tail risks that occur rarely (for example, those with a probability of 1 in a million) by orders of magnitude. No scoring rule adequately penalizes this type of miscalibration.
这里有一个思想实验来说明为什么这是真的。假设两位预测者 F 和 G 预测两组不同的事件，并且两组中事件的“真实”概率在 0 到 1 之间均匀分布。我们高度乐观地假设 F 和 G 都知道他们所预测的每个事件\(e\)的真实概率\(P_e\)。F 总是输出\(P_e\)，但 G 稍微保守，从不预测小于 1%的值。也就是说，如果\（P_e \geq 1\%），G 输出\(P_e\)，否则输出 1%。
Here’s a thought experiment to show why this is true. Suppose two forecasters F and G forecast two different sets of events, and the “true” probabilities of events in both sets are uniformly distributed between 0 and 1. We assume, highly optimistically, that both F and G know the true probability \(P_e\) for every event \(e\) that they forecast. F always outputs \(P_e\), but G is slightly conservative, never predicting a value less than 1%. That is, G outputs \(P_e\) if \(P_e \geq 1\%), otherwise outputs 1%.
根据构造，F 是更好的预测者。但这会从他们的历史记录中明显看出吗？换句话说，我们需要评估每个人多少次预测，才能有 95%的概率 F 得分高于 G？使用对数评分规则，结果大约是 1 亿次。使用 Brier 分数，大约是 1 万亿次。我们可以对这些假设提出质疑，但关键是，如果预测者系统性地高估尾部风险，这根本是无法通过经验检测的。
By construction, F is the better forecaster. But would this be evident from their track records? In other words, how many forecasts from each would we have to evaluate so that there’s a 95% chance that F outscores G? With the logarithmic scoring rule, it turns out to be on the order of a hundred million. With the Brier score, it is on the order of a trillion. We can quibble with the assumptions here but the point is that if a forecaster systematically overestimates tail risks, it is simply empirically undetectable.
评估预测者在预测 x-risk 方面的技能的最后一个障碍是，长期预测需要太长时间来评估（当然，灭绝预测是不可能评估的）。这有可能被克服。研究人员开发了一种称为相互评分的方法——预测者根据他们预测彼此预测的好坏来获得奖励——并在一些现实世界环境中进行了验证，例如预测 Covid-19 政策的效果。在这些环境中，相互评分产生的预测与传统评分方法一样好。这很合理。但相互评分并不能绕过参考类问题或尾部风险问题。
The final barrier to assessing forecaster skill at predicting x-risk is that long-term forecasts take too long to evaluate (and extinction forecasts are of course impossible to evaluate). This can potentially be overcome. Researchers have developed a method called reciprocal scoring — where forecasters are rewarded based on how well they predict each others’ forecasts — and validate it in some real-world settings, such as predicting the effect of Covid-19 policies. In these settings, reciprocal scoring yielded forecasts that are as good as traditional scoring methods. Fair enough. But reciprocal scoring is not a way around the reference class problem or the tail risk problem.
_到目前为止我们的论证总结，说明为什么三种预测方法都无法对 AI x-risk 给出可信的估计。_
_Summary of our argument so far, showing why none of the three forecasting methods can yield credible estimates of AI x-risk._
回顾一下，归纳和演绎方法都不起作用，主观预测五花八门，而且无法判断哪些预测更可信。
To recap, inductive and deductive methods don't work, subjective forecasts are all over the place, and there's no way to tell which forecasts are more trustworthy.
因此，为了得出可能为政策提供参考的更可靠估计，一些研究人员转向了预测聚合方法，即结合多个预测者的预测。一个值得注意的努力是 AI Impacts 关于 AI 进展的调查，但它因包括无应答偏差在内的严重方法学局限性而受到批评。更重要的是，目前尚不清楚为什么聚合应该提高预测准确性：毕竟，大多数预测者可能具有相同的偏差（而且再次强调，他们中没有人有可靠预测的依据）。
So in an attempt to derive more reliable estimates that could potentially inform policy, some researchers have turned to forecast aggregation methods that combine the predictions of multiple forecasters. A notable effort is the AI Impacts Survey on Progress in AI, but it has been criticized for serious methodological limitations including non-response bias. More importantly, it is unclear why aggregation should improve forecast accuracy: after all, most forecasters might share the same biases (and again, none of them have any basis for a reliable forecast).
预测者可能系统性高估 AI 存在风险的原因有很多。首先是选择偏差。以 AI 研究人员为例：相信 AI 能够改变世界是成为 AI 研究人员的的主要动机之一。一旦某人进入这个社区，他们就处于一个不断强化这一信息的环境中。如果一个人相信这项技术具有可怕的强大力量，那么认为其改变世界的影响很可能是负面而非正面的，这是完全理性的。
There are many reasons why forecasters might systematically overestimate AI x-risk. The first is selection bias. Take AI researchers: the belief that AI can change the world is one of the main motivations for becoming an AI researcher. And once someone enters this community, they are in an environment where that message is constantly reinforced. And if one believes that this technology is terrifyingly powerful, it is perfectly rational to think there is a serious chance that its world-altering effects will be negative rather than positive.
而在 AI 安全子社区中，这个社区有点孤立，回音室效应可能震耳欲聋。声称拥有高 \(p_{\text{doom}})\（一个人对 AI 末日概率的估计）似乎已经成为一种表明自己身份和对事业承诺的方式。
And in the AI safety subcommunity, which is a bit insular, the echo chamber can be deafening. Claiming to have a high \(p_{\text{doom}})\ (one's estimate of the probability of AI doom) seems to have become a way to signal one's identity and commitment to the cause.
在预测专家方面，存在一种略有不同的选择偏差。预测社区与有效利他主义和存在风险担忧（尤其是 AI 风险）有很强的重叠。这并不意味着个别预测者有偏见。但拥有高 \(p_{\text{doom}})\ 可能会使某人更倾向于将预测作为一种活动。因此，整个社区可能偏向于有存在风险担忧的人。
There is a slightly different selection bias at play when it comes to forecasting experts. The forecasting community has a strong overlap with effective altruism and concerns about existential risk, especially AI risk. This doesn't mean that individual forecasters are biased. But having a high \(p_{\text{doom}})\ might make someone more inclined to take up forecasting as an activity. So the community as a whole is likely biased toward people with x-risk worries.
预测者善于根据证据更新信念，但问题在于，与诸如小行星撞击风险不同，在 AI 存在性风险（AI x-risk）方面，几乎没有证据能改变人们的信念，因此我们怀疑预测结果深受人们加入该社区时所持先验的影响。XPT 报告指出：“在 XPT 期间，即使是最活跃的参与者，也很少有人改变想法，尽管有金钱激励促使他们说服他人。”在一项后续研究中，他们发现许多分歧源于超越 AI 的根本世界观差异。
Forecasters are good at updating their beliefs in response to evidence, but the problem is that unlike, say, asteroid impact risk, there is little evidence that can change one’s beliefs one way or another when it comes to AI x-risk, so we suspect that forecasts are strongly influenced by the priors with which people enter the community. The XPT report notes that “Few minds were changed during the XPT, even among the most active participants, and despite monetary incentives for persuading others.” In a follow-up study, they found that many of the disagreements were due to fundamental worldview differences that go beyond AI.
再次强调，我们关于偏见的观点特指 AI 存在性风险。如果有一个选举预测者社区存在系统性偏见（例如偏向现任者），那么在几次选举后，将预测与现实比较就会变得显而易见。但对于 AI 存在性风险，正如我们在上一节所示，技能评估对尾部风险的高估不敏感。
To reemphasize, our points about bias are specific to AI x-risk. If there were a community of election forecasters who were systematically biased (say, toward incumbents), this would become obvious after a few elections when comparing predictions with reality. But with AI x-risk, as we showed in the previous section, skill evaluation is insensitive to overestimation of tail risks.
有趣的是，技能评估对尾部风险的**低估**极为敏感：如果你对实际发生的罕见事件赋予 0 概率，那么在对数评分规则下，你将遭受**无限**惩罚，无论你预测其他事件有多好，都无法弥补。这被视为对数评分的主要优点之一，也是 Metaculus 采用它的原因。
Interestingly, skill evaluation is extremely sensitive to _underestimation_ of tail risks: if you assign a probability of 0 for a rare event that actually ends up occurring, you incur an _infinite_ penalty under the logarithmic scoring rule, from which you can never recover regardless of how well you predicted other events. This is considered one of the main benefits of the logarithmic score and is the reason it is adopted by Metaculus.
现在考虑一个没有精确估计的预测者——而且对于像 AI 存在性风险这样具有众多不确定性维度的事物，肯定没有预测者能有精确估计。鉴于不对称的惩罚，理性的做法是选择其估计范围中较高的一端。⁴
Now consider a forecaster who doesn’t have a precise estimate — and surely no forecaster has a precise estimate for something with so many axes of uncertainty as AI x-risk. Given the asymmetric penalties, the rational thing to do is to go with the higher end of their range of estimates.⁴
无论如何，当预测者的估计高度不确定时，他们实际报告什么并不清楚。也许他们并不响应评分函数的激励。毕竟，长期预测不会很快得到验证。回想一下，在 XPT 的案例中，激励实际上是预测彼此的预测，以绕过长时间跨度的问题。互评评分论文认为，这将激励预测者提交他们真实的、高努力程度的估计，并考虑了对此说法的各种反对意见。他们对方法的辩护基于两个关键假设：通过付出更多努力，预测者可以更接近真实估计；并且他们没有更好的方法来预测其他预测者会做什么。
In any case, it’s not clear what forecasters actually report when their estimates are highly uncertain. Maybe they don’t respond to the incentives of the scoring function. After all, long-term forecasts won’t be resolved anytime soon. And recall that in the case of the XPT, the incentive is actually to predict each others’ forecasts to get around the problem of long time horizons. The reciprocal scoring paper argues that this will incentivize forecasters to submit their true, high-effort estimates, and considers various objections to this claim. Their defense of the method rests on two key assumptions: that by exerting more effort forecasters can get closer to the true estimate, and that they have no better way to predict what other forecasters will do.
如果这些假设不满足会怎样？正如我们在本文中一直论证的，对于 AI 存在性风险（x-risk），我们不应期待证据会改变预测者的先验信念，因此第一个假设是可疑的。而且，既然 XPT 的一轮已经结束，该锦标赛公布的中位数估计就成为一个强大的锚点（博弈论中的“焦点”）。未来，具有互惠评分激励的预测者可能会以现有的中位数预测为起点，只做微小调整以纳入自上次锦标赛以来出现的新信息。随着现有估计成为未来估计的锚点，估计的范围可能会缩小。所有这些绕来绕去无非是说：可供参考的实际证据越少，群体思维的风险就越大。
What if these assumptions are not satisfied? As we have argued throughout this post, with AI x-risk, we shouldn’t expect evidence to change forecasters’ prior beliefs, so the first assumption is dubious. And now that one iteration of the XPT has concluded, the published median estimates from that tournament serve as a powerful anchor (a “focal point” in game theory). It is possible that in the future, forecasters with reciprocal scoring incentives will use existing median forecasts as a starting point, only making minor adjustments to account for new information that has become available since the last tournament. The range of estimates might narrow as existing estimates serve as anchors for future estimates. All that is a roundabout way to say: the less actual evidence there is to draw upon, the more the risk of groupthink.
无论如何，以下是 XPT 对 AI 导致的灭绝风险和亚灭绝灾难性风险的中位数估计：
For what it’s worth, here are the median estimates from the XPT of both extinction risk and sub-extinction catastrophic risks from AI:
重申一下，我们的观点是，我们不应过于认真对待这些数字。它们更多地反映了不同参与者样本对 AI 的担忧程度，而非其他。
To reiterate, our view is that we shouldn’t take any of these numbers too seriously. They are a reflection of how much different samples of participants fret about AI than anything else.
和之前一样，预测专家（超级预测者）和 AI 专家的估计相差一个数量级或更多。如果我们对这些估计有所采信，那也应该是预测专家的估计，而不是 AI 专家的估计。过去研究的一个重要见解是，领域专家的表现不如预测专家，因为预测专家受过整合多样信息和最小化心理偏差的训练。尽管如此，正如我们上面所说，即使是他们的预测也可能被大大高估，我们就是无法确定。
As before, the estimates from forecasting experts (superforecasters) and AI experts differ by an order of magnitude or more. To the extent that we put any stock into these estimates, it should be the forecasting experts’ rather than the AI experts’ estimates. One important insight from past research is that domain experts perform worse than forecasting experts who have training in integrating diverse information and by minimizing psychological biases. Still, as we said above, even their forecasts may be vast overestimates, and we just can’t know for sure.
那么有什么大不了的？政策制定者认为某个时间范围内的风险是 1% 而不是 0.01% 又怎样？无论哪种情况，看起来都相当低！
So what’s the big deal? So what if policymakers believe the risk over a certain timeframe is 1% instead of 0.01%? It seems pretty low in either case!
这取决于他们如何利用这些概率。大多数情况下，这些估计仅仅是一种表明某组专家认为风险显著的方式。如果仅此而已，那也罢了。但考虑到不同人对相同数字的解读千差万别，目前尚不清楚所有这些精细的量化努力是否对这种信号传递目的有帮助。
It depends on what they do with those probabilities. Most often, these estimates are merely a way to signal the fact that some group of experts thinks the risk is significant. If that’s all they are, so be it. But it’s not clear that all this elaborate effort at quantification is even helpful for this signaling purpose, given that different people interpret the same numbers wildly differently.
例如，美国联邦贸易委员会主席莉娜·汗（Lina Khan）表示，她在这个问题上的观点是技术乐观主义的，因为她认为的 _p(doom)_（末日概率）只有 15%，这让专家们感到困惑。（值得一提的是，这个数字比我们愿意称之为技术乐观主义者的数值高出约一千倍。）要在决策中正确地在头脑中处理非常小或非常大的数字，而不是简单地将它们归入“小到无关紧要”之类的类别，需要大量的定量训练。大多数人没有接受过这种训练。
For example, Federal Trade Commission chair Lina Khan said her views on the matter were techno-optimistic since her _p(doom)_ was only 15%, which left experts bewildered. (For what it’s worth, that number is about a thousandfold higher than what we would be comfortable labeling techno-optimist.) It takes a lot of quantitative training to be able to mentally process very small or very large numbers correctly in decision making, and not simply bucket them into categories like “insignificantly small”. Most people are not trained this way.
简而言之，似乎正在发生的是，专家的模糊直觉和恐惧被转化为伪精确的数字，然后又被政策制定者转化回模糊的直觉和恐惧。让我们停止这种量化的把戏吧！人工智能安全中心（Center for AI Safety）关于人工智能风险的声明在这方面直率得令人钦佩（当然，我们强烈不同意其实质内容）。
In short, what seems to be happening is that experts’ vague intuitions and fears are being translated into pseudo-precise numbers, and then translated back into vague intuitions and fears by policymakers. Let’s just cut the charade of quantification! The Center for AI Safety’s Statement on AI Risk was admirably blunt in this regard (of course, we strongly disagree with its substance).
在决策中使用概率的一种有原则的定量方法是通过成本效益分析进行效用最大化。这个想法很简单：如果我们认为一个结果具有主观价值或效用 U（可以是正的或负的），并且它有，比如说，10% 的概率发生，我们可以把它当作确定会发生并且价值为 0.1 * U 来处理。然后，我们可以将每个可选方案的成本和收益相加，并选择成本减收益最大化的那个（即“期望效用”）。
A principled, quantitative way to use probabilities in decision making is utility maximization through cost-benefit analysis. The idea is simple: if we consider an outcome to have a subjective value, or utility, of U (which can be positive or negative), and it has, say, a 10% probability of occurring, we can act as if it is certain to occur and has a value of 0.1 * U. We can then add up the costs and benefits for each option available to us, and choose the one that maximizes costs minus benefits (the “expected utility”).
这就是问题真正变得棘手的地方。首先，有些人可能认为灭绝具有不可估量的大负价值，因为它排除了未来可能出生的所有人类生命（无论是物理的还是模拟的）的存在。逻辑结论是，x-risk（存在风险）应该始终是每个人的首要任务！这让人想起帕斯卡的赌注，即认为信仰上帝是理性的，因为即使上帝存在的概率极小，不信仰的代价也是无限的（永恒的地狱与永恒的幸福相比），因此期望效用也是无限的。幸运的是，政策制定者并不太相信涉及无穷大的决策框架。但这个想法已经牢牢抓住了人工智能安全社区，并驱使一些人坚信人工智能存在风险应该是社会的首要任务。
This is where things get really problematic. First, some people might consider extinction to have an unfathomably large negative value, because it precludes the existence of all the human lives, physical or simulated, that might ever be born in the future. The logical conclusion is that x-risk should be everyone’s top priority all the time! It is reminiscent of Pascal’s wager, the argument that it is rational believe in God because even if there is an infinitesimally small chance that God exists, the cost of non-belief is infinite (an eternity in hell as opposed to eternal happiness), and hence so is the expected utility. Fortunately, policymakers don’t give too much credence to decision making frameworks involving infinities. But the idea has taken a powerful hold of the AI safety community and drives some people’s conviction that AI x-risk should be society’s top priority.
即使我们仅限于灾难性但非存在性的风险，我们谈论的是数十亿生命危在旦夕，因此即使是 1% 风险的预期成本也如此之高，以至于政策影响是剧烈的——政府应该将人工智能存在风险缓解的支出增加几个数量级，并考虑诸如停止人工智能发展等严厉措施。这就是为什么理解这些估计没有任何方法论支持是如此至关重要。基于如此少的证据做出改变世界的政策决定将是非常不明智的。
Even if we limit ourselves to catastrophic but not existential risks, we are talking about billions of lives on the line, so the expected cost of even a 1% risk is so high that the policy implications are drastic — governments should increase spending on AI x-risk mitigation by orders of magnitude and consider draconian measures such as stopping AI development. This is why it is so vital to understand that these estimates are not backed by any methodology. It would be incredibly unwise to make world-changing policy decisions based on so little evidence.
预测在人工智能政策中能发挥作用吗？我们认为可以——只是不要预测存在性风险。预测人工智能里程碑，例如在某些能力基准上的表现或经济影响，更可行也更有意义。如果一位预测者已经展示了预测各种人工智能里程碑何时达成的能力，这确实为我们提供了证据，表明他们未来会做得很好。我们不再谈论独特或罕见的事件。而且，当考虑较低风险的政策干预时——为潜在的经济混乱做准备，而不是阻止杀人机器人——预测无需让每个理性人都满意，这一点不那么关键。
Is there a role for forecasting in AI policy? We think yes — just not forecasting existential risk. Forecasting AI milestones, such as performance on certain capability benchmarks or economic impacts, is more achievable and meaningful. If a forecaster has demonstrated skill in predicting when various AI milestones would be reached, it does give us evidence that they will do well in the future. We are no longer talking about unique or rare events. And when considering lower-stakes policy interventions — preparing for potential economic disruption rather than staving off killer robots — it is less critical that forecasts be justified to the satisfaction of every reasonable person.
预测社区在里程碑预测上投入了大量精力。在 Metaculus 上，问题“2040 年前会出现人机智能对等吗？”基于超过 1,300 名预测者给出的综合预测为 96%。这太惊人了！如果我们同意这一预测，我们会支持这样的立场：管理向 AGI（通用人工智能）的安全过渡应成为全球优先事项。为什么我们不呢？
The forecasting community devotes a lot of energy to milestone forecasting. On Metaculus, the question “Will there be Human-machine intelligence parity before 2040?” has an aggregate prediction of 96% based on over 1,300 forecasters. That’s remarkable! If we agreed with this forecast, we would be in favor of the position that managing the safe transition to AGI should be a global priority. Why don’t we?
答案在细节中。对于像 AGI（通用人工智能）这样模糊的概念，其定义尚无共识。即使我们固定一个定义，确定它是否已经实现也可能很困难或不可能。对于有效的预测，避免模糊结果极其重要。预测社区解决这个问题的方式是用相对狭窄的技能来定义它，例如考试成绩。
The answer is in the fine print. There is no consensus on the definition of a fuzzy concept such as AGI. Even if we fix a definition, determining whether it has been achieved can be hard or impossible. For effective forecasting, it is extremely important to avoid ambiguous outcomes. The way the forecasting community gets around this is by defining it in terms of relatively narrow skills, such as exam performance.
Metaculus 的智能对等问题是根据数学、物理和计算机科学研究生考试的表现来定义的。基于这个定义，我们确实同意 96% 的预测。但我们认为这个定义被稀释得太厉害，以至于对政策没有多大意义。忘掉存在性风险吧——正如我们之前所写，人工智能在考试中的表现构建效度如此之低，以至于它甚至不能让我们预测人工智能是否会取代工人。
The Metaculus intelligence parity question is defined in terms of the performance on graduate exams in math, physics, and computer science. Based on this definition, we do agree with the forecast of 96%. But we think the definition is so watered down that it doesn’t mean much for policy. Forget existential risk — as we’ve written before, AI performance on exams has so little construct validity that it doesn’t even let us predict whether AI will replace workers.
其他基准也好不了多少。简而言之，预测人工智能能力时间线很棘手，因为基准与现实世界影响之间存在巨大差距。幸运的是，反映重要现实世界任务的更好基准正在开发中。除了基准，我们还需要自然主义评估，即使成本更高。一种自然主义评估是衡量人们在人工智能协助下工作方式的不同。直接预测经济、社会或政治影响——例如劳动力市场转型或军方在人工智能相关支出——可能更有用，尽管更难明确界定和衡量。
Other benchmarks aren’t much better. In short, forecasting AI capability timelines is tricky because of the huge gap between benchmarks and real-world implications. Fortunately, better benchmarks reflecting consequential real-world tasks are being developed. In addition to benchmarks, we need naturalistic evaluation, even if it is more costly. One type of naturalistic evaluation is to measure how people perform their jobs differently with AI assistance. Directly forecasting economic, social, or political impacts — such as labor market transformation or AI-related spending by militaries — could be even more useful, although harder to unambiguously define and measure.
避免在政策中误用概率的责任在于政策制定者。我们并非呼吁预测者停止发布预测，以“保护”政策制定者免遭误导。尽管如此，我们认为预测应附带对所用过程和所考虑证据的清晰说明。这将使政策制定者能够就所呈现的论证是否达到他们可接受的阈值做出知情决策。XPT 是透明性的一个良好范例，本文亦是如此（尽管本文并非关于存在性风险）。另一方面，仅仅调查一批研究者并呈现汇总数字是具有误导性的，政策制定者应予以忽视。
The responsibility for avoiding misuses of probability in policy lies with policymakers. We are not calling for forecasters to stop publishing forecasts in order to "protect" policymakers from being misled. That said, we think forecasts should be accompanied by a clear explanation of the process used and evidence considered. This would allow policymakers to make informed decisions about whether the justification presented meets the threshold that they are comfortable with. The XPT is a good example of transparency, as is this paper (though it is not about x-risk). On the other hand, simply surveying a bunch of researchers and presenting aggregate numbers is misinformative and should be ignored by policymakers.
那么，政府应如何应对 AI 存在性风险？我们的观点并非政府应无所作为。但政府应拒绝那些在我们视存在性风险为紧迫且严重时可能显得有说服力的政策，尤其是：限制 AI 发展。正如我们将在本系列后续文章中所论证的，此类政策不仅没有必要，反而可能_增加_存在性风险。相反，政府应采取与一系列可能的 AI 风险估计相兼容、且即便风险可忽略不计时总体上仍有益的政策。幸运的是，此类政策确实存在。政府还应改变政策制定的_过程_，使其对新证据更具响应性。更多内容即将推出。
So what should governments do about AI x-risk? Our view isn't that they should do nothing. But they should reject the kind of policies that might seem compelling if we view x-risk as urgent and serious, notably: restricting AI development. As we'll argue in a future essay in this series, not only are such policies unnecessary, they are likely to _increase_ x-risk. Instead, governments should adopt policies that are compatible with a range of possible estimates of AI risk, and are on balance helpful even if the risk is negligible. Fortunately, such policies exist. Governments should also change policymaking _processes_ so that they are more responsive to new evidence. More on all that soon.
* XPT 报告题为《预测存在性风险：来自长期预测锦标赛的证据》。
* The XPT report is titled Forecasting Existential Risks: Evidence from a Long-Run Forecasting Tournament.
* Tetlock 和 Gardner 的著作《超级预测》总结了 Tetlock、Barbara Mellers 等人的研究。
* Tetlock and Gardner’s book Superforecasting summarizes research by Tetlock, Barbara Mellers, and others.
* Scott Alexander 反驳了原则上对独特事件赋予概率有问题的说法（我们基本同意）。我们的论证与他所针对的立场有两个关键区别。我们不是在谈论一般的预测，而是其在政策制定中的应用。而且我们原则上不反对它。我们的论证是经验性的：AI 存在性风险预测极不可靠且缺乏依据。理论上这未来可能会改变，但我们并不指望。
* Scott Alexander refutes the claim that there is something wrong in principle with ascribing probabilities to unique events (we largely agree). Our argument differs in two key ways from the position he addresses. We aren’t talking about forecasting in general; just its application to policymaking. And we don’t object to it in principle. Our argument is empirical: AI x-risk forecasts are extremely unreliable and lack justification. Theoretically this could change in the future, though we aren’t holding our breath.
* Friedman 和 Zuckhauser 的论文解释了为什么概率不是全部：两个具有相同风险估计的预测可能对政策制定者有不同的影响。我们认为，AI 存在性风险预测在信心的三个维度中的两个上表现不佳：可靠的证据基础和合理的意见范围狭窄。
* A paper by Friedman and Zuckhauser explains why probabilities aren’t the whole story: two forecasts that have the same risk estimate might have very different implications for policymakers. In our view, AI x-risk forecasts fare poorly on two of the three dimensions of confidence: a sound basis in evidence and a narrow range of reasonable opinion.
* 我们普林斯顿 CITP 同事 Shazeda Ahmed 领导的一篇论文解释了 AI 安全的认知文化，它由“连贯、交织的知识生产和社区建设的社会结构”组成。这有助于理解为什么预测等实践已成为 AI 安全社区形成信念的支柱，而不是更广泛的科学社区以同行评审等实践为中心。当然，我们不应该仅仅因为一种观点不符合科学正统就拒绝它。但与此同时，我们也不应该对自称 AI 安全社区对 AI 安全的观点给予任何尊重。重要的是要理解，AI 安全社区的中位数成员对 AI 安全持有一种特定的立场——一种极具争议且在我们看来相当危言耸听的立场。
* A paper by our Princeton CITP colleagues led by Shazeda Ahmed explains the epistemic culture of AI safety, which consists of “cohesive, interwoven social structures of knowledge-production and community-building”. It helps understand why practices such as forecasting have become pillars of how the AI safety community forms its beliefs, as opposed to the broader scientific community that centers practices such as peer review. Of course, we shouldn’t reject a view just because it doesn’t conform to scientific orthodoxy. But at the same time, we shouldn’t give any deference to the self-styled AI safety community’s views on AI safety. It is important to understand that the median member of the AI safety community holds one particular stance on AI safety — a stance that is highly contested and in our view rather alarmist.
* 我们撰写了大量关于基于证据的 AI 安全的文章。我们最著名的工作包括文章《AI 安全不是模型属性》和论文《论开放基础模型的社会影响》，后者是大型合作的结果。
* We have written extensively about evidence-based AI safety. Our best-known work includes the essay "AI safety is not a model property" and the paper titled "On the Societal Impact of Open Foundation Models", which was the result of a large collaboration.
致谢。我们感谢 Benjamin Edelman、Ezra Karger、Matt Salganik 和 Ollie Stephenson 对草稿的反馈。本系列文章基于一篇即将发表的论文，该论文受益于许多人的反馈，包括 Seth Lazar 和澳大利亚国立大学 MINT 实验室的成员、普林斯顿大学“预测的极限”课程的学生、Shazeda Ahmed 和 Zachary Siegel。
Acknowledgements. We are grateful to Benjamin Edelman, Ezra Karger, Matt Salganik, and Ollie Stephenson for feedback on a draft. This series of essays is based on an upcoming paper that benefited from feedback from many people, including Seth Lazar and members of the MINT lab at Australian National University, students in the Limits to Prediction course at Princeton, Shazeda Ahmed, and Zachary Siegel.
具体来说，通过使用核武器试验的数据并做出一些基于物理的假设，我们可以计算出需要多少才能扬起足够的尘埃，使天空变暗很长一段时间，并导致全球农业崩溃。
Specifically, by using data from nuclear weapons tests and making a few physics-informed assumptions, we can calculate what it would take to kick up enough dust to darken the skies for a prolonged period and lead to a collapse of global agriculture.
假设保守预测者 G 的预测有一个下限 \(\epsilon\)（在我们的例子中为 0.01）。只有 \(\epsilon\) 比例的事件中，两个预测者之间的差异才是相关的。即使我们只考虑真实概率小于 \(\epsilon\) 的事件集，一些计算表明，F 的预期对数分数比 G 好一点点——\(O(\epsilon)\)，而使用 Brier 分数时，为 \(O(\epsilon^2)\)。因此，当考虑所有事件时，预期分数的差异分别为 \(O(\epsilon^2)\) 和 \(O(\epsilon^3)\)。同时，对于两种评分函数，他们分数的方差约为 0.1。为了能够自信地断言两个预测者分数的均值不同，我们需要事件数量 N 足够大，使得平均分数差异的标准差（即 \(\sqrt{0.1 / N}\)）远小于预期差异。因此，对于对数分数，我们需要 \(N \sim O(1/\epsilon^4)\)；对于 Brier 分数，我们需要 \(N \sim O(1/\epsilon^6)\)。
Suppose G, the conservative forecaster, has a floor of \(\epsilon\) for their forecasts (in our example, 0.01). There is only an \(\epsilon\) fraction of events where the difference between the two forecasters is even relevant. Even limiting ourselves to the set of events with a true probability less than \(\epsilon\), some calculation shows that F’s expected log score is better than G’s by a tiny amount — \(O(\epsilon)\), and with the Brier score, \(O(\epsilon^2)\). So when looking at all events, the differences in expected scores are \(O(\epsilon^2)\) and \(O(\epsilon^3)\) respectively. Meanwhile, the variance in their scores comes out to about 0.1 for both scoring functions. To be able to confidently assert that the means of the two forecasters’ scores are different, we need the number of events N to be large enough that the standard deviation of the difference in mean scores, which is \(\sqrt{0.1 / N}\), to be much less than the expected difference. So we need \(N \sim O(1/\epsilon^4)\) for the log score and \(N \sim O(1/\epsilon^6)\) for the Brier score.
当然，预测者可能低估风险的原因也存在。我们认为这些原因不太有说服力，但更重要的是，我们认为它们超出了本讨论的范围。原因很简单：倡导政府实施限制自由的政策是由 _高_ 的 x-risk 估计来证明的。因此，举证责任是不对称的。那些呼吁此类政策的人必须证明他们的概率估计，包括回应关于向上偏差的担忧。如果在某个奇怪的未来世界中，有人基于 _低_ 的 x-risk 估计倡导限制性政策，那么对潜在的 _低估_ 进行同样的审查将变得很重要。
Of course, there are also reasons why forecasters might underestimate the risk. We find those reasons less persuasive, but more importantly, we consider them out of scope for this discussion. The reason is simple: the advocacy for governments to implement freedom-restricting policies is being justified by _high_ x-risk estimates. The burden of evidence is thus asymmetric. Those who call for such policies must justify their probability estimates, including responding to concerns about upward biases. If, in some strange future world, some people advocate for restrictive policies based on _low_ x-risk estimates, it will become important to subject potential _underestimation_ to the same scrutiny.
在预测理论中，对数评分以及其他所谓的恰当评分规则（proper scoring rules）在数学上能够保证引出预测者的“真实信念”。但恰当评分规则的定义假设真实信念是一个单一值，而它往往是一个较宽的范围。在这种情况下，预测者会被激励去报告其分布的均值，但我们直觉上所说的“真实信念”或许更对应于中位数。
In the theory of forecasting, the logarithmic score and other so-called proper scoring rules are mathematically guaranteed to elicit the forecaster's "true belief". But the definition of a proper scoring rule assumes that the true belief is a single value, whereas it tends to be a wide range. In such a scenario, the forecaster is incentivized to report the mean of their distribution, but what we intuitively mean by the "true belief" might better correspond to the median.