AI 安全运动：大帐篷还是小帐篷？

安全辩论背后未言明的分歧 The unstated disagreement that underpins safety debates

最近几周，关于 AI 安全出现了两种叙事：要么 AI 存在性风险是真实且迫在眉睫的，要么 AI 领导者和吹哨人对此的声称是言不由衷的——是一种“心理战”或炒作，或是一种扭曲的监管俘获。

In recent weeks, two narratives about AI safety have emerged: either AI existential risk is real and imminent, or AI leaders’ and whistleblowers’ claims to that effect are insincere — a “psyop” or hype or a twisted form of regulatory capture.

很少有人考虑第三种可能性：存在性风险警告是真诚的，但完全错误，且_对 AI 安全适得其反_。仿佛每个人都理所当然地认为，发出警告的人是天才，唯一的问题是他们是善良的天才还是邪恶的天才。

Few have considered the third possibility that x-risk warnings are sincere but simply wrong and _counterproductive to AI safety_. It’s as if everyone takes for granted that those raising the alarm are geniuses, and the only question is whether they’re the benevolent or the evil kind of genius.

我们不认同这两种叙事。在我们关于 AI 安全的写作中，我们一直试图认真对待这些论点，但反驳许多我们认为不合理的说法，同时强调我们认为真实的安全关切。本文是这一议程的延续。

We subscribe to neither narrative. In our writing on AI safety, we’ve consistently tried to take the arguments seriously but rebut the many claims we think are unsound while highlighting safety concerns that we think are real. This essay is a continuation of that agenda.

我们确实系统性地对灾难性和系统性风险的韧性投入不足 It is true that we systematically underinvest in resilience against catastrophic and systemic risks

让我们先以最有力的方式（steelmanning）阐述人工智能安全与有效利他主义（EA）的部分论点。在防御灾难性风险的投资方面，世界确实存在严重问题，安全与 EA 社区在这一点上是正确的。即使完全将超级智能排除在外，多年来基于证据的风险论证依然成立，却未产生相应的政策回应。

Let's start by steelmanning a subset of AI safety and Effective Altruist (EA) arguments. Something is deeply wrong in the world when it comes to how much we invest in defending against catastrophic risks, and the safety and EA communities are right about that. Take superintelligence out of the equation entirely and it remains true that evidence-grounded risk arguments have been made for years and not produced a commensurate policy response.

最著名的例子是大流行病防范的失败。我们多年来一直收到警告和定量估计。世界卫生组织/世界银行在 COVID 之前召集的一份 2019 年报告开篇即指出，存在“非常真实的威胁……一种呼吸道病原体可能导致 5000 万至 8000 万人死亡，并摧毁全球近 5%的经济。”该报告估计，对大多数国家而言，充分防范的成本仅为每人每年 1–2 美元。

The most well known example is the failure to prepare for pandemics. We had years of warnings and quantitative estimates. A 2019 report convened by the WHO/World Bank — before COVID — began by saying there is a "very real threat ... of a respiratory pathogen killing 50 to 80 million people and wiping out nearly 5% of the world's economy." It estimated the cost of adequate preparedness at a mere $1–2 per person per year for most countries.

自 2009 年 H1N1 大流行以来，至少有 11 个高级别小组和委员会提出了改善全球大流行病防范的具体建议，然而“大多数建议从未得到实施”。更值得注意的是，即使在 COVID 之后，也只有一小部分需求得到了资助。同一个独立小组在 2024 年写道：“在危机期间提供大量即时资源，而不是主动预防危机，这种做法的吸引力是短视的。然而历史表明，这不会有好结果。”根本原因似乎是选民奖励用于救济的支出，而非防范。

Since the 2009 H1N1 pandemic, at least 11 high-level panels and commissions have made specific recommendations to improve global pandemic preparedness, and yet "the majority of recommendations were never implemented". More remarkably, even after COVID, only a fraction of the need was funded. The same independent panel wrote in 2024 that "There is a cynical attraction to providing significant and immediate resources during a crisis, rather than proactively preventing one. Yet history shows this will not end well." The root cause seems to be that voters reward spending on relief, not preparedness.

需要明确的是，大流行病防范不足绝非有效利他主义独有的洞见，更不用说人工智能安全了。但 EA 确实值得称赞，因为它倡导了这一事业。同样真实的是，EA 以期望效用计算为核心的结果主义道德框架，在揭示和量化我们防范不足方面非常有效。该框架极具争议，但我们认为它本身或其应用于生物风险等领域并无问题。然而，我们强烈反对将其扩展到长期主义和人工智能存在性风险。我们也不同意将诸如人工智能放大生物风险等问题重新定义为“人工智能安全”问题，并通过对齐等方法来解决的倾向。我们的观点是，人工智能是现有系统性风险的放大器。

To be clear, the inadequacy of pandemic preparedness is by no means a uniquely Effective Altruist insight, let alone AI safety. But EA does deserve credit for championing the cause. It's also true that EA's consequentialist moral framework that centers expected-utility calculations is highly effective at revealing and quantifying our underpreparedness. The framework is highly controversial, but we don't think there's anything wrong with it or its application to areas like biorisk. We do, however, strongly disagree with its extensions to longtermism and AI existential risk. We also disagree with the tendency to reframe problems such as AI's amplification of biorisk as "AI safety" problems to be solved through methods such as alignment. Our view is that AI is an amplifier of existing systemic risks.

防范不足的问题在网络安全领域同样存在——实际上更糟。与大流行病或金融不同，网络安全社区没有对系统性或灾难性风险进行建模的文化。我们在上一篇文章中描述的经济方法成功地使日常攻击的预期损失保持在可控水平，但市场在合理定价尾部风险和外部性方面是出了名的糟糕。

The underpreparedness problem is also true of cybersecurity — actually, it's worse. Unlike pandemics or finance, there is no culture of modeling systemic or catastrophic risks in the cybersecurity community. The economic approach we described in the previous essay has been successful in keeping the expected losses from everyday attacks at a manageable level, but markets are notoriously bad at appropriately pricing tail risks and externalities.

保险公司深知这一点。劳合社（Lloyd’s）直言不讳地指出：“损失有可能远远超出保险市场所能承受的范围”，并且不为严重的国家支持的网络安全攻击提供保险（这是 2022 年的情况）。

Insurance companies know this. Lloyd’s is pretty blunt about the fact that “losses have the potential to greatly exceed what the insurance market is able to absorb”, and does not insure against severe state-backed cyberattacks (that was in 2022).

再一次，这与 AI 近期的能力关系不大。五年前，在 AI 尚未出现的时候，Arvind 在普林斯顿大学讲授本科信息安全课程时就强调，级联故障和灾难性风险是可能的，可能导致整个互联网在较长时期内瘫痪，并带来可能改变文明的后果。¹

Once again, this has little to do with AI’s recent capabilities. Five years ago, before AI was in the picture, when teaching undergraduate information security at Princeton, Arvind emphasized that cascading failures and catastrophic risks are possible, resulting in essentially the whole internet going down for an extended period, with potentially civilization-altering consequences.¹

_（回顾我幻灯片中的演讲者备注，我想起我曾为在一门本应讲授基础知识的本科课程中展示原创研究而道歉，但解释说，这是因为安全社区对这一关键话题几乎完全缺乏关注，让我别无选择。——AN。）_

_(Looking back at the speaker notes in my slides, I’m reminded that I apologized for presenting original research in an undergrad class that’s supposed to be about basics, but explained that this is because the near-total lack of attention to this critical topic in the security community left me no choice. — AN.)_

换句话说，Dario Amodei 关于智能体集群可能“接管整个互联网”的说法，虽然措辞有些夸张，但显然并非不可能，至少值得研究。网络安全社区拥有合适的技能来做到这一点，但不幸的是，他们似乎认为这甚至不值得思考。

In other words, Dario Amodei’s claim about agent swarms potentially being capable of “taking over the entire internet”, while worded somewhat hyperbolically, is not obviously outside the realm of possibility, and deserves study at the very least. The cybersecurity community has the right skillset to do that, but unfortunately they don’t seem to find this even worth thinking about.

还有一些系统性风险是分散且渐进的，而非灾难性的。想想聊天机器人对新闻流量的影响——点击率远低于传统搜索，加速了传统新闻业的衰落。这种转变的二阶社会效应在相当长一段时间内不会显现。而这只是数十种悄然发生的制度衰败之一。Atoosa Kasirzadeh 引入了累积风险的概念来描述这种社会韧性的逐渐侵蚀。² 在这里，不作为的心理学与流行病的情况相反：流行病如此罕见，以至于我们能够将事件遗忘，而衰败如此普遍和渐进，以至于我们变得习以为常。

There are also systemic risks that are diffuse and gradual rather than catastrophic. Consider what chatbots are doing to news traffic — clickthrough rates are far lower than traditional search, accelerating the decline of traditional journalism. The second-order societal effects of this shift won’t be apparent for quite a while. And that’s just one of dozens of types of creeping institutional decay. Atoosa Kasirzadeh introduced the concept of accumulative risk to describe this gradual erosion of societal resilience.² Here, the psychology of inaction is the reverse of the case of pandemics: rather than being so rare that we are able to memory-hole the event, decay is so pervasive and gradual that we become inured to it.

简而言之，完全基于有充分依据而非推测性的因果路径，我们有理由相信，政策制定者在保护社会免受 AI 风险方面投入严重不足；人们也很难不对那些想要加大言辞力度的人产生共情。

In short, based entirely on well-grounded rather than speculative causal pathways, it is reasonable to believe that policymakers are drastically underinvesting in protecting society against AI risks, and it is hard not to empathize with those who want to ratchet up the rhetoric.

当然，问题在于更紧迫的公开警告是否真的能带来更好的政策制定。

Of course, the question is whether more urgent public warnings actually lead to better policymaking.

Coxon 事件表明扭曲的媒体环境如何将存在性风险置于更切实的关切之上 The Coxon incident shows how the warped media environment elevates x-risk over more grounded concerns

本通讯主要是一次换位思考的练习。让我们花点时间阐明为什么 AI 泡沫内外的人往往各说各话。

This newsletter is largely an exercise in perspective-taking. Let’s take a moment to articulate the primary reason why people inside and outside the AI bubble tend to talk past each other.

对于 AI 领域的人——无论他们是否关心安全——都很难理解现在世界上还有什么其他事情重要。正如计算机科学家 Scott Aaronson 在其文章《奇迹与恐怖的时代》中所说：

For people in AI — whether or not they’re concerned about safety — it’s hard to understand how anything else in the world matters right now. As computer scientist Scott Aaronson put it in his essay The Age of Wonders and Terrors:

这篇文章在社区中广为流传，因为这种观点被广泛认同。即使是我们两人——以反对这一观点的某些要素而闻名——也认为 AI 足够重要，以至于我们很久以前就决定将职业生涯奉献给它。“AI 作为常规技术”的立场是，AI 将“仅仅”像工业革命一样具有影响力。

This essay made the rounds in the community because this perspective is widely shared. Even the two of us — best known for pushing back against some elements of this view — consider AI important enough that we decided long ago to devote our careers to it. The AI as Normal Technology position is that AI will “only” be as impactful as the industrial revolution.

AI 领域之外的人不理解 AI 社区中这种观点的强烈程度——即这是一个 AI 的世界，而我们只是生活在其中。

People outside AI don’t understand the intensity of this view in the AI community — that it’s an AI world and we’re all just living in it.

但另一方面——AI 领域的人大大高估了公众对 AI 的关心程度。在盖洛普民意调查中，将 AI（或“计算机/技术的进步”）评为最重要问题的人比例约为 0.5%。其中大多数人可能想到的是相对平凡的方面，比如在他们社区建设数据中心。在 AI 领域之外，将其视为我们时代决定性议题的人比例小得难以理解。

But there’s a flip side — people in AI vastly overestimate how much the public cares about AI. In Gallup polls, the share of people who rate AI (or “advancement of computers/technology”) as the most important issue is about half a percent. Most of _those_ people are likely thinking of relatively mundane aspects such as a datacenter being built in their community. The share of people outside AI who treat it as the defining issue of our time is incomprehensibly small.

AI 人士未能认识到这一点的原因在于，AI 是一个**显著度**与**关切度**相互脱钩的话题。人们对 AI 有很高的关切（也就是说，许多人在**被特别问及**时认为 AI 很重要或表达担忧），但显著度却非常低（也就是说，很少有人将其列为重要议题）。

The reason AI people fail to recognize this is that AI is a topic where _salience_ and _concern_ are decoupled from each other. There is high concern about AI (that is, many people think AI is important or express concern when _specifically asked about it_) but very low salience (that is, few people rank it as an important issue versus others).

无论如何，所有这些都只是开场白，用以解释为什么尽管有许多灾难性和系统性的 AI 放大风险——我们一致认为世界迫切需要采取行动——它们却往往无法进入公众意识。

Anyway, all this is mere throat-clearing to explain why, even though there are many catastrophic and systemic AI-amplified risks that we agree the world urgently needs to act on, they don’t tend to break through to public consciousness.

夏季智能体集群黑客事件曝光后，出现了一些零星的头条新闻，但其媒体关注度和公众认知度比 Coxon 的辞职和存在性风险预测后来引发的关注低了几个数量级。

After the agent swarm hacks over the summer came to light, there was a smattering of headlines, but it was orders of magnitude below the level of media attention and public awareness that Coxon’s resignation and x-risk predictions would later draw.

安全社区对这些担忧未能进入公众视野表示惊讶和沮丧。这让整个社区感到不安。作为社区的观察者，我们感到社区中的焦虑程度与过去任何时刻都有质的不同。

The safety community expressed amazement and frustration that these concerns hadn’t broken through to the public. This put the whole community on edge. As observers of the community, we felt the level of anxiety in the community to be qualitatively different from any other moment in the past.

不难解释为什么 AI 网络风险没有获得更多头条新闻：AI 是一个低显著度的话题，黑客事件每天都在新闻中，而这次事件甚至没有对普通人造成任何已知伤害。普通人为何应该关心这一点远非显而易见。

It is not hard to explain why AI cyberrisk didn’t get more headlines: AI is a low-salience topic, hacks are in the news every day, and this one didn’t even result in any known harm to ordinary people. It is far from obvious why everyday people should care.

相比之下，存在性风险更具冲击力，而 Coxon 的辞职让 AI 安全议题终于取得突破。需要明确的是，此前已多次发出存在性风险（x-risk）警告，但这次有两点不同。第一，有一个一触即发的火药桶，即安全社群渴望与媒体对话，并一旦出现有新闻价值的事件作为由头，便放大相关担忧。这种回应如此声势浩大且口径一致，以至于让人们断定这必定是一场协同行动，但我们认为事实并非如此。

In contrast, existential risk is visceral, and the Coxon resignation made AI safety finally break through. To be clear, x-risk warnings have been made many times before, but there were two differences this time. First, there was a tinderbox waiting to be lit, namely the safety community's eagerness to talk to the media and amplify concerns once there was a hook in the form of a newsworthy event. This response was so full-throated and unanimous that it left people concluding that it must have been a co-ordinated effort, but we don't think that was the case.

这次有所不同的另一个原因在于：有披露显示，许多高管持有很高的“p（doom）”（末日概率），却仍继续沿着竞相构建超级智能的道路前行——这一选择在公众看来既鲁莽又不可理喻。当然，AI 领导者会援引一套复杂的合理化说辞，来解释他们为何认为自己别无选择，但真正传到公众耳中的并非这些。或许更大的脱节在于，这一切都建立在一套公众并不认同的功利主义价值体系之上。

There's another reason this time was different: the revelation that many executives have a high "p(doom)" and yet continue down the path of racing to build superintelligence, a choice that feels reckless and incomprehensible to the public. Of course, there is a complex set of rationalizations that AI leaders invoke to explain why they think they have no other choice, but that's not what reaches people. Perhaps the bigger disconnect is that it is all underpinned by a utilitarian value system that the public doesn't share.

订阅以获取基于证据的 AI 安全与影响分析。

Subscribe for evidence-based analysis of AI safety and impacts.

为何存在性风险框架可能对 AI 安全政策适得其反 Why the x-risk framing may be counterproductive for AI safety policy

回顾迄今为止的论证，关于 AI 安全意味着什么，存在两类信念。像任何二元分类一样，这种划分忽略了一些细微差别，但我们认为它足够准确，因而有用。

To recap the argument so far, there are two clusters of beliefs about what AI safety means. Like any binary, this one leaves out some nuances, but we think it is accurate enough to be useful.

1. 存在**具体的**（可能具有灾难性但尚未达到存在性级别）风险，这些风险在 AI 出现之前就已存在，但被 AI 放大，我们迫切需要**通过预防和韧性**来防御。

1. There are _specific_ (potentially catastrophic but sub-existential) risks, predating AI but amplified by it, that we urgently need to defend against through prevention and resilience.

2. AI 发展本身就是一个存在性风险——要么是因为一旦超级智能 AI 被建造出来，就有太多下游风险需要防御，要么是因为未知的未知——我们要么需要弄清楚如何使超级智能安全（例如通过**对齐**），要么阻止其被建造。

2. AI development itself is an existential risk — either because there are too many downstream risks to defend against once superintelligent AI is built, or because of unknown unknowns — and we either need to figure out how to make superintelligence safe (such as through alignment) or stop it from being built.

关键的是，由于信息环境，**存在性风险框架**是驱动公众担忧和许多政策讨论的框架，尽管**可信的因果路径尚未被阐明**（而且尝试阐明它们的努力至少可以说是缺乏说服力的）。

Critically, due to the information environment, the x-risk framing is the one that is driving public concern and much policy discourse, despite the fact that credible causal pathways haven’t been articulated (and attempts to do so are unconvincing to say the least).

那么接下来会发生什么？一种乐观的可能性是，当前时刻的部分能量被引导到构建广泛的防御体系上（或许伴随着 AI 发展的放缓，尽管在我们看来那要次要得多）。

So what’s going to happen next? One happy possibility is that some of the energy of the current moment gets channeled into building a broad set of defenses (perhaps accompanied by a slowdown in AI development, though that is much less important in our view).

但这条路径在我们看来不太可能。更可能的结果是，存在性风险框架的主导地位实际上适得其反，主要有两个原因。3

But this path seems unlikely to us. The more likely outcome is that the dominance of the x-risk framing is actively counterproductive, for two main reasons.3

**极化与党派性。** 末日框架一直具有极化性，因为我们许多人仍然完全不相信它，但现在极化开始沿着党派界线分裂。党派性对安全政策不利，因为它导致不作为、高度不确定性和党派政策逆转。更糟的是，当人们意识到“10%灭绝风险”预测背后没有定量模型时，他们会感到被误导，从而对安全政策产生更多抵制，尤其是在一个已经对专家胡编乱造保持警惕的极化环境中。

Polarization and partisanship. The doom framing has always been polarizing because many of us remain completely unconvinced by it, but now the polarization is starting to split along partisan lines. Partisanship is bad for safety policy because it leads to inaction, high uncertainty, and partisan policy reversals. Worse, when people realize that there is no quantitative model behind the “10% extinction risk” predictions, they will feel misled, resulting in even more pushback to safety policies, especially in a polarized environment where one side is already wary of experts making stuff up.

此外，引起公众恐慌是适得其反的，因为它将解决方案空间限制在最直截了当且提供情感慰藉的方案上，而这些方案往往不是最有效的。

Further, causing the public to panic is counterproductive because it constrains the solution space toward those that are the most blunt and offer an emotional salve, which don’t tend to be the most effective.

**误导。** 我们已经驳斥了针对存在性风险担忧的分心论点——如果我们担心失业和安全，我们可以同时解决两者，它们并不会互相分散注意力。但这不是关于分心：两种安全观代表了对_同一问题_的两种不同诊断。它们似乎可能会争夺资源。我们称之为误导，以区别于分心。

Misdirection. We have repudiated the distraction argument against x-risk concerns — if we’re worried about job loss and safety, we can address both, and they don’t distract from each other. But this isn’t about distraction: the two views of safety represent two different diagnoses of the _same problem_. It seems likely that they will compete for resources. Let’s call it misdirection, to distinguish it from distraction.

特别是，广义的存在性威胁只允许一种政策——禁令。以概率方式表述威胁使我们更加偏向那个方向。“如果我们开发超级智能，就有 10%的灭绝风险”这句话让人觉得干预的唯一杠杆就是停止开发。不幸的是，对超级智能的禁令在定义上模糊到近乎不连贯。4 而且 AI 安全仍然不是模型属性。其他潜在的干预措施，如暂停数据中心建设，作为安全措施极其无效。

In particular, a generalized existential threat admits exactly one policy — a ban. Framing the threat in terms of probabilities biases us even more in that direction. The statement “if we develop superintelligence there’s a 10% risk of extinction” makes it look like the only lever for intervention is stopping development. Unfortunately, a ban on superintelligence is definitionally ambiguous to the point of being incoherent.4 And AI safety is still not a model property. Other potential interventions such as pausing datacenter construction are exceedingly impotent as safety measures.

最重要的是，禁止开发对于**现有** AI 系统所带来的灾难性和系统性风险毫无作用。我们应对这些风险所需的对策远没有那么吸引眼球，这反过来意味着，为解决这些风险而建立的**运动**，与针对超级智能的运动截然不同：需要一百项各自不同的干预措施来加固防御与韧性，而不是将精力和注意力集中于竖起一堵巨大而炫目的墙。这正是我们担心 AI 安全的这两种框架相互竞争的原因。我们希望自己判断错了！

Most importantly, bans on development do nothing about the catastrophic and systemic risks from _existing_ AI systems. The response we need to those risks is much less sexy, which in turn means that the _movement_ one would build to address them is quite different from one targeted at superintelligence: a hundred disparate interventions to shore up defenses and resilience rather than concentrating effort and attention toward erecting one mammoth, flashy wall. This is why we worry that the two framings of AI safety are in competition. We hope we are wrong about this!

结论：大帐篷还是小帐篷的人工智能安全运动？ Conclusion: a big tent or small tent AI safety movement?

我们所说的“大帐篷安全”有三个主要特征。

There are three main characteristics of what we call big-tent safety.

* **信念。** 大帐篷运动将包括不关心存在性风险（x-risk）的成员（并且像我们一样，可能不认为超级智能是一种威胁）。他们可能关心网络风险、生物风险和其他特定的灾难性风险、各种累积性风险，以及构建社会韧性的必要性。

* **Beliefs.** A big-tent movement would include members who are not concerned about x-risk (and, like us, may not consider superintelligence a threat). They may care about cyberrisk, biorisk and other specific catastrophic risks, various accumulative risks, and the need to build societal resilience.

* **价值观和运动建设。** 社会运动必须在价值观上而不仅仅是信念上具有多元性，认识到 EA/安全主义道德框架在公众中不受欢迎。公众吸引力不应基于恐慌，而应阐述积极的愿景。

* **Values and movement building.** The social movement must be pluralist in terms of values and not just beliefs, recognizing that EA / safetyist moral frameworks are unpopular with the public. The public appeal should be based not on panic but articulating positive visions.

* **政策。** 大帐篷运动将认识到，尽管信念存在分歧，但在政策上有很多共同点（不过，如果你狭隘地关注存在性风险，这些政策可以说是不够的）。两类这样的政策已经变得清晰：透明度和责任。嵌入式评估是紧迫的，构建和使用 AI 的组织应系统地对它们创造的负外部性负责。大帐篷安全政策侧重于摘取这样的低垂果实，同时为更强大的干预措施（如禁令）积蓄力量（例如禁止完全自主的递归自我改进）。但后者应该涉及做艰苦的工作，说服怀疑者相信其必要性，而不是追求恐慌政策。

* **Policy.** A big-tent movement would recognize that there is much common ground on policy despite divergence in beliefs (though these policies arguably fall short if you're narrowly focused on x-risk). Two such clusters of policies have become clear: transparency and liability. Embedded evaluation is urgent, and organizations that build and use AI should be systematically accountable for the negative externalities they create. Big-tent safety policy focuses on picking such low-hanging fruit while building muscle for stronger interventions like bans (such as banning fully autonomous recursive self-improvement). But the latter should involve doing the hard work of convincing skeptics of their necessity rather than pursuing policy-by-panic.

相比之下，小帐篷安全在信念、价值观和政策目标层面上更为狭窄。它重视意识形态的同质性（同时声称相反）。这有可能导致纯洁性螺旋，逐渐驱逐大帐篷安全人士。小帐篷安全不优先考虑用于防御“仅仅是”灾难性风险（！）的支出。最后，这样的运动没有认识到自身与公众在价值观和 AI 安全显著性方面的差距。结果，它大大高估了自己的能力，有可能引发反弹。

In contrast, small-tent safety is narrower at the level of beliefs, values, and policy goals. It prizes ideological homogeneity (while claiming the contrary). This risks a purity spiral, gradually driving out big-tent safety people. Small-tent safety de-prioritizes spending on defenses against risks that are “only” catastrophic (!). Finally, such a movement doesn't recognize the gap between itself and the public in terms of both values and the salience of AI safety. As a result, it vastly overplays its hand, risking backlash.

就个人而言，我们常常纳闷，为什么我们的工作有时受到 AI 安全社区的欢迎，有时却遭到攻击，我们认为“大帐篷”与“小帐篷”的区分很好地解释了这一点。

On a personal note, we’ve often wondered why our work has been sometimes welcomed and sometimes attacked by the AI safety community, and we think the big-tent versus small-tent distinction captures it well.

_我们感谢 Abi Olvera、Joshua Saxe 和 Rohit Krishnan 对草稿的反馈。_

_We are grateful to Abi Olvera, Joshua Saxe, and Rohit Krishnan for feedback on a draft._

为什么网络安全社区对灾难性风险的讨论如此反感？我们有几个猜测。

Why is the cybersecurity community allergic to discussion of catastrophic risk? We have a few guesses.

历史上，“网络珍珠港”式的警告已被发出过太多次，而且通常没有成效（甚至分散了对低严重性/高频率风险的注意力）。

Historically, “Cyber Pearl Harbor” warnings have been issued one too many times and were generally unproductive (and even a distraction from low-severity / high-frequency risks).

我们从未经历过真正的灾难性网络事件——在 2001-2004 年蠕虫病毒的全盛时期，互联网规模要小得多，经济对它的依赖也小得多。因此，鉴于我们对灾难的实际样貌和恢复路径没有任何数据点，为灾难性网络风险做准备就更加困难。

We’ve never had an actual catastrophic cyber incident — during the heyday of worms in 2001-2004, the internet was much smaller and the economy depended much less on it. So it is harder to prepare for catastrophic cyber-risk given that we have zero data points on what a catastrophe actually looks like and what the recovery path would be.

计算机网络社区确实有度量集中度的文化，而这项工作有助于解释为何灾难性攻击是可能的——仅击垮 Cloudflare 一家，就能扰乱数万个其他网站和服务。但有两个重要告诫。第一，这项工作止步于此——它并未延伸到级联失效、韧性/恢复等方面。第二，仅度量集中度本身并不具有很强的可操作性。集中化有其正当理由，包括它**改善**了平均情况下的安全性。我们不认为集中化趋势是可逆的。事实上，当今 AI 带来的额外风险很可能会导致集中度进一步上升。

The computer networking community does have a culture of measuring concentration, and this work helps explain why catastrophic attacks are possible — bringing down Cloudflare alone can disrupt tens of thousands of other websites and services. But there are two major caveats. First, the work stops there — it doesn't extend to cascading failure, resilience / recovery, etc. Second, measuring concentration is not very actionable on its own. There are legitimate reasons for concentration, including that it _improves_ average-case security. We don't see the trend toward concentration as reversible. In fact, the additional risk that today's AI poses will likely lead to an increase in concentration.

她讨论了累积性**存在性**风险，以区别于决定性存在性风险；但即便我们不认为它属于存在性风险，累积性与决定性之间的区分也极有帮助。

She discusses accumulative _existential_ risk, in contrast to decisive existential risk, but the accumulative-versus-decisive distinction is extremely helpful even if we don't think it's existential.

这些是修辞本身的弊端。如果 x-risk 推动者真的成功推行了一项防扩散议程，那么对安全与增长两方面的后果都将糟糕得无法估量，正如我们此前所写。幸运的是，开放权重模型或许已经木已成舟。

These are downsides of the rhetoric itself. If x-riskers actually succeed in enacting a nonproliferation agenda, the consequences both for safety and growth will be incalculably bad, as we've written about. Fortunately open-weight models may have left the barn.

禁止在无人类监督下的递归自我改进似乎更具可操作性，而且即便人们不认为 RSI 会导致超级智能，这一禁令也可能是合理的。

A ban on recursive self improvement without human oversight seems more tractable, and could make sense even if one doesn't think RSI will lead to superintelligence.