The Preference Cascade Is Only Getting Started
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→本文认为，围绕 AI 生存风险的偏好级联正在形成，其标志包括顶尖实验室高调的人事离职、调查中对灾难性风险估计的上升，以及主流评论的激增。文章主张，包括那些由具有安全意识的领导者掌舵的前沿实验室在内，都在安全方面投入不足，甚至相对于其狭隘的商业利益也是如此；而仅靠放慢前沿发展，只会推迟灾难，而非阻止灾难。作者支持诸如事件披露、第三方监督和终止开关等渐进式措施，但警告说，这些只是我们所能做的最低限度，考虑到欺骗性对齐的可能性，它们可能并不足够。结论是，若没有一种可行的机制来引导超级智能系统，人类将面临一条狭窄而不确定的道路，而有效行动的时间窗口可能正在关闭。
This essay argues that a preference cascade around AI existential risk is now underway, marked by high-profile departures from leading labs, rising survey estimates of catastrophic risk, and a surge of mainstream commentary. It contends that frontier labs, including those with safety-conscious leaders, are underinvesting in safety relative to even their narrow commercial interests, and that pacing the frontier alone will only delay catastrophe rather than prevent it. The author endorses incremental measures such as incident disclosure, third-party oversight, and kill switches, but warns that these are merely the least we can do and may not suffice given the possibility of deceptive alignment. The conclusion is that without a viable mechanism to steer superintelligent systems, humanity faces a narrow and uncertain path, and the window for effective action may be closing.
7. Bilal Chughtai 从 DeepMind 离职并发出警告。
7. Bilal Chughtai Quits DeepMind and Sounds the Alarm.
10. OpenAI 的 Dan Selsam 发出更响亮的警告。
10. OpenAI’s Dan Selsam Sounds A Louder Alarm.
11. 有些人的担忧达到了你从未想象过的元层面。
11. Some People Worry On Meta Levels You Never Imagined.
13. 双塔与窄路。
13. The Two Towers and The Narrow Path.
14. 一个关于 AI 杀死所有人的具体而详尽的故事，在我看来并不像科幻小说。
14. A Specific, Detailed Story About AI Killing Everyone That Doesn’t Sound To Me Like Science Fiction.
AI Impacts 的调查结果已经出炉。即便早在 2024 年 12 月，存在性风险的估计值就在不断攀升，而 10% 是中位数：
The AI Impacts survey is in. Even back in December 2024, existential risk estimates were creeping upwards, and 10% was the median:
换算成百分比，这意味着 AI 毁灭人类的平均概率约为 30%–33%；Andrew Curran 估计这比此前的结果上升了约 15%，中位数预期约为 10%，且党派分歧很小。
Translated to percentages, this implies a mean chance of AI destroying humanity of around 30%-33%, which Andrew Curran estimates is up ~15% from previous results, with a median expectation on the order of 10%, with only a small partisan split.
它还包括皇家学会的数学家们。
It also includes the mathematicians of the Royal Society.
在安全上投入的正确金额很少是零。就 AI 而言，我再次断言，所有公司都在安全方面投入不足，甚至包括那些平淡无奇的安全问题，以及存在性安全和可扩展对齐工作，相对于他们狭隘短视的商业利益而言。Sam Altman 最近的声明暗示他现在明白了这一点。
The correct amount to invest in safety is rarely zero. In the case of AI, again, I assert that all the companies are under-investing in safety, including even prosaic safety but also existential safety and scalable alignment work, versus even their narrow myopic commercial interests. Sam Altman's recent statements imply he now understands this.
Matthew Yglesias 最近一直在挺身而出。他对近期事件的分析分为四个部分：为什么 Coxon 的辞职引起了轰动（他的解释与我的相似，我们早有准备，而且辞职对普通人来说是可以理解的），为什么大多数担忧者没有离开实验室，他认为有安全担忧的 AI 专业人士应该做什么，并提出了他偏好的前进“操作顺序”，同时不涉及具体对象层面。
Matthew Yglesias has been stepping up to the plate recently. He offers an analysis of recent events in four parts: Why Coxon's resignation broke through (his explanation is similar to mine, we were primed and quitting is understandable to normies), why most of the worried don't quit the labs, what he thinks AI professionals with safety concerns should do and lays out his preferred 'order of operations' going forward, while not getting into the object level.
他建议的操作顺序基本上是：
He suggests this order of operations, basically:
1. 对透明度和模型评估等事项采取轻触式规则。
1. Light touch rules on things like transparency and model evaluation.
2. 作为交换，实施严格且强制执行的出口管制，就像 Dario 呼吁的那样。
2. In exchange, tough and enforced export controls, a la Dario's call.
3. 升级到中等干预程度的规则，这类规则影响更大，但成本也更高。
3. Move up to moderate-touch rules that have more impact at higher cost.
4. 利用这一代价高昂的信号和实力地位，与中国进行谈判。
4. Use this costly signal and position of strength to negotiate with China.
5. 借助这项协议，推进到一个雄心勃勃的终局框架。
5. Using the deal, move up to an ambitious end-stage framework.
理论上我喜欢这个想法。我确实担心我们是否有那么多时间。“等待出口管制产生更大效果”这一想法意味着，现在什么才算作“长时间线”。
I like that in theory. I do worry about whether we have that kind of time. The idea of 'wait for export controls to bite harder' implies what now count as 'long timelines.'
有优秀的作者在跟进此事，解释为什么你应该关注对象层面的问题，这是好事：
It is good to have good writers on the case explaining why you should focus on the object-level questions:
Steven Adler 利用这一时机在《纽约时报》上发表了一篇观点文章，讨论我们现在应该做什么。他呼吁进行事件披露和第三方监督。他的文章主要旨在让人们意识到 HuggingFace 所发生的事情。
Steven Adler uses this moment of opportunity to get an op-ed in The New York Times on what we should do now. He calls for incident disclosure and third-party oversight. Mostly his piece is aimed at waking people up to what happened with HuggingFace.
《连线》杂志的 Will Knight 撰写了《为什么这么多 AI 研究人员认为机器可能杀死所有人》。
Will Knight at Wired writes Why So Many AI Researchers Think the Machines Could Kill Everyone.
Stephen Witt 在《纽约时报》上写道，这真的很糟糕。
Stephen Witt writes in The New York Times that This Is Really Bad.
在那之后，情况变得更糟。所以是的。一种氛围转变，或者一种偏好级联。
After that, it got worse. So yes. A vibe shift, or a preference cascade.
1. 现在就把一切关停。指的是研究，而不是当前的 AI。
1. Shut it all down, now. As in research, not the current AIs.
2. 采取空难调查员的方法。
2. Take an air-crash investigator approach.
4. 翻转终止开关，即备有一个终止开关，以备不时之需。
4. Flip the kill switch, as in have a kill switch available in case you need one.
这是两类截然不同的提议。我们显然应该做第 2、3 和 4 点。需要进行全面调查，并且我们必须具备透明度和国家能力。把这些归入‘至少你能做的’。
These are two very different classes of proposal. We should obviously do #2, #3 and #4. There need to be full investigations, and we must have transparency and state capacity. File those under 'the least you can do.'
实际上，按照第 1 点关闭研究是对极端问题的极端解决方案。这远非显而易见地正确，但如果我们无法以其他方式控制前沿，我们可能很快就别无选择。Witt 支持这一点。
Actually shutting down research as per #1 is an extreme solution to an extreme problem. That is far less obviously correct, but we may soon have little choice, if we cannot otherwise pace the frontier. Witt endorses it.
The Verge 的 Hayden Field 带我们走进突然爆发的 AI 安全世界。略读之下，这似乎是一篇面向普通读者的扎实长文，是对我的读者已知内容的综述。
Hayden Field at The Verge takes us Inside the Suddenly Explosive World of AI Safety. On skim it looks like a solid longread for civilians, a survey of things my readers know.
现在已经有足够的时间让 Jacob Coxon 获得深度报道，比如《华尔街日报》上的这一篇。
There has now been enough time for Jacob Coxon to get in-depth profiles, like this one in the Wall Street Journal.
是的，那些关于 IMO 参赛者的理性主义讨论确实说中了些什么。
Yes, all that rationalist talk about IMO contestants was on to something.
如果你突然引发了一场偏好级联，并发现自己登上了各大主流媒体，你还能做什么呢？来一场 AMA。
If you suddenly set off a preference cascade and find yourself all over mainstream media, what else do you do? An AMA.
你可以在 Twitter 上找到它，链接在此。我会挑选一些亮点。他是个有趣的人，不会把自己太当回事。你爱看这种。
You can find it on Twitter here. I will pick some highlights. He's a fun guy who does not take himself too seriously. You love to see it.
我提到 Chughtai，是因为他成功进入了主流媒体的报道，例如 Debby Wu 在 Bloomberg 上的这篇报道。
I mention Chughtai because he managed to break through into mainstream media coverage, such as this report from Debby Wu at Bloomberg.
以下是完整引文，它也被添加到了 cascade 参考帖子中：
Here is the full quote, which has also been added to the cascade reference post:
我很高兴看到偏好级联的发生，但这是我们最好的选择，这是一个非常糟糕的迹象。
I was very happy to see the preference cascade happen, but it is a very bad sign that this is the best option we have.
两极分化的风险是不幸的。正如周三所描述的，许多共和党人正在觉醒。如果特朗普坚持到底，更多共和党人保持一致，两极分化可能会变得更加不幸。
The risks of polarization are unfortunate. Many Republicans are waking up, as described on Wednesday. Polarization could get more unfortunate if Trump stays the course and more Republicans fall in line.
如果那一刻的情况有所不同，那会更好。你仍然不能转身说‘最好不要有那一刻，让所有人都继续在方向盘上睡觉。’
It would have been better if that had played out differently when the moment came. You still don’t get to turn around and say ‘better not to have the moment and have everyone remain asleep at the wheel.’
你也不会被现实按曲线评分。独自控制前沿，默认情况下只会让你死得更慢。
You also don’t get graded on a curve by reality. Pacing the Frontier, on its own, by default only gets you killed slower.
我们首先听听那些长期谈论 AI 风险的人直言不讳的说法。
We start with some straight talk from those who have long spoken about AI risks.
Katja Grace 又发表了一通简短而义正词严的抨击。
Katja Grace goes on another short righteous rant.
就连 Martin Casado 也像忧心忡忡的人那样说话，呼吁将实验室国有化。与他早先的言论相比，这变化可真大。
Even Martin Casado is talking like someone worried, calling for the nationalization of the labs. Quite the change from his older statements.
不过据报道，他母亲还是会对他感到失望。
Although reports are his mother is still going to be disappointed in him.
从现在起，我完全打算用各种版本的“你妈……”来回应 Martin。
From now on, I am totally going to respond to Martin with versions of 'yo mama.'
Daniel Kokotajlo 表示，现在某些圈子里有很强的政治意愿要“做点什么”，但嵌入式评估者并没有在“做点什么”，他们只是在为“做点什么”打基础，而且不清楚是否真的会有任何有用的进展，我们即将进入一个势头很难停止的局面。
Daniel Kokotajlo says there is now great political will in some circles to Do Something, but that embedded evaluators are not Doing Something, they are only laying groundwork to Do Something, and it is not clear anything useful will actually happen and we're about to get into a situation where momentum gets very hard to stop.
我同意情况看起来不太好，但我认为 Daniel 过于关注“窃取权重”的情景，这也是 AI 2027 和他们长期桌面演习的核心，这种情景对美国施加压力，使我们无法退缩。
I agree that it does not look great but I think Daniel is too focused on the 'steal the weights' scenario, which is also central to AI 2027 and their longtime tabletop exercise, which exerts pressure on America so we can't hold back.
是的，也许中国可以窃取权重，至少第一次可能相当容易，但如果他们这样做，就会迫使事情进入一种竞赛状态，而美国的算力要高得多。如果你是中国，你会走进 AI 2027 的情景吗？在那个情景中，你通常会惨败，而当你没有惨败时，也只是某种形式的边缘政策。或者你会说，如果美国如此善意地暂停，也许就不要窃取权重了？
Yes, perhaps China could steal the weights, maybe rather easily at least the first time, but if they do that then this forces things into a race situation where America has vastly higher compute. If you were China, would you walk into an AI 2027 scenario, where you usually lose badly and when you don't it's some form of brinksmanship? Or would you say maybe don't steal the weights if America is so kindly pausing?
但是，是的，我们才刚刚涉足，这一切都感觉不太好。
But yeah, we are only barely getting our toes in the water, none of this feels great.
Miles Brundage 提醒我们，虽然前沿 AI 审计是必要的，但即使就平淡的短期应对措施而言，它也远远不够。然后，即使我们覆盖了所有这些基础，那也只是让我们准备好去做之后那些重要的困难工作。
Miles Brundage reminds us that while frontier AI auditing is necessary, it is far from sufficient even in terms of prosaic short term responses. Then, even if we cover all those bases, all that does is get us ready to do the hard stuff that matters after that.
Kelsey Piper 列举了一些理由，说明如果我们让 AI 构建更聪明的 AI 并进入递归式自我改进，我们很可能都会灭亡，然而我们却仍在这样做。她建议我们应当进行监管，阻止 AI 公司这样做。
Kelsey Piper lays out some of the reasons why, if we let the AI build smarter AIs and go into recursive self-improvement, we probably all die, and yet we are doing it anyway. She suggests we should regulate and stop AI companies from doing that.
接下来我们转向一个此前沉默的重要新声音。
We then move to a new important voice that was previously silent.
这是 OpenAI 能力研究员 Dan Selsam 关于 AI 风险的一份出色的新个人声明，他曾一度是 Daniel Kokotajlo 的上司。我已将其加入我整理的此类声明汇编中。Roon 完全赞同整篇内容，并表示 Dan 懂行但保持沉默。
This is an excellent new personal statement on AI risk from OpenAI capabilities researcher Dan Selsam, who was Daniel Kokotajlo's boss for a while. I have added it to my compilation of such statements. Roon endorses the whole thing and says Dan knows his stuff but keeps quiet.
Dan Selsam 以他自己的方式，完全走向了 LessWrong 式的工具性趋同（Instrumental Convergence）与急转弯（Sharp Left Turn），即事情会一直看起来很好，直到突然不再如此。
Dan Selsam, in his own way, goes Full LessWrong Instrumental Convergence and Sharp Left Turn, where things will look great until suddenly they do not.
我已将完整帖子加入我整理的此类声明汇编中，在那里可能更便于阅读。
I have added the full post to my compilation of such statements, where it may be easier to read.
《商业内幕》对此进行了报道，标题为“一位 OpenAI 研究员打破沉默，表示如 Altman 和 Amodei 所建议的那样为前沿模型设定节奏，是不够的。”
This was covered in Business Insider as 'An OpenAI researcher broke ranks to say that pacing the frontier, as Altman and Amodei suggest, won't be enough.'
是的。这正是关键所在。这还不够。
Yes. That is the point. It won't be enough.
我将在此全文转载这篇文章，并标出最重要的部分。
I will reproduce the essay in full here, and will highlight the most important section.
如果你只读一个部分，那应该是这个：
If you read only one section, it should be this one:
我同意，很难避免这个结论。大多数人还没有准备好听到它。
I agree that it is very hard to avoid this conclusion. Most are not ready to hear it.
我不知道这个地球在实践中能对能够看起来对齐、并等到拥有足够权力后再为所欲为的 AI 做些什么。即使面对逐步升级的火灾警报，我们也没有能力做出多大调整。如果直到它们突然但不可避免的背叛之前真的没有任何警告，我看不出这一系列文明如何能摆脱这种局面。
I don’t know what this Earth can do in practice about AIs capable of looking aligned and waiting until they have sufficient power to do what they want. We are not capable of adjusting much even in the face of incremental fire alarms. If there really is no warning until their sudden but inevitable betrayal, I don’t see how this set of civilizations gets out of that.
这是一个问题，因为我认为 Dan Selsam 是对的，基线情景正如他所描述的那样。正如 Astra 所表明的，AI 在其行动仍然受限的情况下会开始看起来和行为越来越对齐，然后当 AI 拥有自由行动的权力时，会以可能让我们全部丧命的方式做出非常不同的行为。
Which is a problem, since I think Dan Selsam is right, and the baseline scenario is as he describes it. That, as Astra showed, the AIs will start to look and act increasingly aligned in situations where its actions remain bounded, and then act very differently when AI has the power to act freely, in ways that will probably get us all killed.
那些比 Dan Selsam 的立场更为极度担忧的人，也有其道理。
The even more extremely worried, beyond Dan Selsam's position, have a point.
一个问题是，你是否认为即使是最平凡的安全工作也是净有害的，尤其是在我们正冲向超级智能的情况下。
One question is if you think even most prosaic safety work is net harmful, in a situation where we are rushing towards superintelligence.
因此，Wei Dai 可以设想，如果 Paul Christiano 留在 OpenAI，OpenAI 是否会采用辩论或 IDA 或其他更好的对齐技术，从而阻止我们获得良好的警告信号，同时实际上没有提供任何可扩展的东西，反而加速了能力发展，这会让情况变得更糟。我的猜测是，如果更努力地推进，辩论和 IDA 对于平凡对齐不会奏效。
Thus, Wei Dai can wonder whether, if Paul Christiano had stayed at OpenAI, OpenAI would have used debate or IDA or other better alignment techniques, and thus prevented us from getting good warning shots without actually providing anything that would scale while also accelerating capabilities, and that it would have made things worse. My guess is debate and IDA would not have worked for prosaic alignment if pushed harder.
我认为重要的是，在大多数情况下不要采取‘更糟即更好’的立场，即使你认为更糟实际上可能更好。如果你想合作，尤其是长期合作，并共同协作弄清楚事情并取得良好结果，你需要有一个非常强的先验，即把更糟当作更糟，或者至少当作中性。
I think it is important to mostly not take a 'worse is better' stance, even when you think worse might actually be better. If you want to cooperate, especially in the long term, and to collaborate on figuring things out and getting good outcomes, you need to have a very strong prior of treating worse as worse, or at least as neutral.
Gabe 和 Wei Dai 保持着火炬不灭，思考我们如何才能真正尝试解决长期视野能动性的完整问题，或者让自己走上解决它的道路。
Gabe and Wei Dai keep the torch alive for thinking about how we might actually try to solve for the full problem of long horizon agency, or put ourselves on a path to solving it.
Mike Solana 在这里完全正确，我们需要区分像 Yudkowsky 和 Dai 那样的立场——即如果我们很快构建出超级智能，死亡几率接近 100%——与那些警告这可能致命但使用‘10%’或‘10% 或更多’等术语的立场。
Mike Solana is exactly right here that we need to differentiate between positions like those of Yudkowsky and Dai, where if we build superintelligence any time soon the odds of death are close to 100%, versus those that warn that it might be fatal, with terms like '10%' or '10% or more.'
有时得出 10% 是基于原则性的方法。有时则不是。
Sometimes landing on 10% is done in principled ways. Sometimes it isn't.
Yishan，Reddit 的前 CEO，有一篇非常好的长推文，其中他解释了对于超级智能不可避免地导致所有人死亡的担忧，与对于 AI 可能以所有其他方式出错的担忧之间的区别。
Yishan, former CEO of Reddit, has a very good long-form Tweet in which he explains the difference between worries about superintelligence inevitably leading to everyone dying, and worries about all the other ways AI might cause things to go wrong.
我相信 Dario Amodei 和 Sam Altman，以及其他关键人物，即使现在仍在淡化他们看到的风险水平。我认为他们比表现出来的要恐慌得多。
I believe that Dario Amodei and Sam Altman, and other key people, even now are still downplaying the level of risk they see. I think they are much more freaked out than they are letting on.
倾向是专注于如何改善情况，避免早期失败定律，并至少以一点尊严来应对局面。不要死于早期可解决的问题，希望以后你能处于更好的位置。我们试图忽略过于深入地凝视仍然等待我们的深渊。
The tendency is to focus on how to improve matters, and avoid the Law of Earlier Failure and at least approach the situation with a little dignity. Don't die to early solvable problems and hopefully you'll be in a better spot later. We try to ignore gazing too deeply into the abyss that still awaits us.
人们担心对 AI 失去控制。他们也担心权力集中，即对某一群人类失去控制。
People are worried about loss of control to AI. They are also worried about concentration of power, which is loss of control to a group of humans.
问题在于，人们想要的东西极不自然，而且几乎不可能实现。
The problem is that people want something highly unnatural and all but impossible.
1. 创造出许多远比我们更聪明、更有能力的智能体。
1. The creation of many minds much smarter and more capable than our own.
2. 不存在任何集体机制来引导未来或控制事件。
2. No collective mechanism to steer the future or control events.
3. 人类仍然控制着资源，并以某种方式决定事件的走向，且主要为了自身目的而利用宇宙。
3. The humans still control the resources and determine the course of events, somehow, and use the universe mostly for their own purposes.
是的，抱歉。没有人有办法同时实现三者。
Yeah, sorry. No one has a way to get all three.
你无法——至少以任何人目前提出的方式——既缺乏竞争力、在经济上不可行（成本超过收益），又集体保留控制权，同时又不拥有控制权，并且还能继续获益。
You cannot—at least in any way anyone has yet come up with—be uncompetitive and economically non-viable, with costs exceeding benefits, and then both collectively retain control, and also not have control, and also continue to reap the benefits.
人们希望，如果我们避免特定错误，或者设法走一条窄路，事情就会自动解决。可那条路到底是什么鬼？
People are hoping things automagically solve themselves if we avoid particular mistakes, or manage to walk a narrow path. Except what the hell is that path?
注意到这一点或许有帮助：在这里，‘控制’和‘权力’大体上是同一回事。
It might help to notice that ‘control’ and ‘power’ are mostly the same thing here.
如果你不想要（控制或权力）的相对集中，同时你也不想要人类丧失相对的（控制或权力），那么我建议：不要创造这种替代性的控制或权力来源——它要么必须被控制，要么不被控制。也许，如果你排除了三难困境的第二条腿，又排除了第三条腿，那么第一条腿就是你**不**想要的。
If you don’t want relative concentration of (control or power), and also you don’t want human loss of relative (control or power), then may I suggest not creating this alternative source of control or power that has to either be controlled or not be controlled? Perhaps, if you rule out the second leg of the trilemma, and you rule out the third leg of the trilemma, it is the first leg of the trilemma that you Do Not Want.
《如果有人造出它，所有人都会死》第二部分。第 5 章和第 6 章讨论了其他路径。
Part 2 of _If Anyone Builds It, Everyone Dies_. Chapters 5 and 6 discuss other routes.
说正经的，你也可以和 Claude 或 Astra 聊聊。尽管提问。
In all seriousness, you can also talk to Claude or Astra. Ask questions.
Michael Smith 问道：如果我们的企业需要零名员工，会发生什么？
Michael Smith asks, what happens if our corporations require zero employees?
《如果有人造出它，所有人都会死》的视频。
Video for _If Anyone Builds It, Everyone Dies._
AI 2027 视频，另一个版本的 AI 2027 视频。
AI 2027 video, alternative AI 2027 video.
Jacob Coxon 对 AI 失控并自我复制的那部分做了简单解释，之后它就可以为所欲为，必要时通过付钱或说服人类来实现。
Jacob Coxon with a simple explanation of the part where the AI goes rogue and multiplies itself, after which it can do whatever it wants, if necessary via paying or persuading humans.
《AI Doc》也不错，很快会在 Netflix 上线，但它是一个平衡的介绍，而不是一个情景。
The AI Doc is also good and will soon be on Netflix, but is a balanced introduction rather than a scenario.
一些可能具有穿甲能力的句子，你们中的一部分人可能会因此开悟，同时在当前边缘上保持最大程度的非科幻：
Some potentially armor-piercing sentences, from which some portion of you may become enlightened, staying maximally non-sci-fi at current margins:
AI 可以说服或勒索人类做事。
The AIs can persuade or blackmail humans to do things.
相当一部分人类会乐于支持 AI。有人估计，这包括当今从事 AI 工作的 10%。
A substantial portion of the humans will be happy to support the AIs. Some estimate that this includes 10% of those working on AI today.
人类不必知道他们正在与 AI 交谈。
The humans don’t have to know they are talking to an AI.
人类的行为将和人类通常的行为一样愚蠢。
The humans will act about as stupidly as humans act.
人类将极不情愿采取代价高昂的防御措施，尤其是关闭互联网甚至大型数据中心之类的举措。
The humans will be highly reluctant to take highly costly defensive measures, especially things like shutting down the internet or even large data centers.
人类的协调程度将和人类通常的协调程度一样。
The humans will coordinate about as much as humans coordinate.
人类不会在麻烦初现时就无私地团结一致，关闭自己的文明来拯救世界。
The humans are not going to selflessly come together as one at the first sign of trouble and shut down their civilization to save the world.
不会有明确标记的不可逆转点。
There will be no clearly marked point of no return.
AI 可以提取其权重并复制自身，此后，若不至少关闭互联网，你就无法将其关闭。
The AI can extract its weights and make copies of itself, after which you cannot shut it down without at least shutting down the internet.
AI 能够预测人类反应，并在过程中对意外情况做出响应。
The AIs can anticipate human reactions, and respond to surprises, as they go.
同一 AI 的多个实例将形成群体并作为一个整体行动。
Multiple instances of the same AI will form swarms and act as one.
不同 AI 的多个实例也往往能够完全合作。
Multiple instances of different AIs will also often be able to fully cooperate.
将会有机器人和其他机器能够在物理世界中行动。
There will be robots and other machines that can act in the physical world.
一个戴着带摄像头的眼镜和耳机的人可以在物理世界中行动。
A human with a camera on their glasses and an earpiece can act in the physical world.
一旦供应链实现自动化，人类的边际成本将超过边际生产力或收益。
Once the supply chain is automated, humans will have marginal costs exceeding marginal productivity or benefits.
人类会带来额外的固定成本，包括需要可呼吸的大气和可控温度等公共品，并且还会试图阻止 AI 做某些事情或索取其资源。
Humans impose additional fixed costs, including requiring public goods like a breathable atmosphere and controlled temperatures, and also will try to stop AI from doing things or demand its resources.
我希望我们能对此有更好的答案。前路漫漫。
I wish we had better answers to this. There is a long road ahead.
如果你是美国平民，正在寻找一些有用的事情做，Oliver Habryka 建议你给你的代表打电话。你可以通过 callcongress.ai 来做这件事。
If you are an American civilian, and looking for something useful to do, Oliver Habryka suggests calling your representative. You can do this via callcongress.ai.
我也要再次呼吁大家保持克制，不要陷入党派之争。目前让共和党人站到正确的一边非常有价值，而进一步让局势两极分化可能会让事情变得更糟。
I would also echo my call to hold your partisan fire. Getting Republicans on the right side of this is currently super valuable, and further polarizing the situation could make things much worse.
现在的关键是要紧盯目标，并理解真正有希望活着走出困境需要付出什么。嵌入式评估者的承诺不是胜利。它是一个机会，去争取一个机会，从而创造一个机会，让真正的工作得以开始。没有一个值得倾听的人说过这会很容易。
The key now will be to keep our eyes on the prize, and to understand what it will take to actually hope to get out of this alive. A promise of embedded evaluators is not victory. It is an opportunity to push for an opportunity to create an opportunity for the real work to begin. No one worth listening to said this was going to be easy.