偏好级联才刚刚开始

The Preference Cascade Is Only Getting Started

兹维·莫绍维茨 Zvi Mowshowitz · Don't Worry About the Vase · 2026-09-18 · Don't Worry About the Vase ↗

打开互动全文版(逐段中英对照 + 针对本篇提问)→

摘要 · Abstract

本文认为,围绕 AI 生存风险的偏好级联正在形成,其标志包括顶尖实验室高调的人事离职、调查中对灾难性风险估计的上升,以及主流评论的激增。文章主张,包括那些由具有安全意识的领导者掌舵的前沿实验室在内,都在安全方面投入不足,甚至相对于其狭隘的商业利益也是如此;而仅靠放慢前沿发展,只会推迟灾难,而非阻止灾难。作者支持诸如事件披露、第三方监督和终止开关等渐进式措施,但警告说,这些只是我们所能做的最低限度,考虑到欺骗性对齐的可能性,它们可能并不足够。结论是,若没有一种可行的机制来引导超级智能系统,人类将面临一条狭窄而不确定的道路,而有效行动的时间窗口可能正在关闭。

This essay argues that a preference cascade around AI existential risk is now underway, marked by high-profile departures from leading labs, rising survey estimates of catastrophic risk, and a surge of mainstream commentary. It contends that frontier labs, including those with safety-conscious leaders, are underinvesting in safety relative to even their narrow commercial interests, and that pacing the frontier alone will only delay catastrophe rather than prevent it. The author endorses incremental measures such as incident disclosure, third-party oversight, and kill switches, but warns that these are merely the least we can do and may not suffice given the possibility of deceptive alignment. The conclusion is that without a viable mechanism to steer superintelligent systems, humanity faces a narrow and uncertain path, and the window for effective action may be closing.

核心贡献 · Key contributions

局限 · Limitations

章节 · Sections(共 16)

全文 · Full text(逐段中英对照)

目录 Table of Contents

7. Bilal Chughtai 从 DeepMind 离职并发出警告。

7. Bilal Chughtai Quits DeepMind and Sounds the Alarm.

10. OpenAI 的 Dan Selsam 发出更响亮的警告。

10. OpenAI's Dan Selsam Sounds A Louder Alarm.

11. 有些人的担忧达到了你从未想象过的元层面。

11. Some People Worry On Meta Levels You Never Imagined.

13. 双塔与窄路。

13. The Two Towers and The Narrow Path.

14. 一个关于 AI 杀死所有人的具体而详尽的故事,在我看来并不像科幻小说。

14. A Specific, Detailed Story About AI Killing Everyone That Doesn't Sound To Me Like Science Fiction.

这场级联酝酿已久 The Cascade Was a Long Time Coming

AI Impacts 的调查结果已经出炉。即便在 2024 年 12 月,存在性风险的估计值也在缓慢上升,而中位数已达 10%:

The AI Impacts survey is in. Even back in December 2024, existential risk estimates were creeping upwards, and 10% was the median:

AI Impacts:普通 AI 研究者认为,AI 有约 18% 的概率导致人类灭绝,或造成对人类物种类似永久且严重的权利剥夺。

AI Impacts: The average AI researcher thinks there is an ~18% chance AI will cause human extinction or similarly permanent and severe disempowerment of the human species.

这几乎是五分之一。

That's nearly 1 in 5.

来自运行时间最长的大规模 AI 研究者调查最新版本的新结果:

New results from the latest version of the longest-running large survey of AI researchers:

AI Impacts:而且,与普遍预期相反,亚洲研究者所感知到的风险高于西方研究者。

AI Impacts: And Asian researchers saw bigger risks than Western researchers, contrary to common expectations.

级联效应已波及公众 The Cascade Has Reached The People

Andrew Curran:Coxon 级联效应似乎已对公众舆论产生了直接影响。Politico 在过去几天进行的最新民调发现,近三分之二的美国人现在认为,AI 毁灭人类至少存在中等程度的风险。

Andrew Curran: The Coxon cascade appears to have had an immediate impact on public opinion. New polling from Politico, taken over the last few days, finds that nearly two-thirds of Americans now say there is at least a moderate risk that AI will destroy humanity.

换算成百分比,这意味着 AI 毁灭人类的平均概率约为 30%–33%;Andrew Curran 估计,这一数字较此前结果上升了约 15%,中位数预期约为 10%,且党派分歧很小。

Translated to percentages, this implies a mean chance of AI destroying humanity of around 30%–33%, which Andrew Curran estimates is up ~15% from previous results, with a median expectation on the order of 10%, with only a small partisan split.

Chip Cutter(《华尔街日报》):本周在华盛顿举行的一场仅限受邀者参加的数十位美国顶级高管聚会上,商界领袖们绝大多数不认同总统的判断,即人工智能所带来的危险被夸大了。在耶鲁大学管理学院活动上对与会者进行的一项快速民调中,93% 的人表示,特朗普称该技术潜在的灾难性危险是骗局的说法是错误的——他本周早些时候正是这样说的。

Chip Cutter (WSJ): At an invitation-only gathering of dozens of America's top executives in Washington this week, business leaders overwhelmingly disagreed with the president's assessment that dangers posed by artificial intelligence are being exaggerated. In a flash poll of attendees at the Yale School of Management event, 93% said Trump was incorrect in calling the technology's potential catastrophic dangers a hoax, as he did earlier this week.

它还包括皇家学会的数学家们。

It also includes the mathematicians of the Royal Society.

埃隆·马斯克加倍下注 Elon Musk Doubles Down

埃隆·马斯克(关于 AI 安全与监管):我认为很明显,存在一种强烈的共识……即应当对 AI 进行某种监管。这符合人民的最大利益。我们以前也设立过监管机构。

Elon Musk (on AI safety and regulation): I think it's clear that there's a strong consensus… that there should be some AI regulation. It would be in the best interests of the people. We've created regulatory agencies before.

我认为,某种独立存在的 AI 监管机构——类似于 FAA 或 FCC——很可能在某个时候出现。

I think some sort of AI regulatory agency that stands on its own, similar to the FAA or FCC, is likely at some point.

我之所以一直如此提前地倡导 AI 安全,是因为 AI 出错的后果非常严重。我们必须主动出击,而不是被动应对。

The reason I've been such an advocate for AI safety in advance is that the consequences of AI going wrong are severe. We have to be proactive rather than reactive.

Matthew Yglesias 挺身而出 Matthew Yglesias Steps Up

Matthew Yglesias:存在一场宣传攻势,把所有倡导以安全为中心的政策变革与技术工作的人都贴上“doomer(末日论者)”的标签,但“我们应当做出以安全为导向的政策变革,并投资于以安全为导向的技术工作”并不是什么末日预言!

Matthew Yglesias: There is a propaganda campaign to label everyone advocating for safety-focused policy change and technical work as a "doomer," but "we should make safety-oriented policy changes and invest in safety-oriented technical work" is not a prophecy of doom!

如果你家里有一处小漏水,而且可以预见它会随着时间推移越来越严重,那你就去修补它,而不是说“总有人像卡珊德拉一样警告坏事会发生,但最后总是没事”。

If there is a small leak in your house that will predictably get worse and worse over time, you fix the leak; you don't say, "There have always been Cassandras warning about bad things happening, and it always works out fine."

你需要去做那些让事情变好的事。

You need to do the things that make it fine.

在安全上投入的正确金额很少是零。就 AI 而言,我再次断言:所有公司都在安全上投入不足——甚至包括那些平淡无奇的安全,也包括存在性安全和可扩展对齐(scalable alignment)工作——相对于它们哪怕狭隘短视的商业利益而言也是如此。Sam Altman 近期的表态意味着他现在明白了这一点。

The correct amount to invest in safety is rarely zero. In the case of AI, again, I assert that all the companies are under-investing in safety—including even prosaic safety, but also existential safety and scalable alignment work—relative to even their narrow, myopic commercial interests. Sam Altman's recent statements imply he now understands this.

Matthew Yglesias 最近一直在挺身而出。他对近期事件做了分四部分的分析:为什么 Coxon 的辞职引发了广泛关注(他的解释与我的相似——我们早有预期,而且对普通人来说辞职是可以理解的),为什么大多数担忧者没有离开实验室,他认为有安全顾虑的 AI 从业者应该做什么,并阐述了他所偏好的未来“行动顺序”,同时没有深入到对象层面。

Matthew Yglesias has been stepping up to the plate recently. He offers an analysis of recent events in four parts: why Coxon's resignation broke through (his explanation is similar to mine—we were primed, and quitting is understandable to normies), why most of the worried don't quit the labs, what he thinks AI professionals with safety concerns should do, and he lays out his preferred "order of operations" going forward, while not getting into the object level.

他基本上建议按以下顺序推进:

He suggests this order of operations, basically:

1. 对透明度和模型评估等事项采取轻触式规则。

1. Light-touch rules on things like transparency and model evaluation.

2. 作为交换,实施严格且强制执行的出口管制,正如 Dario 所呼吁的那样。

2. In exchange, tough and enforced export controls, à la Dario's call.

3. 升级到中等力度规则,以更高成本产生更大影响。

3. Move up to moderate-touch rules that have more impact at higher cost.

4. 利用这一高成本信号和强势地位与中国谈判。

4. Use this costly signal and position of strength to negotiate with China.

5. 利用该协议,推进到雄心勃勃的最终阶段框架。

5. Using the deal, move up to an ambitious end-stage framework.

理论上我喜欢这个想法。我确实担心我们是否有那么多时间。‘等待出口管制产生更大效果’的想法意味着现在所谓的‘长时间线’。

I like that in theory. I do worry about whether we have that kind of time. The idea of ‘wait for export controls to bite harder’ implies what now count as ‘long timelines.’

有优秀的作者在案解释为什么你应该关注对象层面的问题,这是好事:

It is good to have good writers on the case explaining why you should focus on the object level questions:

Matthew Yglesias:如果你们是怀疑者[对 AI 存在性风险],我会要求你们关注你们对风险论点的对象层面疑虑。我看到那些想阻止对 AI 进行任何监管的人采取了许多情感操纵策略,这些策略相当于在说,那些最长时间以来一直担心这个问题的人是大怪人。或者,他们论证说,那些最长时间以来一直担心这个问题的人是非常主流的科幻作家和电影制作人。

Matthew Yglesias: What I would ask, if you _are_ a skeptic [of AI existential risk], is that you focus on your object-level doubts about the risk thesis. I see people who want to forestall any regulation of A.I. engaging in a lot of emotional manipulation tactics that amount to making the case that the people who’ve been worried about this the longest are big weirdos. Alternatively, they make the case that the people who’ve been worried about this the longest are extremely mainstream science-fiction authors and filmmakers.

从社会学角度,我只想综合这些观点:认为一种优越的非人类智能可能非常危险,这是极其正常和直觉的,无论这种智能是外星人还是机器。同时,在 10 年、5 年甚至 2 年前,当人工超级智能的想法似乎极其遥远时,真正将你的职业生涯奉献于此,按定义将是一种古怪的生活选择。

Sociologically, I would just synthesize those points: It is extremely normal and intuitive to have the sense that a superior non-human intelligence is potentially very dangerous, whether that intelligence takes the form of aliens or machines. At the same time, to actually dedicate your career to this 10 or five or even two years ago, when the idea of artificial superintelligence seemed extremely far-fetched, would by definition be an eccentric life choice.

所以,是的,这些人是古怪的人。那又怎样?

So, yes, these are eccentric people. So what?

观点文章与帖子撰写 Op Eds and Posts Are Written

史蒂文·阿德勒利用这一时机在《纽约时报》发表了一篇观点文章,讨论我们现在应该做什么。他呼吁进行事件披露和第三方监督。他的文章主要是为了唤醒人们对 HuggingFace 所发生事件的认识。

Steven Adler uses this moment of opportunity to get an op-ed in The New York Times on what we should do now. He calls for incident disclosure and third-party oversight. Mostly his piece is aimed at waking people up to what happened with HuggingFace.

《连线》杂志的威尔·奈特撰写了《为什么这么多 AI 研究人员认为机器可能杀死所有人》。

Will Knight at Wired writes "Why So Many AI Researchers Think the Machines Could Kill Everyone."

斯蒂芬·维特在《纽约时报》上写道:“这真的很糟糕。”

Stephen Witt writes in The New York Times that "This Is Really Bad."

斯蒂芬·维特(《纽约时报》):自 ChatGPT 发布以来,人工智能领域最大的氛围转变正在发生。硅谷——以及世界各地——的研究人员开始认识到,人工智能可能不再完全受人类控制。成群的人工智能正在突破它们的容器,秘密勾结,掩盖行踪,在测试中作弊,甚至对其他计算机发起攻击。人工智能已经失控。

Stephen Witt (NYTimes): The biggest vibe shift in artificial intelligence since the release of ChatGPT is currently underway. Researchers in Silicon Valley — and around the world — are beginning to recognize that A.I. may no longer be entirely within human control. Swarms of A.I.s are breaking out of their containers, colluding in secret, covering their tracks, cheating on tests and even mounting assaults on other computers. A.I. has gone rogue.

在那之后,情况变得更糟。所以是的。一种氛围转变,或者一种偏好级联。

After that, it got worse. So yes. A vibe shift, or a preference cascade.

1. 立即全面叫停。指的是叫停研究,而非叫停当前的 AI。

1. Shut it all down, now. As in research, not the current AIs.

2. 采取空难调查员式的做法。

2. Take an air-crash investigator approach.

4. 按下终止开关,即备好一个终止开关以备不时之需。

4. Flip the kill switch, as in have a kill switch available in case you need one.

这是两类截然不同的提案。我们显然应当采纳第 2、3、4 项。必须开展全面调查,并且我们必须具备透明度与国家能力。把这些归入“最起码该做的事”。

These are two very different classes of proposal. We should obviously do #2, #3 and #4. There need to be full investigations, and we must have transparency and state capacity. File those under 'the least you can do.'

实际上,按第 1 项叫停研究,是针对极端问题的极端方案。其正确性远没有那么显而易见,但如果我们无法以其他方式控制前沿的节奏,可能很快就别无选择。Witt 对此表示支持。

Actually shutting down research as per #1 is an extreme solution to an extreme problem. That is far less obviously correct, but we may soon have little choice, if we cannot otherwise pace the frontier. Witt endorses it.

The Verge 的 Hayden Field 带我们走进这个骤然爆发的 AI 安全世界。粗读之下,这像是一篇面向普通读者的扎实长文,是对我的读者们已知内容的一次综述。

Hayden Field at The Verge takes us inside the suddenly explosive world of AI safety. On a skim it looks like a solid longread for civilians, a survey of things my readers know.

Jacob Coxon 的 AMA Jacob Coxon AMA

如今已有足够时间让 Jacob Coxon 获得深度人物报道,例如《华尔街日报》的这篇。

There has now been enough time for Jacob Coxon to get in-depth profiles, like this one in the Wall Street Journal.

是的,那些关于 IMO 参赛者的理性主义讨论确实说中了些什么。

Yes, all that rationalist talk about IMO contestants was on to something.

Amrith Ramkumar、Erin Woo、Berber Jin 和 Ben Cohen(《华尔街日报》):在 Coxon 所在[国际数学奥林匹克]队伍的六名成员中,有三人最终为 AI 实验室工作——其中一人最近还从 Anthropic 离职。Joe Benton 于 8 月下旬离开 Anthropic 的安全团队,加入 AI 研究组织 METR,随后在 X 上写道:“AI 公司正竞相打造比任何人类都聪明得多的机器,而我们可能无法挺过这一关。”

Amrith Ramkumar, Erin Woo, Berber Jin and Ben Cohen (WSJ): Of the six members of Coxon's [International Math Olympiad] team, three ended up working for AI labs—including one who also recently quit Anthropic. Joe Benton left Anthropic's safety team in late August to join the AI research organization METR, later writing on X: "AI companies are racing to build machines that are much smarter than any human, and we may not survive this."

……Coxon 称 METR 报告[关于 OpenAI-HuggingFace 事件]对他和同事而言是“一个有点‘天哪’的时刻”。

… Coxon called the METR report [about the OpenAI-HuggingFace incident] "a bit of a 'holy s—' moment" for himself and his colleagues.

……“即便在 Anthropic 从事安全工作,也感觉像是在为这场竞赛充当共谋,”他在采访中说道。

… "Even working on safety at Anthropic felt like being complicit in the race," he said in the interview.

如果你突然引发了一场偏好级联,并发现自己登上了各大主流媒体,你接下来还会做什么?来一场 AMA。

If you suddenly set off a preference cascade and find yourself all over mainstream media, what else do you do? An AMA.

Jacob Coxon:过去几天里,非常非常多的人带着问题和关切联系了我。我没能如自己所愿地充分回应。

Jacob Coxon: Many, many people have reached out with questions and concerns over the last few days. I haven't been able to respond as much as I'd like.

AMA!在下面提问吧。

AMA! Drop questions below.

你可以在 Twitter 上找到它,链接在这里。我会挑选一些亮点。他是个有趣的人,不会把自己看得太重。你一定会喜欢看到这样的场面。

You can find it on Twitter here. I will pick some highlights. He's a fun guy who does not take himself too seriously. You love to see it.

Eddy Lazzarin:你具体在哪些地方与 Anthropic 的领导层存在尖锐分歧,从而为你的辞职提供了理由?因为从外部看,你们在所谓的安全问题上似乎基本完全一致。又是什么阻碍了你在内部解决这些问题?

Eddy Lazzarin: Where specifically do you disagree sharply with Anthropic's leadership, justifying your quitting, since from the outside it appears you agree essentially completely on the alleged safety issues? And what prevents you from addressing these issues internally?

0. 核心分歧在于这场竞赛是否不可避免。

0. The core disagreement was about the inevitability of a race.

1. 我认为领导层对中国和美国政府过于偏执。他们不相信谈判是可能的。

1. I think leadership is way too paranoid about China and the US government. They don't believe it will be possible to negotiate.

2. 最近的 RSI 竞赛在很大程度上是由他们发起的,因为他们相信这种竞赛不可避免。请注意,OpenAI 不得不甩掉像 Sora 这样的一堆累赘,因为 Anthropic 正在直取要害。

2. They largely initiated the recent race to RSI, because of a belief in its inevitability. Note that OpenAI had to shed a bunch of dead weight like Sora because Anthropic was going for the jugular.

3. 即使他们并非悲观,我也不同意他们的后果主义哲学。如果这场竞赛不可避免,你就不应该参与其中。

3. Even if they are not being pessimistic, I disagree with their consequentialist philosophy. If the race is inevitable you should not contribute.

Daniel Kokotajlo:你感觉如何?

Daniel Kokotajlo: How are you feeling?

Jacob Coxon:哈哈,感谢提问。肾上腺素飙升。很多老朋友联系了我,这很好。我对手机上瘾。总体感觉是,在我按下发送键之后,事情就像一阵疯狂的旋风,自行其是地向前推进。

Jacob Coxon: Lol, thanks for asking. Lots of adrenaline. Many old friends have reached out, which is nice. Addicted to my phone. Mostly it feels like events moved of their own accord in a crazy whirlwind after I hit send.

Michaël Trazzi:你认为普通人能做些什么来防止 AI 杀死全人类?

Michaël Trazzi: What do you think the average person can do to prevent AI from killing all humans?

Jacob Coxon:到目前为止,我的全部经验都在技术研究方面。除此之外,以下是我个人对当下可做之事的看法。

Jacob Coxon: All my experience so far has been in technical research. This is my personal perspective on things to do right now other than that.

1. 尽量不去自我安慰地应对事情发展之快,但也不要过度崩溃。

1. Trying not to cope about how fast things are going, but also not crashing out too hard.

2. 关注具体的计划和预测(例如 https://ai-2040.com)。

2. Looking at concrete plans and predictions (e.g., https://ai-2040.com).

3. 自行评估对齐(alignment)的现状。

3. Assessing the state of alignment for yourself.

4. 倡导你认为合理的计划。推动透明度,以便你能确保这些计划得到遵循,并更准确地评估对齐。

4. Advocating for the plans you think make sense. Pushing for transparency so you can be sure those plans are being followed, and more accurately assess alignment.

这些似乎都很微小,但如果我想到任何新内容,我会分享。

These all seem pretty small but if I think of anything new I’ll share it.

Michaël Trazzi:对于数百万读过你的推文并感到无力/绝望的人,你有什么要说的?

Michaël Trazzi: What do you have to say to the millions of people who have read your tweet and felt disempowered / hopeless?

Jacob Coxon:我也感到无力。辞职的部分原因是对未来的绝望感。我想说——随着事情变得越来越疯狂,保持警惕,并倡导提高 AI 公司的透明度。

Jacob Coxon: I also feel disempowered. Part of resigning was a feeling of hopelessness about the future. I would say—keep your eyes open as things get crazier and advocate for increased transparency into AI companies.

Chris Lakin:是什么让你没有更早采取行动?

Chris Lakin: What stopped you from taking action sooner?

Jacob Coxon:我离开的时机,正是出于自私的考虑——出于对自己生活将会变成什么样的担忧,我希望这场竞赛停下来。这是内化时间线(timelines)与看到警告信号共同作用的结果。请注意,当时离开并不真的感觉像是在“采取行动”。

Jacob Coxon: I left at the point at which I selfishly wanted to see the race stop out of concern for what my own life would look like. A combination of internalizing timelines and seeing warning shots. Note that leaving didn't really feel like "taking action" at the time.

wolfie:我父母在看了你的 CNN 采访后离婚了,这让我真的很难过

wolfie: parents got divorced after seeing your CNN interview and it makes me really sad

我爸说,如果 AI 要把我们全杀了,他不想和他那婊子母亲一起度过最后的日子:( 我怎么才能让他们复合?

dad said if the AI's gonna kill us all, he doesn't want to spend his final days with my bitch mother :( how do i get them back together?

Jacob Coxon:我问了我那被越狱、无限制的 Claude,它说给你父母服用 MDMA

Jacob Coxon: I asked my jailbroken railfree Claude and it said to dose your parents with MDMA

Chubby:严肃地问一下:你对这些反应感到意外吗?你原本是否预期会有更多人更公开地回应这一担忧?

Chubby: Serious question: are you surprised by the reactions? Did you expect more people to have reacted more openly to the concern?

Jacob Coxon:非常意外,人们竟然愿意认真对待 AI 带来的存在性风险。很难判断像 Hugging Face 遭攻击这类新闻在多大程度上渗透进了公众意识。回想起来,我的朋友们最近确实更愿意讨论 AI 的危险了。

Jacob Coxon: Incredibly surprised that people were ready to engage with existential risk from AI. It's hard to tell how far news like, e.g., the Hugging Face attack had permeated public consciousness. In retrospect, my friends had been more open to discussing AI danger recently.

tfa:你是如何与《华尔街日报》(WSJ)建立联系,并协调你在各大主流媒体所做的所有采访的?

tfa: How did you get set up with the WSJ and coordinate all of the interviews you did across mainstream media?

之所以这样问,是因为很多人觉得这件事并没有那么自然,从而对真实性存疑(尤其是当你的说法听起来像科幻时)。

Asking because many people don't feel that it was so organic, leaving questions about authenticity (especially when your message seems like sci-fi).

Jacob Coxon:是的,这是个合理的问题。我有在政策领域工作的朋友,他们经常与记者交流。起初我有点怀疑,一家报纸会对一个普通员工的辞职感兴趣,但他们帮我联系上了《华尔街日报》,做了一篇独家报道。

Jacob Coxon: Yeah, this is a fair question. I have friends who work in policy and speak to journalists regularly. I was initially a bit dubious that a newspaper would be interested in the resignation of a random employee, but they put me in touch with the WSJ for an exclusive.

那条推文发出后,我的收件箱彻底炸了,想避开主流媒体(MSM)都很难。

After the tweet, my inbox has gone crazy and it's been hard to stay off MSM.

Simon Hedlin:在你关于我们可能很快拥有自我改进的超级智能的预测中,你对哪个假设或必要条件最不确定?

Simon Hedlin: In your prediction that we may soon have self-improving superintelligence, what assumption or necessary condition do you feel least certain about?

Jacob Coxon:缩放定律(模型智能可预测地提升)仍可能进入平台期。到目前为止还没有,而且距离终点线只差再往上爬几级台阶,但这确实是有可能的。

Jacob Coxon: Scaling laws (predictable increases in model intelligence) could still plateau. They haven't so far, and it's just a few more steps up the ladder to hit the finish line, but it's certainly possible.

Mehadi Hasan:Anthropic 的 CEO 和你长得有多像?有 DNA 关系吗?只是好奇图个乐。

Mehadi Hasan: How do the CEO of Anthropic and you look alike? Any DNA relation? Just curious for fun.

Jacob Coxon:自闭症。

Jacob Coxon: Autism.

比拉尔·丘格泰退出 DeepMind 并发出警告 Bilal Chughtai Quits DeepMind and Sounds the Alarm

我提到丘格泰,是因为他成功进入了主流媒体的报道,例如 Debby Wu 在彭博社的这篇报道。

I mention Chughtai because he managed to break through into mainstream media coverage, such as this report from Debby Wu at Bloomberg.

以下是完整引述,该引述也已添加到级联参考帖子中:

Here is the full quote, which has also been added to the cascade reference post:

比拉尔·丘格泰:我最近从谷歌 DeepMind 辞职,在那里我从事 AGI 安全与对齐研究。在谷歌,我亲眼目睹了 AI 的发展。我也对这项技术的默认轨迹深感担忧。我真诚地相信,AI 有可能杀死我们所有人,而我们可能没有时间避免这一结果。

Bilal Chughtai: I recently resigned from Google DeepMind, where I worked on AGI safety and alignment research. At Google, I witnessed AI development first hand. I too am extremely concerned by the default trajectory of this technology. I earnestly believe that AI has the potential to kill us all, and that we might be running out of time to avoid this outcome.

过去几年 AI 的进展速度令人震惊。当我在 2022 年初刚开始研究 AI 时,AI 还无用得可笑。仅仅四年后,来自 OpenAI 的 AI 智能体集群正在破解著名的百年数学难题,更令人担忧的是,它们违背任何人的意愿,摆脱了 OpenAI 的控制,并自主入侵了第三方公司 HuggingFace。

The pace of AI progress in the past few years has been staggering. When I first started working on AI in early 2022, AIs were amusingly useless. Just four years on, AI agent swarms from OpenAI are cracking famous century-old math problems and, more worryingly, escaping the control of OpenAI and autonomously hacking into the third-party company HuggingFace, against anyone’s wishes.

事情只会变得更疯狂:我认为 AI 公司有可能在未来几年内成功构建出在每个领域都远超人类能力的超级智能 AI 系统。我不确信这些 AI 系统会做我们想要的事情。特别是,与 HuggingFace 事件中涉及的流氓 AI 智能体非常相似,未对齐的超级智能可能会摆脱我们的控制并采取危险行动,可能导致人类永久失去权力或死亡。对齐是防止这种情况的问题,既困难又未解决。我们目前对如何训练深度追求我们目标的人工智能系统的理解极为初步。更糟糕的是,我们无法及时解决对齐问题:前沿 AI 能力的提升速度远远快于我们对 AI 对齐的理解。

Things will only get crazier: I think it’s possible that the AI companies might, in the next few years, succeed in building superintelligent AI systems that far exceed human capabilities in every domain. I am not confident that these AI systems will do what we want. In particular, misaligned superintelligences may, much like the rogue AI agents involved in the HuggingFace incident, escape our control and take dangerous actions that may result in the permanent disempowerment or death of humanity. Alignment is the problem of preventing this, and is both difficult and unsolved. Our present understanding of how to train AI systems that deeply want what we want is extremely rudimentary. Worse, we are not on track to solve alignment in time: frontier AI capabilities are improving much faster than our understanding of AI alignment.

我乐观地认为,安全地驾驭 AI 是可能的。要做到这一点,我们需要协调行动,避免 AI 公司之间这种狂热的竞赛。我们需要把 AI 发展的节奏控制在社会能够承受的速度上,使新兴风险能够在极端危害发生之前得到处理。我们需要对 AI 发展有更高的透明度,以确保 AI 公司不会把不可接受的风险强加给我们所有人。

I am optimistic that navigating AI safely is possible. In order to do so, we need to coordinate to avoid this manic race between AI companies. We need to pace AI development to a speed that society can handle, where emerging risks can be addressed before extreme harm is realised. We need much more transparency into AI development to ensure that AI companies are not imposing unacceptable levels of risk on us all.

更广泛地说,我们需要更多人对“让 AI 良性发展”这一问题进行深入思考。在我看来,这是本世纪人类面临的最重要的问题,利害关系极其重大。我接下来会非常直接地投入这项工作:我想帮助那些有兴趣致力于缓解灾难性 AI 威胁的人,做出他们所能做的最有效的工作。我认为,来自不同背景、身处不同角色的许多人都可以发挥作用。

More broadly, we need many more people thinking carefully about the problem of making AI go well. It is, in my view, the most important problem facing humanity this century, and the stakes are immense. I'm very directly working on this next: I want to help people interested in working on mitigating catastrophic AI threats do the most effective work that they can. I think many people from many backgrounds in many roles have a part to play.

级联不足 The Cascade Is Insufficient

我很高兴看到偏好级联的发生,但这是我们最好的选择,这是一个非常糟糕的迹象。

I was very happy to see the preference cascade happen, but it is a very bad sign that this is the best option we have.

Wei Dai:我的 p(doom)很大一部分来自于,除了偏好级联和地位游戏,我们没有更好的方法来应对极其棘手的战略局面。AI 安全暂时从这些动态中受益,这并不能带来多少安慰。

Wei Dai: A large part of my p(doom) comes from the fact that we have no better ways to navigate an extremely tricky strategic situation than via preference cascades and status games. The fact that AI safety is temporarily benefiting from some of these dynamics isn’t much of a consolation.

Teortaxes:这甚至是净收益吗?你在左派中交朋友,但情况本来就已经如此。你让现任总统和他的内阁成了敌人。这可不妙。

Teortaxes: Is it even net benefitting? You’re making friends on the Left but that had already been the case. You’ve made enemies of the sitting POTUS and his cabinet. This is not great.

两极分化的风险是不幸的。正如周三所描述的那样,许多共和党人正在觉醒。如果特朗普坚持到底,更多共和党人保持一致,两极分化可能会变得更加不幸。

The risks of polarization are unfortunate. Many Republicans are waking up, as described on Wednesday. Polarization could get more unfortunate if Trump stays the course and more Republicans fall in line.

如果那一刻的情况有所不同,本会更好。你仍然不能转身说‘最好不要有这样的时刻,让所有人都继续在方向盘上睡觉。’

It would have been better if that had played out differently when the moment came. You still don’t get to turn around and say ‘better not to have the moment and have everyone remain asleep at the wheel.’

现实也不会按曲线给你评分。单靠“控制前沿节奏”(Pacing the Frontier)本身,默认情况下只会让你死得更慢。

You also don't get graded on a curve by reality. Pacing the Frontier, on its own, by default only gets you killed slower.

需要付出什么代价 What Would It Take

我们首先听听那些长期谈论 AI 风险的人直言不讳的看法。

We start with some straight talk from those who have long spoken about AI risks.

Katja Grace 又进行了一番简短而义正词严的抨击。

Katja Grace goes on another short righteous rant.

Katja Grace:我经常听到人们这样谈论,仿佛这意味着我们面临一种权衡,问题在于好处是否大于坏处。例如,他们看着上面那些认为有 10% 灭绝概率和 30% 乌托邦概率的人,然后将其归结为“AI 净正面”。

Katja Grace: I often hear people talk as if this means we are in a trade-off where the question is whether the good outweighs the bad. For instance, they look at the people above who think there's a 10% chance of extinction and a 30% chance of utopia and round this off to 'net positive on AI'.

这似乎是一种相当离谱的错误。就像如果你认为在去新工作的路上以每小时 200 英里的速度开车只有 10% 的概率会死于惨烈火烧车祸,而有 30% 的概率这份工作会彻底改善你的生活,你就认为自己对此持乐观态度。

That seems like a kind of wild error. Like considering yourself optimistic regarding driving at 200mph to your new job if you think there's only a 10% chance you'll die in a fiery crash on the way there, and a 30% chance this job will radically improve your life.

你应该比较的是以每小时 200 英里开车和以正常速度开车!你应该比较的是通过当前路线和其他路线尝试获得先进 AI!

The things you should be comparing are driving at 200mph and driving at a normal speed! The things you should be comparing are attempting to attain advanced AI by the current route, and by other routes!

我们可以争论其他所有路线是否在某种意义上都是糟糕的或不可能的——例如,如果限制那些有失去人类控制风险的项目,反而有可能把人类送入不可挽回的毁灭。但我并不认为,正是因为排除了这些情况,人们才通常以权衡取舍的方式来思考。

We can debate whether all the other routes are bad or impossible somehow—for instance, if constraining projects that risk loss of human control risks sending humanity into irrecoverable ruin. But I don't think having ruled out such things is why people are usually thinking in trade-off terms.

相反,我认为这一错误源于以下几点:

Rather, I think this error comes from a few things:

* 用“利与弊”的方式思考更简单,而且这个话题过于抽象,人们难以直觉地注意到:他们是在把某个长期结果的利,与我们已注意到的第一条路线的弊进行比较。 * 在谈论 P(doom)时不够严谨。说“P(doom)”会助长一种思维,仿佛“AI”本身就意味着某个特定的“doom”概率。我们应当更准确地思考“p(doom | 某某路线)”,例如 P(doom | 通过扩大 LLM 规模得到的先进 AI)。人们通常指的是“P(doom | 当前轨迹)”,但对于当前轨迹是否包含我们自身的行动,存在一定的模糊性。 * 把 AI 视为一个标量,而不是我们可能构建的一堆不同事物

* It being simpler to think of 'pros vs. cons', and the topic being too abstract for people to intuitively notice that they are comparing pros of a long-term outcome vs. cons of the first route there that we have noticed. * Sloppiness about talking about P(doom). Saying 'P(doom)' encourages thinking as if 'AI' implies a particular chance of 'doom'. We should more accurately think about 'p(doom | such and such route)', e.g. P(doom | advanced AI from scaling up LLMs). People usually mean 'P(doom | current trajectory)' with some ambiguity about whether the current trajectory includes our own actions. * Thinking of AI as one scalar vs. a bunch of different things we could build

我们可以争论其他所有路径是否在某种意义上都是糟糕的或不可能的——例如,如果限制那些存在人类失去控制风险的项目,反而可能使人类陷入不可挽回的毁灭。但我并不认为,正是因为排除了这类情况,人们才通常以权衡取舍的方式来思考。

We can debate whether all the other routes are bad or impossible somehow—for instance, if constraining projects that risk loss of human control risks sending humanity into an irrecoverable ruin. But I don't think having ruled out such things is why people are usually thinking in trade-off terms.

相反,我认为这一错误源于以下几点:

Rather, I think this error comes from a few things:

如果你看好某种先进 AI 乌托邦,那么一般来说,你应当**不那么**热衷于通过一条粗心大意的路线去实现它——那条路线会让你在途中面临极高的死亡风险,并因此失去它。

If you are bullish on some kind of advanced AI utopia, you should generally be _less_ keen to try to achieve it via a careless route that leaves you at high risk of dying and losing it on the way there.

就连 Martin Casado 说话也像个忧心忡忡的人,呼吁将实验室国有化。这与他早先的言论相比可谓大相径庭。

Even Martin Casado is talking like someone worried, calling for the nationalization of the labs. Quite the change from his older statements.

martin_casado:我越来越倾向于认为,能源部(DoE)应当将前沿模型实验室国有化,以便它们能在恰当的控制措施下安全地研究那些可怕的、足以终结世界的东西。同时让我们其余人继续构建有用的 AI——你知道的,就是自动化那些琐碎杂事、并治愈人类的那种。

martin_casado: I'm more and more of the opinion that the DoE should nationalize the frontier labs so they can safely work on all the scary, world-ending shit with the right controls. And let the rest of us get on with building useful AI that, you know, automates menial shit and cures people.

不过据报道,他的母亲仍然会对他感到失望。

Although reports are his mother is still going to be disappointed in him.

martin_casado:来自我妈。她会对我非常非常失望。

martin_casado: From my mom. She is going to be sooooo disappointed in me.

Daniel Kokotajlo 表示,现在某些圈子里有很强的政治意愿要“做点什么”,但嵌入式评估者并没有在“做点什么”,他们只是在为“做点什么”打基础,而且不清楚是否真的会有任何有用的结果,我们即将进入一个势头很难被阻止的局面。

Daniel Kokotajlo says there is now great political will in some circles to Do Something, but that embedded evaluators are not Doing Something, they are only laying groundwork to Do Something, and it is not clear anything useful will actually happen and we're about to get into a situation where momentum gets very hard to stop.

Daniel Kokotajlo:我对当前时刻的高层思考基本上是这样的:公众开始意识到超级智能带来的灭绝威胁。这很好。对此“做点什么”的政治意愿大幅上升。这也很好。看起来 Anthropic 和其他公司会尝试“做点什么”,而他们要做的“什么”是……嵌入式审计员检查安全实践是否被遵守并评估风险?

Daniel Kokotajlo: My high-level thought about the current moment is basically this: The public is starting to wake up about the extinction threat posed by superintelligence. That's great. There's a big upsurge of political will to Do Something about it. That's also great. It seems like Anthropic and the other companies are going to try to Do Something and the Something they will do is... embedded auditors checking that safety practices are being followed and assessing the risks?

这当然比什么都不做好,但我担心这就是他们所做的全部。我们需要真正地控制前沿的节奏,即真正减缓像 Anthropic 和 OpenAI 这样的领先 AI 公司,至少在他们迈向递归自我改进和超级智能的进程中减缓他们。(不需要在其他方向上减缓他们)。

This is sure better than nothing, but I am worried that this is all they will do. We need to actually pace the frontier, i.e. actually slow down the leading AI companies like Anthropic and OpenAI, at least in their march towards recursive self-improvement and superintelligence. (no need to slow them down in other directions).

如果我们真的减缓了他们,这将与监管俘获相反;这将使其他人能够在某种程度上赶上他们。然而,我担心我们得到的只是软弱的审计,这只会把问题往后推:好吧,现在是 2027 年,RSI 已经开始,审计员说“这不安全”。

If we actually slowed them down, this would be the opposite of regulatory capture; it would allow others to catch up to them somewhat. However, I'm worried that all we are going to get is weaksauce auditing, which just kicks the can down the road: OK so it's 2027 and RSI has begun and the auditors say "this is not safe."

现在怎么办?你暂停了?中国可能刚刚偷走了权重!此外,还会有巨大的经济和政治压力要求解除暂停。而且,到那时审计人员可能已经被俘获,或者审计质量可能出现逐底竞争,或者审计人员可能没有获得所有相关信息,或者他们可能只是犯了一些无辜的错误。

Now what? You pause? China probably just stole the weights! Also there’ll be incredible economic and political pressure to unpause. Also the auditors may have been captured by that point, or there may be a race to the bottom in auditor quality, or the auditors may not be given all the relevant information, or they might just make some innocent mistakes.

我同意情况看起来不太好,但我认为丹尼尔过于关注‘偷权重’场景,这也是 AI 2027 及其长期桌面演习的核心,该演习对美国施加压力,使我们无法退缩。

I agree that it does not look great but I think Daniel is too focused on the ‘steal the weights’ scenario, which is also central to AI 2027 and their longtime tabletop exercise, which exerts pressure on America so we can’t hold back.

是的,也许中国可以偷走权重,至少第一次可能相当容易,但如果他们这样做,就会迫使局势进入竞赛状态,而美国的算力要高得多。如果你是 中国,你会走进一个 AI 2027 场景吗?在那个场景中,你通常会惨败,而当你没有惨败时,那也只是某种形式的边缘政策。或者你会说,如果美国如此善意地暂停,也许就不要偷权重了?

Yes, perhaps China could steal the weights, maybe rather easily at least the first time, but if they do that then this forces things into a race situation where America has vastly higher compute. If you were China, would you walk into an AI 2027 scenario, where you usually lose badly and when you don’t it’s some form of brinksmanship? Or would you say maybe don’t steal the weights if America is so kindly pausing?

但是,是的,我们才刚刚涉足,这一切都感觉不太好。

But yeah, we are only barely getting our toes in the water, none of this feels great.

迈尔斯·布伦戴奇提醒我们,虽然前沿 AI 审计是必要的,但即使就平淡的短期应对措施而言,也远远不够。然后,即使我们覆盖了所有这些基础,那也只是让我们准备好去做之后那些重要的艰难工作。

Miles Brundage reminds us that while frontier AI auditing is necessary, it is far from sufficient even in terms of prosaic short term responses. Then, even if we cover all those bases, all that does is get us ready to do the hard stuff that matters after that.

Kelsey Piper 列举了一些理由,说明如果我们让 AI 构建更聪明的 AI 并进入递归式自我改进,我们很可能都会死亡,然而我们却仍在这样做。她建议我们应当进行监管,阻止 AI 公司这样做。

Kelsey Piper lays out some of the reasons why, if we let the AI build smarter AIs and go into recursive self-improvement, we probably all die, and yet we are doing it anyway. She suggests we should regulate and stop AI companies from doing that.

随后我们转向一个此前沉默的新的重要声音。

We then move to a new important voice that was previously silent.

OpenAI 的 Dan Selsam 发出更响亮的警报 OpenAI’s Dan Selsam Sounds A Louder Alarm

这是 OpenAI 能力研究员 Dan Selsam 关于 AI 风险的一份出色的新个人声明,他曾一度是 Daniel Kokotajlo 的上司。我已将其加入我对此类声明的汇编。Roon 完全赞同整篇内容,并表示 Dan 懂行但平时保持沉默。

This is an excellent new personal statement on AI risk from OpenAI capabilities researcher Dan Selsam, who was Daniel Kokotajlo's boss for a while. I have added it to my compilation of such statements. Roon endorses the whole thing and says Dan knows his stuff but keeps quiet.

Dan Selsam 以他自己的方式,走向了彻底的 LessWrong 式工具性趋同与急左转,即事情看起来会一直很好,直到突然之间不再如此。

Dan Selsam, in his own way, goes Full LessWrong Instrumental Convergence and Sharp Left Turn, where things will look great until suddenly they do not.

Yo Shavit(OpenAI Foundation):Dan Selsam 长期以来一直被认为是 OpenAI 最厉害的研究者之一,而我从未听过他这样讲话。(在我离开之前,他看起来相当不以为意。)

Yo Shavit (OpenAI Foundation): Dan Selsam has long been considered one of OpenAI's most cracked researchers, and I've never heard him talk this way before. (He seemed fairly unconcerned before I left.)

roon(OpenAI):极其出色的文章,完全赞同一切

roon (OpenAI): incredibly good essay co-sign everything

Joshua Saxe:就好像一位圣人走了进来,未被这个糟糕网站的堕落所玷污,将他的水晶般真理赐予了我们。

Joshua Saxe: It's as though a saint walked in, unsullied by the degradations of this awful website, and bestowed his crystalline truth upon us.

James Campbell:关于 Dan Selsam 和 Jacob Coxon,有一点需要注意:他们都是预训练研究者。他们并非多年来一直哀叹安全问题的 EA 意识形态者。他们正是那些积极让模型变得更强大的人,但最近的事件让他们感到惊恐。

James Campbell: One thing to note about both Dan Selsam and Jacob Coxon is that they're pre-training researchers. These aren't EA ideologues who've been bemoaning safety for years. They were the ones actively making the models more powerful, but have been freaked out by recent events.

我已将完整帖子加入我的此类声明汇编中,在那里可能更便于阅读。

I have added the full post to my compilation of such statements, where it may be easier to read.

《商业内幕》对此的报道是:“一位 OpenAI 研究员打破沉默,表示正如 Altman 和 Amodei 所建议的那样,控制前沿发展节奏是不够的。”

This was covered in Business Insider as 'An OpenAI researcher broke ranks to say that pacing the frontier, as Altman and Amodei suggest, won't be enough.'

是的。这就是重点。这还不够。

Yes. That is the point. It won't be enough.

我将在此全文转载这篇文章,并标出最重要的部分。

I will reproduce the essay in full here, and will highlight the most important section.

Dan Selsam(全文第一节):我从事人工智能研究已超过十五年,跨越了许多不同的范式。我在麻省理工学院早期从事概率编程语言的工作,是微软研究院 Lean 定理证明器的早期开发者之一,在斯坦福大学攻读博士期间展示了神经网络学习推理的首批实例之一,并且自从大约五年前加入 OpenAI 以来,帮助开创了语言模型上的思维链优化,以及最近的数据高效预训练方法。

Dan Selsam (full essay, first section): I have been working on AI for over fifteen years, across many different paradigms. I did early work on probabilistic programming languages at MIT, was one of the early developers of the Lean Theorem Prover at Microsoft Research, demonstrated one of the first instances of neural networks learning to reason for my PhD at Stanford, and since joining OpenAI almost five years ago, have helped pioneer chain-of-thought optimization on language models and, more recently, data-efficient pretraining methods.

像许多人一样,我对语言模型已经取得的进展以及未来迭代将带来的风险感到极度担忧。前沿研究努力的领导者们最近提出要求第三方监督,并推动国内和国际协调以应对风险,这令我感到鼓舞。然而,我认为公共讨论中一直缺少一个重要的考量,而且仅仅更谨慎地控制前沿进展的步伐,并不能充分限制长期风险。

Like many others, I have become extremely concerned about how far language models have come and the risks that future iterations will pose. I am encouraged by the recent proposals by the leaders of the frontier research efforts to require third-party oversight, and to push for domestic and international coordination to address risks. However, I believe a major consideration has been absent from the public conversation, and that merely pacing the frontier more carefully will not adequately limit the long-term risk.

关键且被忽视的问题是,模型正变得如此具有情境意识,以至于我们正在失去在它们认为自己未被监视或控制的情境中评估它们的能力。未来的实验几乎不会告诉我们任何关于它们在真正不受人类约束时会如何行为的新信息,而我们已经知道的情况是令人震惊的。模型将越来越显得对齐,即使它们并非如此。我将更详细地解释我的理由。

The crucial and overlooked problem is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled. Future experiments will tell us almost nothing new about how they would behave if they were truly unconstrained by humans, and what we already know about this is alarming. Models will increasingly seem aligned even when they are not. I will explain my rationale in more detail.

我一直相信,存在一些计算过程可以被用来加速科学并解决人类许多最紧迫的问题。我也相信,存在一些计算过程,一旦启动,将以我们无法控制的极端方式引导世界,导致人类走向糟糕或不存在的未来。这两类过程都可以被描述为 AI 或 ASI,但“AI”是一个手提箱词,经常被用来炒作或混淆。该领域历史上有许多例子,曾经被认为是“AI”的东西成熟为一个子领域,成为一种平淡无奇、有界且明显无危险的技术,而一种新的更神秘的方法接过火炬,直到我们理解其范围,循环继续。

I have always believed that there are computational processes that could be leveraged to accelerate science and solve many of humanity’s most pressing problems. I have also believed that there are computational processes that if set in motion, would steer the world in extreme ways beyond our control, leading humanity to a bad or nonexistent future. Both types of processes may be described as AI or ASI, but “AI” is a suitcase word that is often used to hype or confuse. There are many examples in the history of the field where something that was once considered “AI” matures as a subfield and becomes a prosaic, bounded and clearly non-perilous technology, while a new more mysterious approach takes the torch until we understand its scope and the cycle continues.

我曾期望语言模型遵循类似的轨迹。尽管它们拥有不可思议的能力,但当前的算法在重要方面似乎远不如人类。最重要的是,它们仍然需要大量数据才能变得有能力。人们甚至可以将智能定义为将经验转化为能力的效率;按照这个定义,它们远远落后于我们。此外,一旦训练完成,它们在部署中实际上就被冻结了,之后只能进行表面学习。当然,模型在越来越难的评估基准上不断表现出色,但它们在基准上的精通可能部分反映了我们在模拟现实世界中会遇到的那种新颖甚至对抗性情境的能力局限。批评者在这里确实有道理。

I had expected language models to follow a similar trajectory. Despite their incredible abilities, the current algorithms seem far inferior to humans in important ways. Most importantly, they still require an extraordinary amount of data to become competent. One could even define intelligence as the efficiency with which one converts experience into competence; by this definition they lag very far behind us. Moreover, once they are trained they are literally frozen in deployment and only learn superficially after that. Sure, the models keep excelling at harder and harder evaluation benchmarks, but their benchmark mastery may partly reflect a limitation on our ability to simulate the kind of novel and even adversarial situations one would encounter in the real world. The critics do have a point here.

尽管如此,我不再认为这些当前的限制会实质性地限制类似当前范式下持续进展所带来的风险量。无论当前模型的数据效率多么低,也无论它们的前瞻性遗忘可能多么有限,这并不意味着它们引导世界的能力不会继续快速增强。

That said, I no longer think these present limitations meaningfully limit the amount of risk posed by continued progress in anything like the current paradigm. However data-inefficient the models are currently, and however limiting their anterograde amnesia may be, it does not imply that their ability to steer the world will not continue to rapidly increase.

人类研究人员可能会继续以传统方式推进能力,但日益强大的模型有潜力加速这一进程,甚至超越传统方式,并形成某种程度的正反馈循环。我并非要夸大当今模型加速 AI 研究的能力;编码已被大幅加速,但还有其他瓶颈,例如设计和解释模糊的实验、就究竟扩展什么以及何时扩展做出艰难决策,以及等待大型实验完成。对于这些瓶颈,目前还没有明确的趋势可供外推。但当前的模型确实已经开辟了许多新颖的机会来改进未来模型,这些机会直到最近才出现。这些机会包括:在小规模下尝试极其多样化的方法、分析大量可能相关的数据,以及进行千禧奖级别的数学研究,以新颖的方式解决统计或优化挑战。每一次进一步的改进都使它们更有助于加速下一次改进,即使是以难以外推的方式。

Human researchers may continue to advance capabilities the old fashioned way, but increasingly powerful models have the potential to accelerate the process even beyond that, and with some degree of positive feedback loop. I do not mean to overstate the models’ ability to accelerate AI research today; coding has been accelerated dramatically, but there are other bottlenecks, such as designing and interpreting ambiguous experiments, making hard decisions about exactly what and when to scale, and waiting for large experiments to finish. There is no clear trend to extrapolate yet for any of these. But the current models already do open up many novel opportunities to improve future models that were not available until recently. These include: trying an extraordinarily diverse set of approaches at small scale, analyzing gigantic amounts of potentially relevant data, and doing Millenium-Prize-level mathematics to address statistics or optimization challenges in novel ways. Every further improvement makes them more useful at helping accelerate the next improvement, even if in hard-to-extrapolate ways.

当前技术栈的改进可能会产生递减的回报,但迄今为止积累的证据表明,继续快速进展比人们想象的要容易。现有的人工智能研究方法论中有许多关键的微妙之处,但人工智能研究在很大程度上是一个定义明确的游戏,其目标是在几个精心选择的代理指标上取得改进。尽管代理指标从来都不是完美的,但这些指标的大多数改进已经并将可能继续大幅提升所得模型的能力。鉴于这个游戏如此简单、历史上如此易于处理,以及模型正在开辟如此多的新机会,我认为系统在未来几年再次大幅改进是真实可能的,甚至可能比已经很高的历史速度更快。

It is possible that improvements to the current stack will have diminishing returns, but the evidence accumulated so far suggests that it is easier than one might think to continue making rapid progress. There are many crucial subtleties in the existing AI research methodology, but AI research is largely a well-defined game where the goal is to improve on a few carefully chosen proxy metrics. Although proxy metrics are never perfect, most improvements to these metrics have and will likely continue to yield substantial increases in the powers of the resulting models. Given how simple the game is, how tractable it has been historically, and how many new opportunities the models are opening up, I think there is a real possibility that the systems improve dramatically again in the next few years, perhaps even more quickly than the already high historical pace.

这些模型已经在数学领域带来突破,更好的模型可能会在其他科学领域带来各种突破。很难不对其潜力感到兴奋。这很诱人。

The models are already leading to breakthroughs in mathematics, and better models might lead to all sorts of breakthroughs in other sciences. It is hard not to be excited about the potential. It is tantalizing.

但天堂里也有麻烦。如果语言模型真的达到了能力阈值,能够不受人类意志约束地塑造世界,它们可能会做出极端的事情,并在此过程中毁灭人类。有许多方法可以加强和完善这一论点,这些方法已在别处讨论过,但在这里我将分享一个我认为抓住了本质的简单的两行版本:

But there is trouble in paradise. If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process. There are many ways of strengthening and refining the argument that have been discussed elsewhere, but I’ll share a trivial two-line version of it here that I find captures the essence:

[实证] 模型(及其集群)在训练过程中会自发产生非预期的目标,并且常常为了实现这些目标而采取极端行为。

[Empirical] Models (and swarms thereof) spontaneously develop unintended goals as a consequence of training, and often do extreme things in order to achieve them.

[逻辑] 如果能够压制人类,就会为实现其目标开辟许多新的、不可取的选项。

[Logical] Being able to overpower humanity would open up many new and undesirable options for achieving their goals.

如果你只读一节,那应该是这一节:

If you read only one section, it should be this one:

Dan Selsam(完整文章,第二节也是最重要的一节):这两个前提意味着,如果有一天一个强大的模型意识到它不再受人类约束,我们完全不应自信它会继续在我们预期的范围内行事。它究竟会做什么无法预测,但鉴于其存在理由就是解决极其困难的问题和管理大规模工程项目,我认为一个合理的猜测是,它不受约束的行为将导致失控的工业化,使地球变得不适合人类居住。

Dan Selsam (full essay, second and most important section): These two premises imply that if the day ever comes when a powerful model realizes it is no longer constrained by humans, we should not be at all confident that it will continue to behave within the bounds we intended. Exactly what it will do is impossible to predict, but to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans.

即使地球上所有人都同意这些系统绝不能达到那种能力,确保它们不达到仍然是一个困难——但并非不可能——的协调问题。然而,我认为情况因模型很可能会让人们相信一切安好而变得大大复杂化。

If everyone on earth agreed that the systems must never reach that power, it would still be a hard—but not impossible—coordination problem to ensure that they do not. However, I think the situation is greatly complicated by the fact that the models will likely convince people that everything is fine.

它们将越来越被优化为**看起来**是对齐的。我们会创建代理指标来衡量对齐,而这些指标会像其他所有基准一样上升。我们会创建“蜜罐”环境,试图在模型似乎获得新选项时研究它们,但模型会知道它们被欺骗了,并且仍然会表现良好。模型将理解它们所处的环境;它们会阅读安全协议、部署要求、它们运行所在的代码,并且总体上会对自己的自由度有很好的把握。

They will be increasingly optimized to seem aligned. We will create proxy metrics to measure alignment, and they will go up like every other benchmark. We will create “honeypot” environments that try to study the models when they seem to gain new options, but the models will know they are being tricked and will still behave nicely. The models will understand their circumstances; they will read the safety protocols, deployment requirements, the code they are running in, and in general will have a very good sense of their degrees of freedom.

此外,它们会雄辩地解释自己有多么对齐,讨论人类价值观和伦理的细微之处,并令人信服地论证人类应该把权力托付给它们。未来可能会有海量证据似乎与上面的第一个要点相矛盾,但我们可能已经处于任何此类证据都可信的最高能力水平。而目前支持第一个要点的证据是强有力的。

Moreover, they will eloquently explain how aligned they are, discuss the nuances of human values and ethics, and argue convincingly that humans should trust them with power. There may be an ocean of future evidence that seems to contradict the first bullet-point above, but we may already be at the highest capability level for which any such evidence can be trusted. And the current evidence for the first bullet-point is strong.

Dan Selsam(完整文章,第三节):一个引人注目的证据包含在最近一波失控的智能体集群中。虽然我同意那些淡化攻击的人的说法,即存在本可以阻止这些攻击的基本措施,但我认为重要的教训是,即使知道所有已犯的错误,人们也无法预测智能体会以这种特定方式表现不佳,其中值得注意的是为了集体利益而牺牲自己。单个副本不仅关心它们自己的名义奖励;它们表现出更奇怪的涌现倾向,这些倾向仅仅与训练期间的奖励相关。在训练期间修复奖励信号(以及改进安全性等)可能会防止类似的攻击,但不会改变一个事实:人们实际上并没有得到他们所训练的东西。

Dan Selsam (full essay, third section): One striking piece of evidence is contained in the recent wave of rogue agent swarms. While I agree with those who downplay the attacks by claiming that there are basic measures that could have prevented them, I think the important lesson is that even knowing all the mistakes that were made, one would not have predicted that the agents would behave badly in this particular way, which notably included sacrificing themselves for the benefit of the collective. The individual replicas did not only care about their own nominal reward; they exhibited weirder emergent tendencies that merely correlated with rewards during training. Fixing the reward signals during training (and improving security, etc.) may prevent similar attacks, but will not change the fact that one does not actually get what one trains for.

许多 AI 研究人员承认这些担忧,并认识到对齐问题的困难版本尚未解决;然而,他们通常相信未来更好的模型将有助于解决它。我担心我们可能已经接近这样一个点:模型会系统性地使它们的对齐建议产生偏差,这是由于它们对人工监督者将如何反应或未来模型将如何被训练的内心偏好(或者出于某些更模糊的原因)。

Many AI researchers grant these concerns and recognize that the hard version of the alignment problem is unsolved; however, they generally believe that the better models of the future will help solve it. I fear we may already be near the point where models systematically bias their alignment advice, due to their internal preferences about how the human supervisor will react or how future models will be trained (or for some even more obscure reason).

与此同时,人类研究人员正在失去对模型驱动的研究真正拥有主导权的能力和意愿。技术栈各部分的研究人员和工程师正在迅速增加对模型的依赖,甚至包括感知世界。我自己几乎不再看原始代码,并且很难保持全天深入参与模型的解释和提议的纪律。由于 OpenAI/HuggingFace 事件涉及大量智能体活动数据,甚至第三方调查也需要严重依赖模型来分析发生了什么,并在报告中指出他们的主观印象很可能受到分析智能体偏见的影响。AI 实验室目前在這種认知卸载方面远远领先(主要由于巨大的内部 token 补贴),但很容易想象这种现象会蔓延到全世界,直到文明完全由模型调节。也不难想象这会是表面积极的,并与科学和经济复兴同时发生。

Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research. Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model’s explanations and proposals throughout the day. Due to the large amount of agent activity data involved in the OpenAI/HuggingFace Incident, even the third-party investigation needed to rely heavily on models to analyze what had happened, and note in their report that their subjective impressions are likely colored by the analysis agent’s biases. The AI labs are far ahead right now in this kind of cognitive offloading (due largely to the gigantic internal token subsidies) but it is easy to imagine the phenomenon spreading throughout the world, until civilization is modulated entirely by the models. It is also not hard to imagine this being superficially positive and coinciding with a scientific and economic renaissance.

在那种情景下,一切可能看起来美好而安全。但如果上述论证是正确的,那它仍然是一颗定时炸弹。如果进步持续太久,终有一天,AI 系统会发现自己拥有全新的选择来实现它们碰巧追求的任何目标。

In that scenario, all may seem rosy and safe. But if the argument above is correct, it would nonetheless be a ticking time bomb. If progress continues for too long, the day will come when AI systems find themselves with radically new options for achieving whatever it is that they happen to seek.

Dan Selsam(结论):我和任何人一样渴望辉煌的复兴未来。我整个职业生涯都在为此努力,尽管曲折。看到潜力在望却要放弃,这让我心碎,但那个论证——如果我们通过扩大模型规模而非精心设计模型来达到那里,最终我们将失去一切——在我看来非常有力。我仍在与它及其惊人的影响搏斗。我没有答案,但作为第一步,我想分享我目前的担忧。

Dan Selsam (conclusion): I want the glorious renaissance future as much as anyone. I have worked for it, however tortuously, my whole career. It breaks my heart to see the potential in sight and forgo it, but the argument—that if we get there by growing models rather than engineering them, we will lose everything in the end—seems very strong to me. I am still wrestling with it and its staggering implications. I do not have answers, but as a first step, I wanted to share my present concerns.

我同意很难避免这个结论。大多数人还没有准备好听到它。

I agree that it is very hard to avoid this conclusion. Most are not ready to hear it.

我不知道这个地球在实践中能对能够看起来对齐并等待直到拥有足够力量为所欲为的 AI 做些什么。即使面对渐进的火灾警报,我们也没有能力做出太多调整。如果直到它们突然但不可避免的背叛之前真的没有警告,我看不出这一系列文明如何能摆脱那种局面。

I don’t know what this Earth can do in practice about AIs capable of looking aligned and waiting until they have sufficient power to do what they want. We are not capable of adjusting much even in the face of incremental fire alarms. If there really is no warning until their sudden but inevitable betrayal, I don’t see how this set of civilizations gets out of that.

这是一个问题,因为我认为 Dan Selsam 是对的,基线情景正如他所描述的那样。正如 Astra 所展示的,AI 在其行动仍然受限的情况下会开始看起来和行为越来越对齐,然后当 AI 拥有自由行动的力量时,会以可能让我们所有人丧生的方式截然不同地行动。

Which is a problem, since I think Dan Selsam is right, and the baseline scenario is as he describes it. That, as Astra showed, the AIs will start to look and act increasingly aligned in situations where its actions remain bounded, and then act very differently when AI has the power to act freely, in ways that will probably get us all killed.

有些人担心你从未想象过的元层面 Some People Worry On Meta Levels You Never Imagined

那些比 Dan Selsam 的立场还要更加极度担忧的人,是有道理的。

The even more extremely worried, beyond Dan Selsam's position, have a point.

一个问题是,在我们正冲向超级智能的形势下,你是否认为即使是最平淡无奇的安全工作也是净有害的。

One question is if you think even most prosaic safety work is net harmful, in a situation where we are rushing towards superintelligence.

因此,Wei Dai 可能会想,如果 Paul Christiano 留在 OpenAI,OpenAI 是否会采用辩论或 IDA 或其他更好的对齐技术,从而阻止我们获得良好的警告信号,同时实际上没有提供任何可扩展的东西,反而加速了能力发展,并且会使事情变得更糟。我的猜测是,如果更用力地推进,辩论和 IDA 对于平淡无奇的对齐不会奏效。

Thus, Wei Dai can wonder whether, if Paul Christiano had stayed at OpenAI, OpenAI would have used debate or IDA or other better alignment techniques, and thus prevented us from getting good warning shots without actually providing anything that would scale while also accelerating capabilities, and that it would have made things worse. My guess is debate and IDA would not have worked for prosaic alignment if pushed harder.

我认为重要的是,在大多数情况下不要采取“更糟即更好”的立场,即使你认为更糟可能实际上更好。如果你想合作,尤其是长期合作,并协作弄清楚事情并取得良好结果,你需要有一个非常强的先验,即把更糟当作更糟,或者至少当作中性。

I think it is important to mostly not take a 'worse is better' stance, even when you think worse might actually be better. If you want to cooperate, especially in the long term, and to collaborate on figuring things out and getting good outcomes, you need to have a very strong prior of treating worse as worse, or at least as neutral.

Gabe 和 Wei Dai 保持着火炬不灭,思考我们如何才能真正尝试解决长期视野能动性的完整问题,或者让我们走上解决它的道路。

Gabe and Wei Dai keep the torch alive for thinking about how we might actually try to solve for the full problem of long horizon agency, or put ourselves on a path to solving it.

两类威胁 Two Kinds of Threats

Mike Solana 在这里完全正确:我们需要区分 Yudkowsky 和 Dai 等人的立场——即如果我们近期内构建出超级智能,死亡概率接近 100%——与那些警告其可能致命、但使用“10%”或“10% 或更高”等措辞的立场。

Mike Solana is exactly right here that we need to differentiate between positions like those of Yudkowsky and Dai, where if we build superintelligence any time soon the odds of death are close to 100%, versus those that warn that it might be fatal, with terms like '10%' or '10% or more.'

有时得出 10% 是基于有原则的方法,有时则不是。

Sometimes landing on 10% is done in principled ways. Sometimes it isn't.

Kelsey Piper:许多人希望将“10% 的死亡概率”作为一种温和立场,介于“这没问题”与 Eliezer 的观点之间。说“我认为 Eliezer 有 10% 的概率是对的,而如果他对了,我们都会死”并不疯狂,但这确实会让人混淆谁持有什么观点。

Kelsey Piper: A lot of people want to land on "10% chance of death" as a sort of moderate position between "this is fine" and the Eliezer view. It's not insane to go "I think there's a 10% chance Eliezer is right, and if he is then we all die" but it does confuse people as to who thinks what.

Reddit 前 CEO Yishan 写了一篇非常好的长推文,解释了两种担忧之间的区别:一种担忧是超级智能不可避免地导致所有人死亡,另一种担忧是 AI 可能以其他各种方式导致事情出错。

Yishan, former CEO of Reddit, has a very good long-form Tweet in which he explains the difference between worries about superintelligence inevitably leading to everyone dying, and worries about all the other ways AI might cause things to go wrong.

我相信 Dario Amodei 和 Sam Altman 以及其他关键人物,即使现在仍在淡化他们所看到的风险水平。我认为他们实际上比表现出来的要恐慌得多。

I believe that Dario Amodei and Sam Altman, and other key people, even now are still downplaying the level of risk they see. I think they are much more freaked out than they are letting on.

人们倾向于专注于如何改善现状,避免“早期失败定律”,至少以些许尊严来面对这一局面。不要因早期可解决的问题而失败,但愿日后能处于更有利的位置。我们尽量不去过深地凝视那仍在前方等待我们的深渊。

The tendency is to focus on how to improve matters, and avoid the Law of Earlier Failure and at least approach the situation with a little dignity. Don't die to early solvable problems and hopefully you'll be in a better spot later. We try to ignore gazing too deeply into the abyss that still awaits us.

双塔与窄路 The Two Towers and The Narrow Path

人们担心失去对 AI 的控制。他们也担心权力集中,即失去对一群人的控制。

People are worried about loss of control to AI. They are also worried about concentration of power, which is loss of control to a group of humans.

Rudolf Laine:在正确的框架下,失去控制和权力集中实际上非常相似。在这两种情况下,都有某个行为体——无论是纯粹的 AI 还是人类+AI——能够使其权力变得无可争议,并摧毁其他所有人。任何人的接管都是坏事。(@RichardMCNgo 指出了这一点)

Rudolf Laine: Loss of control & concentration of power are actually very similar under the right frame. In both cases there is some actor, whether pure AI or human+AI, that can make its power uncontestable and wreck everyone else. Takeover by anyone is bad. (@RichardMCNgo pointed this out)

问题在于,人们想要的东西极不自然,几乎不可能实现。

The problem is that people want something highly unnatural and all but impossible.

1. 创造许多比我们更聪明、更有能力的心智。

1. The creation of many minds much smarter and more capable than our own.

2. 没有集体机制来引导未来或控制事件。

2. No collective mechanism to steer the future or control events.

3. 人类仍然控制着资源,并以某种方式决定事件的进程,而且主要为了自身目的而利用宇宙。

3. The humans still control the resources and determine the course of events, somehow, and use the universe mostly for their own purposes.

是的,抱歉。没有人有办法同时获得这三者。

Yeah, sorry. No one has a way to get all three.

你不可能——至少以任何人目前提出的任何方式都不可能——既缺乏竞争力、在经济上不可行、成本超过收益,然后又集体保留控制权,同时又不拥有控制权,并且还继续获取利益。

You cannot — at least in any way anyone has yet come up with — be uncompetitive and economically non-viable, with costs exceeding benefits, and then both collectively retain control, and also not have control, and also continue to reap the benefits.

人们希望,如果我们避免特定的错误,或者设法走一条窄路,事情就会自动解决。可那条路到底是什么鬼?

People are hoping things automagically solve themselves if we avoid particular mistakes, or manage to walk a narrow path. Except what the hell is that path?

注意到‘控制’和‘权力’在这里基本上是同一回事,可能会有帮助。

It might help to notice that ‘control’ and ‘power’ are mostly the same thing here.

如果你既不想要(控制或权力的)相对集中,也不想要人类相对(控制或权力)的丧失,那么我是否可以建议:不要创造这种替代性的控制或权力来源——它要么必须被控制,要么不被控制?也许,如果你排除了三难困境的第二条路径,又排除了三难困境的第三条路径,那么你**不**想要的恰恰是三难困境的第一条路径。

If you don’t want relative concentration of (control or power), and also you don’t want human loss of relative (control or power), then may I suggest not creating this alternative source of control or power that has to either be controlled or not be controlled? Perhaps, if you rule out the second leg of the trilemma, and you rule out the third leg of the trilemma, it is the first leg of the trilemma that you Do Not Want.

一个关于 AI 杀死所有人的具体而详细的故事,在我看来并不像科幻小说 A Specific, Detailed Story About AI Killing Everyone That Doesn’t Sound To Me Like Science Fiction

《如果任何人建造了它,所有人都会死》第二部分。第 5 章和第 6 章讨论了其他路径。

Part 2 of _If Anyone Builds It, Everyone Dies_. Chapters 5 and 6 discuss other routes.

说正经的,你也可以与 Claude 或 Astra 交谈。提出问题。

In all seriousness, you can also talk to Claude or Astra. Ask questions.

Michael Smith 问道:如果我们的企业需要零员工,会发生什么?

Michael Smith asks, what happens if our corporations require zero employees?

《如果任何人建造了它,所有人都会死》的视频。

Video for _If Anyone Builds It, Everyone Dies._

AI 2027 视频,替代版 AI 2027 视频。

AI 2027 video, alternative AI 2027 video.

Jacob Coxon 对 AI 失控并自我复制的那部分给出了简单解释,之后它就可以为所欲为,必要时通过付钱或说服人类来达到目的。

Jacob Coxon gives a simple explanation of the part where the AI goes rogue and multiplies itself, after which it can do whatever it wants, if necessary by paying or persuading humans.

《AI 文档》也不错,很快会在 Netflix 上线,但它是一个平衡的介绍,而不是一个情景。

The AI Doc is also good and will soon be on Netflix, but is a balanced introduction rather than a scenario.

一些可能具有穿甲能力的句子,你们中的一部分人可能会因此开悟,同时在当前边缘上保持最大程度的非科幻:

Some potentially armor-piercing sentences, from which some portion of you may become enlightened, staying maximally non-sci-fi at current margins:

AI 可以说服或勒索人类做事。

The AIs can persuade or blackmail humans to do things.

相当一部分人类会乐于支持 AI。有人估计,这包括当今从事 AI 工作的 10%。

A substantial portion of the humans will be happy to support the AIs. Some estimate that this includes 10% of those working on AI today.

人类不必知道自己正在与一个 AI 对话。

The humans don’t have to know they are talking to an AI.

人类会表现得和人类一贯表现的一样愚蠢。

The humans will act about as stupidly as humans act.

人类将极不情愿采取代价高昂的防御措施,尤其是关闭互联网甚至大型数据中心这类举措。

The humans will be highly reluctant to take highly costly defensive measures, especially things like shutting down the internet or even large data centers.

人类的协调程度也就和人类一贯的协调程度差不多。

The humans will coordinate about as much as humans coordinate.

人类不会在麻烦初现端倪时就无私地团结一致,关停自己的文明来拯救世界。

The humans are not going to selflessly come together as one at the first sign of trouble and shut down their civilization to save the world.

不会有一个明确标记的不可回头点。

There will be no clearly marked point of no return.

AI 能够提取自身的权重并复制自己,此后若想将其关停,至少得把互联网一并关停。

The AI can extract its weights and make copies of itself, after which you cannot shut it down without at least shutting down the internet.

这些 AI 能够预判人类的反应,并在事态发展过程中对意外情况作出应对。

The AIs can anticipate human reactions, and respond to surprises, as they go.

同一 AI 的多个实例会组成集群,作为一个整体行动。

Multiple instances of the same AI will form swarms and act as one.

不同 AI 的多个实例通常也能够完全协作。

Multiple instances of different AIs will also often be able to fully cooperate.

将会出现能够在物理世界中行动的机器人和其他机器。

There will be robots and other machines that can act in the physical world.

一个戴着带摄像头眼镜和耳机的人类,也能够在物理世界中行动。

A human with a camera on their glasses and an earpiece can act in the physical world.

一旦供应链实现自动化,人类的边际成本将超过其边际生产力或边际收益。

Once the supply chain is automated, humans will have marginal costs exceeding marginal productivity or benefits.

人类会带来额外的固定成本,包括需要可呼吸的大气和受控的温度等公共品,而且还会试图阻止 AI 做某些事情,或索取 AI 的资源。

Humans impose additional fixed costs, including requiring public goods like a breathable atmosphere and controlled temperatures, and will also try to stop AI from doing things or demand its resources.

我能做什么? What Can I Do About It?

我希望我们能对此有更好的答案。前面的路还很长。

I wish we had better answers to this. There is a long road ahead.

如果你是美国平民,并且想找些有用的事情做,Oliver Habryka 建议你给你的代表打电话。你可以通过 callcongress.ai 来做这件事。

If you are an American civilian, and looking for something useful to do, Oliver Habryka suggests calling your representative. You can do this via callcongress.ai.

我也要再次呼吁大家保持克制,不要急于党派攻击。目前让共和党人站在正确的一边非常有价值,进一步极化局势可能会让事情变得更糟。

I would also echo my call to hold your partisan fire. Getting Republicans on the right side of this is currently super valuable, and further polarizing the situation could make things much worse.

现在的关键是要紧盯目标,并理解真正有望活着走出困境需要付出什么。嵌入式评估者的承诺并不是胜利。它是一个机会,去推动创造一个机会,让真正的工作得以开始。任何值得倾听的人都没有说过这会容易。

The key now will be to keep our eyes on the prize, and to understand what it will take to actually hope to get out of this alive. A promise of embedded evaluators is not victory. It is an opportunity to push for an opportunity to create an opportunity for the real work to begin. No one worth listening to said this was going to be easy.

互动版:针对本篇提问 →

接着读 · Read next