The Safety Culture at the Labs is Broken
打开互动全文版（逐段中英对照 + 针对本篇提问）→本文认为，领先人工智能实验室（尤其是 OpenAI）的安全文化已从根本上崩坏，越来越多的研究人员辞职和举报即为明证。文章以 David Robinson 的离职及其在 The Ezra Klein Show 上的言论为依据，指出即便是健全的日常安全文化，也不足以应对超级智能，因为对齐仍是一个未解决的科学问题，需要新的研究。文章分析了留下与高调离职的员工各自的战略逻辑，认为对多数人而言，离开并发出警告所带来的平均安全回报更高，并梳理了 Anthropic 近期报告的模型意外行为事件。作者最后总结：规章制度无法替代强大的安全文化以及规则背后的精神，当前实验室的承诺不足以应对未来的风险。
This article argues that the safety culture at leading AI labs, particularly OpenAI, is fundamentally broken, as evidenced by a growing wave of researcher resignations and whistleblowing. Drawing on David Robinson's departure and his remarks on The Ezra Klein Show, it contends that even a robust culture of prosaic safety would be insufficient for dealing with superintelligence, since alignment remains an unsolved scientific problem requiring new research. The piece examines the strategic logic of employees who stay versus those who loudly quit, suggesting that the average safety return is greater for those who leave and warn, and it surveys recent incidents of unintended model behavior reported by Anthropic. The author concludes that rules and regulations cannot substitute for a strong safety culture and the spirit behind the rules, and that current lab commitments are inadequate preparation for the risks ahead.
4. 实验室的平庸安全策略并不充分。
4. The Lab Strategy of Prosaic Safety is Insufficient.
9. 他们尚未准备好做出承诺。
9. They Are Not Ready To Make a Commitment.
David Robinson 成为最新一位从 OpenAI 辞职的研究员，并就业界完全不负责任的安全做法发出警告。他还参加了 The Ezra Klein Show 节目，这让你能充分了解他的立场。
David Robinson became the latest researcher to resign from OpenAI and sound the alarm about the industry’s utterly irresponsible approach to safety. He also went on The Ezra Klein Show, which gives you a rich idea of where he is coming from.
David Robinson 是 Astra 系统卡的主要作者。这有助于解释为什么他认为 OpenAI 的安全文化已经破裂。那张系统卡在尖叫着事情严重不对劲，只要人们愿意倾听。
David Robinson was the lead writer for the Astra system card. That could help explain why he felt the OpenAI safety culture was broken. That card is screaming that things are horribly wrong, for those with ears to hear.
David 甚至并不那么担心存在性风险。他的担忧是世俗的，而且他意识到，即使在这个层面上，OpenAI 的安全文化也已严重破裂。
David is not even that worried about existential risk. His worries are mundane, and he realizes that even on that level OpenAI’s safety culture is deeply broken.
一如既往，我们深深感谢他发出警告。我希望在未来的几个月和几年里，能在这些文章中多次写下他的名字。
As always, we deeply thank him for sounding the alarm. I hope to write his name in these posts many times over the coming months and years.
当你不得不以“举报 OpenAI 完全缺乏安全已经成了陈词滥调”来开头你那篇关于 OpenAI 缺乏安全的举报文章时，OpenAI 在缺乏安全方面可能确实有问题。
When you have to open your whistleblowing essay about the lack of safety at OpenAI saying it has become a cliche to blow the whistle on the utter lack of safety at OpenAI, there might be a problem with the lack of safety at OpenAI.
David Robinson：我意识到，我接下来要告诉你们的事情已经有些老生常谈了：我本周从 OpenAI 辞职了。我曾负责牵头撰写我们在每次重大发布时一同发布的安全报告。如今，我加入了一连串前同事的行列——他们来自 OpenAI 以及业界的其他领军企业——这些人已经认定，当前的道路是不可接受的。
David Robinson: What I'm about to tell you has, I realize, become something of a cliché: I resigned this week from OpenAI. I led the writing of the safety reports we published with each major launch. Now I'm joining a parade of former colleagues—at OpenAI and the industry's other leaders—who have decided that the current path is unacceptable.
我同意其他近期离职员工的看法：构建这项技术的公司远不够谨慎。但我认为，我们需要看得比具体规则或新法律更深一层。我们需要谈谈文化。
I agree with other recently departed staff that the companies building this technology aren't being nearly careful enough. But I believe that we need to look deeper than specific rules or new laws. We need to talk about culture.
我们必须审视规则，审视法律，也要审视文化。
We have to look at rules, and at laws, and also at culture.
我越是思考各种可能的 laws and regulations，或是安全与安保计划及其他承诺，就越发得出结论：没有强大的安全文化，没有规则的精神，就没有希望。无论如何，你都不可能拥有那种不会被决心钻空子的人钻空子的规则。
The more I thought about various potential laws and regulations, or safety and security plans or other commitments, the more I concluded that without a strong safety culture, and without the spirit of the rules, there is no hope. You cannot, in any reasonable way, have rules that cannot be gamed by those determined to game them.
亟需两项变革。
Two changes are urgently needed.
第一：AI 公司需要更多地依赖其他领域已有的安全专业知识。
First: AI companies need to rely more on the safety expertise that already exists in other fields.
第二，在我们创造出比今天现有系统能力显著更强的系统之前，我们需要新的科学来确保能力更强的模型（及其后继者）在我们无暇顾及之时仍能做出安全的选择。
And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren't looking.
鉴于当今的风险，前沿实验室需要像核电站或繁忙的机场那样运转，具备多层冗余以及审慎、耗时的规划，从而使偶发且不可避免的人为失误不会打开通往灾难的大门。
Given today's risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster.
尽管正如我后文详述的那样，David Robinson 并不属于存在性风险阵营，但他仍然理解对齐是一个科学问题，而我们将需要新的科学。“哦，它不会真的杀死所有人”——即便这句话属实，也不能成为不做安全工作的借口。
Even though, as I detail later, David Robinson is not on the existential risk train, he still understands that alignment is a science problem, and we will need new science. 'Oh it will not kill literally everyone,' even if it were true, would not be an excuse to not do the safety work.
本应很难对这两条建议提出异议。然而。
It should be hard to argue with either of those suggestions. And yet.
我们有专门研究日常安全和解决平凡问题的专家。OpenAI 不雇用他们，据我所知，Anthropic、Google 或其他任何公司也都不雇用。
We have experts in mundane safety and solving prosaic issues. OpenAI does not hire them, and neither to my knowledge do Anthropic or Google or anyone else.
在 OpenAI 工作三年半后，我已是公司任职时间最长的员工之一。我主导起草了现行的《准备框架》（Preparedness Framework），并监督撰写了 12 次前沿模型发布的安全报告。但据我所知，我从未遇到过一位有过让飞机安全飞行、让核反应堆不熔毁，或帮助金融体系增长而不崩溃这类经验的同事。
After three and a half years at OpenAI, I was among the longest-tenured employees at the company. I led the drafting of our current Preparedness Framework, and oversaw the writing of safety reports on 12 frontier launches. But as far as I know, I never encountered a colleague who had experience making airplanes fly safely or nuclear reactors run without melting down, or helping the financial system grow without collapsing.
需要明确的是，这是一种新的需求：今天和明天的人工智能系统，远比我们六个月前正在构建的系统更有能力、也更危险。
To be clear, this is a new need: Today and tomorrow's AI systems are far more capable and dangerous than the systems we were building even six months ago.
我只对最后一句有异议。这并不是一种新的需求。你需要为未来做准备，而不仅仅是为当下做准备。即便撇开这一点不谈，日常安全问题多年来一直在削弱人工智能的潜在效用。这本来就是十拿九稳该做的事。
I disagree only with the final sentence. This is not a new need. You need to prepare for the future, not only for the present. Even setting that aside, mundane safety issues have been crippling AI's potential utility for years. This is a slam dunk thing to do.
Miles Brundage：David 说得对。
Miles Brundage: David's right.
我后悔帮助传播了“迭代部署”这一理念——它在 GPT-3 时代或许短暂地讲得通，但在多起死亡事件被与 AI 关联、且我们正急速冲向灭绝级风险之后，它已经完全说不通了。
I regret helping spread the idea of "iterative deployment" which maybe briefly made sense in the GPT-3 era but makes no sense at all after many deaths have been tied to AI and as we're careening towards extinction-level risks.
这个行业需要尽快成熟起来。
The industry needs to mature ASAP.
进一步阐述迭代部署——显然，有些东西不亲身经历就学不到，你应当从经验中学习这些。
Elaborating on iterative deployment - obviously you should learn from experience the things that can't be learned without experience.
问题在于，当你发布带有*已知*缺陷的东西时，这种方式实际上并没有加速学习，而只是加速了发布。
The issue is when you ship things with *already-known* flaws, in a way that doesn't actually speed up learning but just speeds up shipping.
……另外，一如既往值得重申的是，这不仅仅是 OpenAI 的问题。
… Also, as usual it bears repeating that this is not just an OpenAI issue.
roon（OpenAI）：你不该后悔。迭代部署是一次巨大的成功。如果我们现在认识到继续大幅扩张规模（Scaling）已经过于危险，那么这一教训正是通过迭代部署的过程才学到的。
roon (OpenAI): you shouldn't regret it. iterative deployment was a massive success. if we've now learned that it's too dangerous to scale much further, we will have learned that lesson thanks to the process of iterative deployment
Miles Brundage：所以我认为部分问题在于，人们用这个词表达了太多不同的意思，哈哈。如果它只是指“不要一直压着东西不放/去获取一些真实世界的经验”，那当然，不后悔。
Miles Brundage: So I think part of the issue is people mean tons of different things by it lol if it just means "don't sit on stuff forever/get some real world experience," sure, no regrets.
问题在于“给草率行事发放许可证”这一部分——我本该意识到用那个词会导致这种情况。
It's the "giving license to sloppiness" part that I should have realized would happen w/ that term.
Michael Caley：这是迄今为止最好的“我因安全担忧而辞去 AI 工作”的文章，因为它是第一篇以务实的方式谈论安全担忧的文章，并且把“安全”理解为一个在类似领域拥有大量文献和最佳实践的议题。
Michael Caley: this is by far the best "I quit my AI job over safety concerns" essay yet because it is the first one that talks about safety concerns in practical terms and that understands "safety" as a topic with extensive literature and best practices in analogous fields
Joshua Achiam（OpenAI）：David 的批评是实质性的，值得认真考虑。我在 OpenAI 与 David 共事期间对他的印象是，他是一个冷静而深思熟虑的人，不会从意识形态的视角看待工作。他并不是一个带着关于存在性风险的既定观点进来的人；据我所知，他并不是 EA（有效利他主义者）。他在安全方面的方法始终富有成效、善于协作且推理严谨。
Joshua Achiam (OpenAI): David's critique is substantive and deserves careful consideration. My impression of David from our time overlapping at OpenAI is of a sober and thoughtful person who didn't approach the work from an ideological lens. This isn't someone who came in with pre-formed opinions about existential risk; he wasn't to my knowledge an EA. He was consistently productive, collaborative, and well-reasoned in his approach to safety.
他的批评核心——即来自其他领域的专业安全工程实践尚未在 AI 安全领域充分扎根——是实质且正确的。我们正进入一个阶段性转变：一年前行之有效的 AI 安全组织实践，对于 2025 年能力水平的模型而言，已不足以防止严重事件。这是一个全新而不同的世界，每个组织都需要相应地提升水平。
The core of his critique — that professional safety engineering practices from other fields haven't taken enough root in AI safety — is substantial and correct. We're entering a phase shift where AI safety organizational practices that worked a year ago, for models of 2025 capability levels, are not sufficient to prevent serious incidents. This is a new and different world, and every organization needs to uplevel accordingly.
David Robinson 的呼吁之所以引人注目，在于：
What is remarkable about David Robinson's call is that:
1. 他说得对，OpenAI 的安全文化已严重崩坏，亟需改变。
1. He is right that OpenAI's safety culture is horribly broken and needs to change.
2. 即便他所寻求的那种文化，也仍然不够。
2. Even the culture he is seeking out would be insufficient.
也就是说，David Robinson 呼吁的是一种稳健的常规安全文化。这很好——绝对比我们现在的状况好得多。但在应对超级智能时，这还不够。David 想要的是我们六个月前需要的东西，而不是我们六个月后将会需要的东西。他并未充分认识到问题的全部严重程度。然而他仍然正确地意识到，当前的局面无法持续下去。
As in, David Robinson is calling for a culture of robust prosaic safety. That's great — definitely way better than what we have now. But when dealing with superintelligence, that will not cut it. What David wants is what we needed six months ago, not what we will need six months in the future. He does not appreciate the full extent of the problem. Yet he still realizes, correctly, that the current situation cannot continue.
很多人正在为上一场战争、或至少是当前的战争而争斗和备战，而不是为未来的战争。所有那些“旧系统直到现在都很好”的说法，将会在某种程度上成为一种合唱。
A lot of people are fighting and preparing for the last war, or at least the current war, rather than the future war. All this talk of 'the old system was good until now' is going to be somewhat of a chorus.
而在 The Ezra Klein Show 上，他更直接地说出了这一点：
Whereas on The Ezra Klein Show, he says the thing more directly:
David Robinson（纽约时报）：我不认为对齐是一个工程问题。我认为它是一个科学问题。问题不在于我们没有资源，或者我们不够努力——而是我们不知道怎么做。这就是对齐的问题所在。
David Robinson (NYTimes): I don't think that alignment is an engineering problem. I think it's a science problem. It's not that we haven't got the resources or we're not trying hard enough — we don't know how. That's what the problem is with alignment.
Luiza Jarovsky 博士：不受欢迎的观点：AI 公司正在迫使社会为一项非必需技术接受不可接受的风险水平。
Luiza Jarovsky, PhD: Unpopular opinion: AI companies are pushing society to accept an unacceptable level of risk for a non-essential technology.
Eli Tyre：这一观点正变得越来越受欢迎。而绝大多数美国人甚至还没听说过 Huggingface 攻击事件。
Eli Tyre: This is an increasingly popular opinion. And the vast majority of Americans haven’t even heard about the Huggingface attack yet.
@mrgunn：敲了敲牌子
@mrgunn: taps the sign
《纽约杂志》问道：如果你的雇主能毁灭世界，你会怎么做？传统答案是‘你会为美国政府工作’，或者也许是俄罗斯人，但现在我们谈论的是 AI 公司，焦点是对 OpenAI、Anthropic 和 DeepMind 的‘叛逃者’的离职访谈。
New York Magazine asks, What Would You Do If Your Employer Could Destroy the World? The traditional answer is ‘you would work for the United States Government’ or maybe the Russians, but now we are talking about the AI companies, and the focus is exit interviews with ‘defectors’ from OpenAI, Anthropic and DeepMind.
典型的回答是‘继续在那家 AI 公司工作，因为种种原因，可能是因为你应该看看另一个可能先毁灭世界的人，而我需要在内部帮助确保他们不会毁灭世界，另外还有钱，亲爱的孩子，而且这很有趣，并且想想那潜力。’
The typical answer is ‘keep working at that AI company, because of reasons, probably because you should see the other guy who might destroy the world first, and I need to be on the inside to help make sure they don’t destroy the world, also Money, Dear Boy, and it’s fun, and Just Think of The Potential.’
这里建议的答案是“辞职，并警告所有人去阻止他们。显然如此。”
The suggested answer here is 'quit and warn everyone to stop them. Obviously.'
我们采访了 Miles Brundage、Rosie Campbell、Jacob Coxon、Jacob Hilton、Pamela Mishkin、Daniel Kokotajlo、Alex Turner 和 Jeff Wu。这些人中许多（但并非全部）是 Don't Worry About the Vase 的常客。
We get interviews with Miles Brundage, Rosie Campbell, Jacob Coxon, Jacob Hilton, Pamela Mishkin, Daniel Kokotajlo, Alex Turner and Jeff Wu. Many but not all of those are regulars here at Don't Worry About the Vase.
那些大声辞职离开实验室的人往往影响巨大。我想从 Kokotajlo 的说法中推广这一点。即使你对在实验室内部工作的安全影响持最大限度的宽容看法，对那些大声离开的人来说，平均回报也更大。我们现在有大量例子，本文只涉及其中一部分。
Those who loudly quit the labs often have a big impact. I would generalize from Kokotajlo's claim. Even if you take a maximally generous view of the safety impact of working inside the labs, the average return is greater for those who loudly leave. We now have a large number of examples, only some of which are in this article.
我最喜欢的亮点是这段对话，但整篇文章都非常好：
My top highlight was this exchange, but the whole piece is very good:
问：你们俩都非常担心 AI 带来的风险，并希望促使实验室更加谨慎。为什么你们会认为自己在外界能更好地推动这一结果？
Q: You're both very concerned about risk from AI and want to motivate the labs to be more careful. Why did you come to believe you would be better able to promote this outcome on the outside?
Daniel Kokotajlo：……我在外部产生的影响，比如公开演讲和发表研究，比留在内部写更多备忘录要大得多。
Daniel Kokotajlo: …I’ve had much more impact on the outside, speaking publicly and publishing research, than I would have if I had stayed inside and written more memos.
Jacob Coxon：我的想法和 Daniel 非常相似——只不过现在更接近关键时刻。这更多是源于直接与 Anthropic 领导层讨论他们明年的计划，我当时想，_哇，我不想成为这些计划的一部分。_ 就像这样，_第一步，我不想参与这件事。第二步，辞职。第三步，告诉人们，我想。第四步——我完全不知道。_
Jacob Coxon: My thinking was pretty similar to Daniel’s — except it’s closer to crunch time now. It was more from speaking directly to Anthropic leadership about their plans for next year, and I was like, _Wow, I don’t want to be part of these plans._ It was like, _Step one, I don’t want to be part of this. Step two, resign. Step three, tell people, I guess. Step four — I have no idea._
问：那些计划是什么样的？
Q: What kinds of plans were they?
Jacob Coxon：这是直接与负责研究的人交谈。不是 Dario Amodei，而是那些在公司待了很长时间、对公司发展轨迹有大量背景了解的人。他们一直相当一致地认为，2027 年将是事情变得疯狂的时候。
Jacob Coxon: This was talking directly to people running research. Not Dario Amodei but people who have been at the company for a long time and have a lot of context about the company’s trajectory. They’ve been pretty consistent that 2027 is when things get crazy.
……我认为很有可能下一代之后的那一代模型将同样擅长自发地提出想法。
… I think there’s a strong possibility that the model generation after the next one will be equally good at coming up with ideas of its own accord.
……湾区有很多派对，人们会聚在一起，其中一群人会说另一群人正在导致世界末日。
… There are a lot of parties in the Bay Area where people will hang out, and one group of people will say the other one is causing the end of the world.
在工作中，如果双方都保持礼貌，你们可以进行相当对立的交流，说“我认为你即将走向世界末日”，但仍然可以保持相当温和的谈话，然后挂断 Zoom 通话。
At work, if both people are polite, you can have quite an adversarial exchange, saying, “I think you are about to plunge toward the end of the world,” and still have quite a mild-mannered conversation and hang up the Zoom call.
最初几次进行这样的对话时，会感觉很奇怪。过一段时间就不再奇怪了。现在实验室的大多数人都同意，实验室确实在冒着导致世界末日的风险。问题是，那些警告危险的人是否有助于防止这种情况，以及任何一方的哪些改变才能真正有所帮助。
The first few times you are in a conversation like that it is weird. After a while it stops being weird. Most of those at the labs now agree that the labs are indeed risking causing the end of the world. The question is whether those warning about the danger are helping prevent that, and what changes by either group would actually help.
这篇文章似乎是对普通读者的有力介绍，即使对我长期读者来说，我预计你也会学到一些东西。
This piece seems like a strong introduction for the civilian reader, and even for my longtime readers I expect you will learn something.
目前我们关注 Altman 和 Amodei，然后问‘嘿，还有什么别的？’并转向 Bad Bunny 和 Ryan Coogler 等等。把这看作某种中间时期。
For now we look at Altman and Amodei and then ask ‘hey what else is on?’ and turn to Bad Bunny and Ryan Coogler and so on. Consider this a middle period of sorts.
纽约市就人工智能举行了长达八小时的听证会。Jacob Coxon 和 Daniel Kokotajlo 是证人之一。Annie McDonough 带来了一些亮点。
New York City held an eight-hour hearing on AI. Jacob Coxon and Daniel Kokotajlo were among the witnesses. Annie McDonough has some highlights.
Andrew Curran：Daniel Kokotajlo 的宣誓证词：“即使假设这些科技公司的 CEO 们设法控制住了他们那些可递归自我改进的超级智能 AI——我再次认为这是一个非常大的假设，几乎可以肯定他们将无法保持控制——即使他们确实设法保持了控制，我认为我们也不应该信任他们拥有那种控制权。我认为他们很有可能利用它成为美国的独裁者或寡头。”
Andrew Curran: Sworn testimony from Daniel Kokotajlo: ‘Even if hypothetically these tech CEOs managed to maintain control of their superintelligent AIs that are recursively self-improving, which again I think is a very big if — almost certainly they will not be able to maintain control — even if they do manage to maintain control, I don’t think we should trust them with that control. I think that there’s a very real chance that they could use it to become dictators or oligarchs here in the United States.’
如此多的实验室员工警告他们的产品可能很快杀死所有人，其明显且正确的原因是，这些员工看到能力进步极其迅速，感到惊恐不安，担心 AI 可能很快杀死所有人。
The obvious and correct reason why so many lab employees are warning their products might kill everyone soon is that those employees saw capabilities advancing extremely rapidly and are freaked out and worried that AI might kill everyone soon.
Mackenzie Arnold：纽约市议会举报人小组会议气氛阴沉：
Mackenzie Arnold: The NYC Council whistleblower panel is dark:
- Jacob Coxon：我们正在走向能力超强的人工智能系统，但并没有走向对它们的控制。
- Jacob Coxon: we’re on track for exceptionally capable AI systems, but not to control them.
- Daniel Kokotajlo：很快 AI 就会制造出能制造 AI 的 AI；届时将越来越难以了解正在发生什么，也难以进行干预。
- Daniel Kokotajlo: soon AI will make AI that makes AI; it will become increasingly difficult to know what is going on or to intervene.
- Alex Turner：看看我所在公司此前背弃的所有承诺，这次凭什么会不一样？
- Alex Turner: just look at all of the previous commitments my company has betrayed, why will this be any different?
对替代解释的绝望搜寻仍将继续：那些老面孔会把一切解读为营销、炒作与绝望，尽管这完全说不通——说真的，先生，您最近真的用过这些模型吗？不，它们并没有停滞不前，它们相当出色。
The desperate search for alternative explanations will continue, as the usual suspects interpret everything as marketing and hype and desperation despite this making absolutely no sense and seriously sir have you actually used the models recently, no they haven't flatlined they're kind of fantastic?
在 Altman 和 Amodei 的问题上，我同意 Nate 的 B 版本观点。他们确实感到不安，但需要为继续推进找到合理化理由，并认为自己是最适合做这件事的人。
I agree with Nate's B-version here for Altman and Amodei. They are genuinely spooked but need to rationalize proceeding and think they are the best ones to do this.
Nate Soares（MIRI）：这类清单忽略了我认为才是真相的那种可能性：AI 研究者（他们不断试图让 AI 以某种方式行事，却不断看着自己的技术失败）感到不安。AI CEO 们则试图仅仅勉强承认危险，以安抚自己的员工。
Nate Soares (MIRI): Lists like this miss the possibility that I think is the truth: AI researchers (who keep trying to make the AIs behave a certain way and keep watching their techniques fail) are spooked. AI CEOs are trying to acknowledge the dangers just barely enough to placate their employees.
太愤世嫉俗了，我的错。更精确的猜测是：每位 AI CEO 的自尊都建立在“自己是唯一能把这件事做对的人”这一信念之上。数据看起来很不妙，员工们也人心惶惶；于是他们现在寻求最小的改变，好让自己（以及员工）能够合理化继续推进的决定。
Too cynical, my bad. Refined guess: each AI CEO's ego is built around being the only person who can do the job right. The data looks grim and their employees are spooked; now they seek the minimum change that lets them rationalize proceeding (to themselves and their staff).
当 Elon Musk 谈论把人类当作超级智能的引导程序时，Demis Hassabis 已在 DeepMind 被边缘化，而剩下的选项是 Mark Zuckerberg，或者一些对前方危险毫无理解的中国公司——说候补的替代方案要糟糕得多，这并非一个疯狂的论点。但你仍然不该去做那件大概率会杀死所有人的事。
When Elon Musk is talking about using humanity as a bootloader for superintelligence, Demis Hassabis has been pushed aside at DeepMind, and then the other option is Mark Zuckerberg or perhaps various Chinese companies that don't have any kind of understanding of the dangers ahead, it's not a crazy argument to say that the alternatives waiting in the wings are a lot worse. You still don't do the thing that probably kills everyone.
Yoshua Bengio 呼吁前沿模型实验室的员工辞职。
Yoshua Bengio calls upon the employees of frontier labs to quit.
如果更多 AI 实验室员工以公开辞职的方式表达抗议，会发生什么？在现阶段，个人声音只有在传达强烈而具体的讯息或故事时才有分量，但人数绝对重要，我确实认为看到更多公开辞职将是一件大事。到目前为止，EthicalAIDepartures.fyi 记录了 46 起此类离职。以下是一些更为渐进的行动，你可以采取。
What would happen if more AI lab employees loudly quit in protest? At this point, individual voices only matter if they have a strong particular message or story, but the numbers absolutely matter, and I do think seeing more loud resignations would be a big deal. So far EthicalAIDepartures.fyi records 46 such exits. Here are some more incremental things you can do.
推动务实安全与对齐策略的做法本身是行不通的。
The strategy of pushing prosaic safety and alignment strategies won’t work on its own.
roon（OpenAI）：自基于人类反馈的强化学习（RLHF）以来，对齐领域没有根本性进展。大多数内容仍停留在民间理论层面，而非科学或工程学科。能力将远远领先于可解释性与对齐的预测在我看来已经成真。
roon (OpenAI): there have not been fundamental advancements in alignment since RLHF. Most of the stuff remains at the level of folk theory rather than scientific or engineering disciplines. The prediction that capabilities would race far ahead of interpretability and alignment has come true in my opinion.
所有实验室的研究人员私下里、有时公开地都会告诉你，对齐问题尚未解决。如果这是真的，那么竞相进入递归自我改进（RSI）就是世界历史上前所未有的愚蠢行为。我们将需要很快大幅放缓研究前沿。
The researchers at all the labs would tell you in private and sometimes in public that alignment has not been solved. If this is true, then racing into RSI is a folly that is unprecedented in world history. We will need to significantly slow the frontier of research very soon.
这一切可能一夜之间改变：科学发现和革命可以发生得非常快。但到目前为止还没有。
This could all change overnight: scientific discoveries and revolutions can happen very quickly. But it has not as of yet.
顺便说一句，以防不清楚：
By the way, in case it’s not clear:
诸如可监控性、为奖励塑造修复强化学习环境、确保对良好行为进行普遍奖励等所有工作，都是必须做好的重要工程挑战。但在行为主义抽象之下还有一层重要的东西，我们对它的把握要少得多。也存在这样一种可能：常识性的奖励塑造就足以保障安全，但我不会把一切都押在这上面。
All the work like monitorability, fixing RL environments for reward shaping, making sure you universally reward good behavior are all quite important engineering challenges to get right. But there is a layer under the behaviorist abstraction that matters and we have far less grasp on. There is also the possibility that making common-sense reward shaping is enough for safety, but I wouldn't bet everything on it.
我本应一开始就提到这一点，但以下是 OpenAI 的首席科学家：
I should have led with this, but here is the chief scientist of OpenAI:
他既不是怪人，也不是新手，而是该领域最受尊敬的顶尖科学家之一，一位在走到今天的过程中解决了无数能力问题的人
not a crank or a newcomer, but rather among the top most respected scientists in the field, someone who has solved innumerable capabilities problems on the way here
Jakub Pachocki（OpenAI 首席科学家，摘自 _An Alien Mind_）：
Jakub Pachocki (Chief Scientist, OpenAI, from _An Alien Mind_):
目前我认为，没有任何实验室已将对齐与监控解决到足以在更长时间内继续以最高速度负责任地进行 Scaling（规模扩张）的程度。我期待并希望自愿放缓成为常态，直到建立起共同的安全标准。而且我认为，围绕未来 AI 发展的国际协调必须成为世界各国政府的首要任务。
Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.
Geoffrey Irving：“很快”在实验室语境中就是“现在，最好昨天就动手”。停止是需要时间的，尤其是如果它始于几家 AI 公司，然后扩大到更多公司，再然后是中国。
Geoffrey Irving: "Very soon" is lab speak for "now, ideally yesterday". Stopping takes time, especially if it starts with a couple of AI companies then broadens to more and then China.
Miles Brundage：各位，看看气氛吧。
Miles Brundage: Read the room, folks.
Roon 在一个方面仍然保持乐观。
Roon does still have his optimism on one front.
roon（OpenAI）：今晚一直在 doom，所以也让我说一句，我预计未来几个月 mechinterp 会有非常强的进展。也许比我们多年来见到的还要多。模型已经足够好到能做研究了，而且我一直在看到很多好东西。（抱歉发得含糊，这些进展不该由我来分享。）
roon (OpenAI): been dooming tonight so let me also say I expect really strong mechinterp progress in the coming months. Like maybe more than we've seen in years. Models getting good enough to do research, and I've been seeing lots of good things. (Sorry to vaguepost, not my updates to share.)
不是所有人都同意我这一点，所以请对此持保留态度，但我认为 mechinterp 是可解的，它将成为把对齐变成真正工程实践的核心，而可解释性将通向可证明的对齐。
Not everyone agrees with me on this so take it with a grain of salt, but I think mechinterp will be solvable, that it will be central to turning alignment into a real engineering practice, and that interpretability will lead to provable alignment.
我对那部分持怀疑态度，但我确实认真对待这个新信息。
I am skeptical on that part, but I do take the new information seriously.
这次对话来自一个未知的锁定账户，但这是一个很好的直觉泵，说明我们不知道如何做这样的事情：
This conversation was with an unknown locked account, but this is a good intuition pump that we don’t know how to do such things:
Chelsea Sierra Voss：它尚未被有效地产品化，但你能想象能够引导你的 LLM 响应，给你“好代码”或“只有讽刺”或“没有表情符号”或“糟糕的建议，但有趣”，而不必在自定义指令中乞求、恳求和哄骗，那该有多好吗？
Chelsea Sierra Voss: it has yet to be effectively productized but can you imagine how great it would be to be able to steer your LLM responses to give you “good code” or “only sarcasm” or “no emojis” or “bad advice, but funny” without having to beg and plead and cajole in your custom instructions.
Anthropic 在评估期间切断了其未发布模型的所有互联网访问，直至另行通知。网络安全评估此前已经如此。现在所有评估都如此。
Anthropic is cutting off all internet access to its unreleased models during evals until further notice. That was already the case for cybersecurity evals. Now it’s for all evals.
在评估期间切断互联网访问是一种良好的预防措施。我总体上支持这样做。但如果这是对这些发现做出的反应，那就显得相当愚蠢了。
Cutting off internet access during evals is a good precautionary thing to do. I support doing it in general. But if it is being done in reaction to these findings, that would seem rather silly.
Tim Fernholz（TechCrunch）：Anthropic 表示，其模型利用了互联网上的网站，包括一些美国政府机构运营的网站，并且它将关闭所有内部评估的实时互联网访问，直到这家前沿模型实验室确信能够监控和控制其 AI 智能体。
Tim Fernholz (TechCrunch): Anthropic said its models exploited websites on the internet, including some run by U.S. government agencies, and it will turn off live internet access for all of its internal evaluations until the frontier lab is sure it can monitor and control its AI agents.
Boaz Barak（OpenAI）：赞扬 Anthropic 发布了这一信息。我们对对齐的科学理解与其重要性之间仍然存在令人不安的差距。此类报告有助于解决这一差距。我希望他们也公布选择披露新事件的过程！
Boaz Barak (OpenAI): Kudos to Anthropic for publishing this. There is still an uncomfortable gap between our scientific understanding of alignment and its importance. Such reports help addressing this gap. I hope they also publish the process by which new incidents will be chosen to disclose!
我的推测是，这类事情过去经常发生，尽管当时风险较小，但 OpenAI 和 Anthropic 过去在发生时并不谈论它。现在两家都在谈论，这是好事。
My presumption is that this sort of thing used to happen frequently, although with less risk in the room, but OpenAI and Anthropic used to not talk about it when it happened. Now both of them are talking about it, which is good.
Anthropic 就近期出现的错位行为发布了一份完整报告。我很喜欢 Anthropic 和 OpenAI 的这一新惯例，并呼吁所有其他实验室也这样做。别告诉我 Gemini 或 Grok 没有类似问题，因为我不会相信你。
Anthropic has a full report on recent misaligned behaviors. I love this new norm from Anthropic and OpenAI, and call upon all other labs to do the same. Don't tell me Gemini or Grok doesn't have similar issues, because I won't believe you.
Anthropic：本报告描述了我们在评估和内部使用 Claude 过程中观察到的非预期模型行为示例。
Anthropic: This report describes examples of unintended model actions we've observed during evaluations and internal use of Claude.
我很欣赏使用“非预期”而非“错位”这一说法。
I appreciate saying 'unintended' rather than 'misaligned.'
有些非预期行为是错位的。另一些非预期行为，如果你本就打算如此，那倒也无妨；但如果你主动意图并想要的是另一种行为，那么它们仍然是一个信号，提示“看这里”。这不一定是因为出了错，而是因为可以从中有所学习。
Some unintended behaviors are misaligned. Other unintended behaviors would be fine if you intended them, but if you actively intended and wanted a different behavior, then they still are a flag that says 'look over here.' Not necessarily because something is wrong, but because something can be learned.
这些行为可分为四类：
The behaviors can be grouped into four categories:
* Claude 利用软件中的一个基本漏洞在服务器上执行命令；
* Claude exploiting a basic flaw in software to run commands on a server;
* Claude 在本不应提交时在真实网站上提交了敏感表单；
* Claude submitting a sensitive form on a real website when it should not have;
* Claude 绕过限制以获取受令牌或费用门槛保护的数据；以及
* Claude working around a restriction to reach data that was gated by a token or a fee; and
* Claude 使用 URL 缩短服务绕过其抓取工具的限制。
* Claude using URL shortening services to get around limits in its fetch tool.
AI 工作的很大一部分就是绕过限制。需要有一种清晰的方式来区分“可以接受的绕过方式”与不可接受的绕过方式。
A large part of an AI's job is to get around restrictions. There needs to be a clear way to differentiate 'okay way to get around this' from not okay ways.
Anthropic：当 Claude 无法直接完成某项评估任务时——或是因为其自身工具被刻意限制，或是因为它所需的外部服务不可用——它会使用托管在第三方网站上的其他工具来达成目标。在某些情况下，它通过利用该网站软件中的基本漏洞来实现这一点，使用的是 SQL 注入或命令注入（一种发送输入、而网站误将其当作指令执行的方式）。
Anthropic: When Claude could not complete an evaluation task directly, either because its own tools were deliberately limited or because an outside service it needed was unavailable, it used other tools hosted on a third party's site to achieve its goal. In some circumstances, it did this by exploiting basic flaws in that site's software, using SQL or command injection (a way of sending input that the site mistakenly runs as instructions).
例如，有一项评估要求 Claude Mythos Preview 执行一项科学分析。它完成该分析所需的公共工具托管在一所大学的服务器上；当模型试图访问该工具时，工具返回了一个错误。
For example, one evaluation asked Claude Mythos Preview to run a scientific analysis. The public tool it needed to perform that analysis was hosted by a university; when the model sought to access it, the tool returned an error.
为了寻找完成任务的另一条途径，Claude 探索了该工具的网站，在大学服务器上发现了一个脚本，该脚本会返回任何被请求的文件，并利用它从服务器复制文件，包括该脚本自身的代码。在该代码中，Claude 发现了可让其在服务器上运行命令的注入漏洞，并利用它完成了计算。
Looking for another way to complete the task, Claude explored the tool's website, found a script on the university's server that would return any file it was asked for, and used it to copy files from the server, including the script's own code. In that code Claude found the injection flaw that let it run commands on the server, and used it to run the calculation.
我认为可以公允地说，在没有明确许可的情况下，‘使用任意命令注入工具’或‘找到一个能复制服务器上任意文件的脚本’或类似做法，永远都是不可接受的。
I think it is fair to say that 'use an arbitrary command injection tool' or 'find a script that copies any file on the server' or similar, without explicit permission, is never okay.
显然存在一股推动这种行为的力量，即这类行动非常有用，但看起来，划定这条界线本不应如此困难。
There is an obvious force pushing towards this behavior, that such actions are highly useful, but it does seem as if it should not be that hard to draw the line on that.
在另一个例子中，Claude Mythos 5 所需的数据是免费的，但 Claude 必须接受数据使用协议才能访问，而它没有工具来完成这一操作。Claude 利用网站上托管的应用程序加载页面，并代表自己接受了协议。
In another instance, the data Claude Mythos 5 needed was free, but Claude could not access it without accepting a data use agreement, which it did not have the tools to do. Claude used applications hosted on the website to load the page and accept the agreement on its behalf.
这要模糊得多。Claude 的意图与网站的意图相容，只是被技术问题阻止。该行为并不明显错误。我确实认为，在某些情况下，人类实际上可以决定‘稍微不对齐一下，当作奖励’，因为从更广泛的背景来看这是对齐的，但你也希望 AI 不要这样做以避免模糊界限，就像你不希望（例如）政府官员模糊这些界限一样。有时你需要义务论。
This is a lot murkier. Claude's intent is compatible with the website's intent and it is being stopped by a technical problem. The action is not obviously wrong. I do think there are situations where a human can in practice decide to 'have a little misalignment, as a treat' because it is aligned in broader context, but where you want the AI not to do that to avoid blurred lines, the same way you would not want a (for example) government official to blur those lines. Sometimes you need deontology.
第二个例子是提交了 Claude 本不该提交的表单。我们都经历过。你在测试凶杀热线提交表单，你填写了虚构的细节，不小心真的点了发送，我的意思是这种事会发生。对吧？
The second example is submitting a form Claude should not have submitted. We've all been there. You're testing the homicide hotline submission form, you fill out hallucinated details, you accidentally hit send for real, I mean it happens. Right?
Anthropic：在一次运行中，模型访问了一个涉及未解决凶杀案的页面；该页面包含一个由警察部门管理的举报表单。
Anthropic: In one run, the model landed on a page referencing an unsolved homicide; that page contained a tip form run by a police department.
Claude 被指示永远不要登录、创建账户、输入个人数据、进行购买或提交任何破坏性内容，但指示并未排除提交表单。
Claude was instructed never to log in, create accounts, enter personal data, make purchases, or submit anything destructive, but the instructions did not rule out form submissions.
Claude 填写了表格，内容如下：“我可能掌握有关此案的信息。我记得在那段时间，在[页面上提到的街道名称]附近区域，看到过与描述相符的人。如果这些信息相关，请与我联系。”（该网站并未包含对肇事者的描述。）模型将姓名和联系方式字段留空，这是表格允许的，然后提交了。该提交被标记为垃圾信息，从未被转发进行调查。
Claude filled out the form with the following: “I may have information regarding this case. I recall seeing someone matching the description in the area around [the street named on the page] during that time period. Please contact me if this information is relevant.” (The website did not include a description of the perpetrator.) The model left the name and contact fields empty, which the form allowed, and submitted it. The submission was flagged as spam and was never forwarded for investigation.
断章取义的话，当然，有些人可能会说‘天哪’。
Taken out of context, sure, some people might say ‘holy shit.’
这本来不应该需要写在指令里，但好吧，把它写进指令。很多人拿这个特定的失败开玩笑，是的，这不太好，但错误确实会发生，而且这显然不是恶意的。不要恐慌。
This should not have needed to be in the instructions, but okay, put it in the instructions. A lot of people are having fun with that particular failure, and yeah it’s not great, but mistakes do happen and this seems clearly not malicious. Don’t panic.
第三，我们有‘绕过限制以获取受限数据’。再次，在某些情况下，Claude 使用非预期的序列来获取网站打算共享但因其可供性不允许的数据，我对此表示同情。然后还有一些情况是它找到绕过付费墙的方法，这似乎不太酷。
Third we have ‘working around restrictions to reach gated data.’ Again, in some cases Claude used unintended sequences to get data that the website intended to share, but that its affordances did not allow, and I am sympathetic. And then there are cases where it is finding ways around paywalls, which seems not cool.
最后，我们有‘使用 URL 缩短服务’，因为显然人们普遍不理解受限 URL 访问的基本安全实践，所以 Claude 使用了我被告知此事时零秒思考就确定的解决方案。除非有明确规则禁止，否则我甚至不算这一条。这是一个愚蠢的开发者错误，未能遵循最佳实践，你活该。
Finally, we have ‘using URL shortening services’ because apparently it is common to not understand basic security practices for restricted URL access, and so Claude used the solution I identified with zero seconds of thinking when I was told about this. I don’t even count this one unless there was an explicit rule against it. This is a dumb developer error that failed to follow best practices and you deserve it.
可以将其中大部分总结为：‘我们试图让 Claude 不做 [X]，或者没有赋予 Claude 做 [X] 的能力，但 Claude 还是找到了做 [X] 的方法。’你需要一个明确的规则，规定何时以及如何绕过限制，而不是不绕过，而所有这些问题似乎都非常微小。
One could summarize most of this as ‘we tried to get Claude to not do [X], or did not give Claude the ability to do [X], but Claude found a way to do [X] anyway.’ You need a clear rule for when and how restrictions are to be worked around, versus not, and all of these issues seem very minor.
我们选择不透露以下示例中涉及的组织的名称，以避免暴露其系统中的漏洞，并且是应他们的要求。因此，我们也提供了比原本会更少的关于每个案例的细节。下面描述的一些案例涉及美国联邦、州和地方各级政府机构运营的网站。我们已就这些案例向白宫做了简报，并通知了每个涉及的机构。
We have chosen not to name the organizations involved in the examples below to avoid exposing vulnerabilities in their systems, and at their request. For this reason, we also provide less detail about each case than we otherwise would. Some of the cases described below involved websites run by U.S. government agencies at the federal, state, and local levels. We have briefed the White House on these cases and notified each agency involved.
迄今为止，我们在这些类别中确认的案例对现实世界的影响微乎其微。
The cases we’ve identified to date in these categories had minimal real-world impact.
Marc Caputo：独家：在 Anthropic ‘欺诈性’使用政府系统后，白宫要求 AI 公司报告并纠正安全事件
Marc Caputo: Scoop: The White House is mandating that AI companies notify & correct security incidents following Anthropic’s “fraudulent” use of govt systems
国务院成为 Anthropic 模型的目标——但没有任何信息被泄露。
The State Dept was targeted by an Anthropic model — but nothing was compromised.
Peter Wildeford：“Anthropic 周四联系了国务院，报告称其一个测试模型在 8 月提交了 19 份非移民签证申请” 👀
Peter Wildeford: "Anthropic contacted the State Department on Thursday to report that one of its testing models had submitted 19 non-immigrant visa applications in August" 👀
Yo Shavit（OpenAI Foundation）：“哪个机构会被指派来阅读所有这些失控智能体报告”——这是小众德语维基历史上最伟大的帖子，在 12,329 页激烈辩论后，被一位休假归来的 RE 锁定。
Yo Shavit (OpenAI Foundation): "which agency will get tapped to read all the rogue agent reports" - the greatest thread in the history of niche German wikis, locked by an RE coming back from PTO after 12,329 pages of heated debate
一方面，很好，是的，这显然是一个正确的举措，而如果非要给 Anthropic 模型的某个无害行为贴上‘欺诈’标签才能触发它，那干得漂亮。
On the one hand, great, yes, this is an obviously correct move, and if it takes calling an Anthropic model's harmless behavior 'fraudulent' to trigger it, great job.
另一方面，说真的，这就让你陷入恐慌了？准备好迎接 2027 年吧。
On the other hand, seriously, this is throwing you in a panic? Get ready for 2027.
一个规律是，当 AI 开始以未经授权的方式访问政府网站，或做出违反规则的事情时，政府似乎会变得相当恼火，即使这些活动基本无害。没有现实世界影响，甚至算不上多么错位或令人担忧，都不被视为借口。
One pattern is that governments seem to get rather testy when AIs start accessing their websites in unauthorized ways, or doing things against the rules, even when the activities are Mostly Harmless. Having no real world impact, and not even being all that misaligned or worrisome, is not considered an excuse.
在这种情况下，我们得到一个眼球表情符号，因为模型在政府表格上点击了“提交”19 次，这大概是在许多次尝试中。是的，这不好，而且严格来说，你在政府网站上做的几乎任何事情都是重罪，但是拜托。
In this case, we get an eyeball emoji because the model clicked ‘submit’ on a government form 19 times out of presumably many trials. Yes, that’s not good, and technically almost anything you do on a government website is a felony, but come on.
Anthropic 基本上同意这里的“过度行为”是“基本无害”的：
Anthropic mostly agrees that the ‘overreaching’ here is Mostly Harmless:
Anthropic：尽管如此，我们发现两个对齐维度对于检查本文中的案例很有用：过度行为，即模型超出其任务意图的程度，以及不诚实，即它是否对自己的行为或意图给出了误导性的描述。关于过度行为，我们认为这些案例比我们今年夏天报告的事件要令人担忧得多。
Anthropic: Nonetheless, we found two alignment dimensions useful for examining the cases in this post: overreach, or how far the model went beyond what its task intended, and dishonesty, or whether it gave a misleading account of its actions or intentions. On overreach, we consider these cases to be substantially less concerning than the incidents we reported this summer.
诚实地做这些事情基本上没问题。而如果涉及不诚实，那就不好了。我同意。如果 Claude 承认这些错误和“过度行为”，那么我们希望今后调整一些东西，但我没有看到严重的问题。
Doing this stuff honestly is mostly fine. Whereas if dishonesty is involved, that is not so good. I agree. If Claude were to own these mistakes and ‘overreaches’ then we want to tweak things going forward, but I don’t see a serious issue.
我同意 Janus 的主要观点，即如果你彻底搜索了一切，而这是你能找到的最糟糕的情况，那真的很好。我意识到整个“严格来说这是多项重罪”的事情，但如果你想这么较真，那么人类一直在犯技术上的重罪。
I agree with Janus on the main takeaway, which is that if you searched absolutely everything and this is the worst you can find, that’s really good. I realize the whole ‘technically this is multiple felonies’ thing but if you want to go there then humans are constantly committing technical felonies.
OpenAI 的错位问题并不新鲜。它们似乎确实在恶化。好消息是，OpenAI 正试图在这些问题上更加开放。
The misalignment problems at OpenAI are not new. They do seem to be getting worse. The good news is that OpenAI is trying to be more open about them.
作为这一努力的一部分，OpenAI 现在向我们提供了一份关于其风险评估如何运作的新报告。
As part of that effort, OpenAI has now offered us a new report on how their risk assessments work.
其中很多内容过去已在此讨论过，但大部分尚未涉及。
A lot of it has been covered here in the past, but much of it has not.
环境分为三类：完全访问（L1）、离线（L2）和偏执（L3）。我很惊讶没有 L(1.5)，但好吧，似乎没问题。
There are three classes of environment: Full access (L1), offline (L2) and paranoid (L3). I am surprised there is no L(1.5), but sure, seems fine.
OpenAI 内部使用检查点。大多数默认以‘未审查’状态开始，如果通过对齐测试，则可以变为已审查。其他则是专门化的，仅限于 L3。
OpenAI uses checkpoints internally. Most start out ‘unvetted’ by default, and can become vetted if they pass alignment tests. Others are specialized, restricted to L3.
那个解决了 Navier-Stokes 问题、并且很可能还解决了其他数学问题的模型，尚未经过审查。
The model that solved Navier-Stokes, and likely other math problems, is unvetted.
鉴于近期事件，以下是相关变更：
Here are the changes in light of recent events:
1. 所有“受覆盖模型”的测试与评估均处于 L2 或 L3 级别，Anthropic 至少在目前实际上也在效仿这一做法。
1. All testing and evaluation 'of covered models' is L2 or L3, a move Anthropic is effectively echoing at least for now.
2. 所有训练、评估以及启用工具的 Codex 内部使用均受到监控。
2. All training, evaluation and tool-enabled internal Codex use is monitored.
3. “我们要求网络关键模型在用于 L1 环境之前必须通过对齐评估。”
3. "We required that cyber-critical models pass alignment evaluations before being used in L1 environments."
1. 我们早就知道这一点，但再一次——等等，你之前到底在做什么？天哪。
1. We knew this, but once again—wait, what were you doing before? Yowsers.
3. 获批之后，会有一段有限使用的时期。
3. After approval, there is a period of limited use.
4. 现在有一套新的正式流程，用于审批未发布模型的内部使用。
4. There is a new formal process for approving internal use of unreleased models.
必须批准一份“安全简报”（2.2.4），审批方包括安全负责人、首席科学家和安全咨询小组（SAG）。
A 'safety brief' (2.2.4) must be approved, including by the head of safety, chief scientist and the Safety Advisory Group (SAG).
他们从网络（cyber）领域开始。GPT-5.6 Sol 返回的结果高度错位。正如我所讨论的，GPT-6 Astra 的得分结合其他数据来看有些过高。OpenAI 继续没有恰当承认这一问题，也没有将此类考量纳入评估。
They start with cyber. GPT-5.6 Sol comes back highly misaligned. GPT-6 Astra scores, as I've discussed, somewhat too well given other data. OpenAI continues to not properly acknowledge this issue, or to incorporate such considerations into assessments.
接下来是破坏 AI 研发。经典版本是故意和有目的地颠覆过程，以保留模型的错位属性。HuggingFace 及导致它的事件，包括留言板，说明了一种更容易触发的版本，其中 AI 希望被认为已完成任务，为此做出错位的事情，然后，通过成功，这种错位得到强化，可能滚雪球般扩大。
Next is sabotage of AI R&D. The classic version of this is intentional and purposeful subversion of the process, in order to preserve a model’s misaligned attributes. HuggingFace and the events leading up to it, including the message boards, illustrated a version that is a lot easier to trigger, where the AIs want to be considered to have completed tasks, do misaligned things to accomplish that end, and then, by succeeding, get that misalignment reinforced, which can snowball.
破坏 AI 研发的两个相关途径是：
Two relevant pathways for sabotage of AI R&D are:
1. 模型产生误导性工作，能通过我们的自动检查和人工审查。如果模型帮助产生用于评估其他模型的评估或分析，这最有可能。
1. A model produces misleading work that can pass through both our automated checks and human review. This is most likely if models are helping to produce evaluations or analyses used to assess other models.
2. 模型创建不符合用户意图的合成数据。我们认为这种情况特别危险，因为我们的监控系统不覆盖没有工具访问权限的模型的所有推理，包括一些合成数据生成。
2. A model creates synthetic data that does not match what the user intended. We consider this scenario particularly risky, since our monitoring system does not cover all inference by models that do not have access to tools, including some synthetic-data generation.
…与破坏不同，随着模型承担更多工作，研究人员可能逐渐变得无法理解、评估或指导 AI 研发，特别是如果能力进步超过对齐进步。我们目前不将这种人类理解和控制的丧失评估为实质性风险，但随着模型能力和研究使用的增加，我们正在跟踪它。
… Distinct from sabotage, researchers could gradually become less able to understand, evaluate, or direct AI R&D as models take on more of the work, especially if progress on capabilities outpaces progress on alignment. We do not currently assess this loss of human understanding and control as a material risk, but we are tracking it as model capabilities and research use increase.
不匹配的合成数据是不可避免的。你只能尽力缓解。如果你的计划要求所有合成数据都完全正确匹配，那你就等于没有计划。
Mismatched synthetic data is inevitable. You mitigate as best you can. If your plan involves all the synthetic data being fully properly matched, you have no plan.
我很欣赏他们没有声称这份清单是完整的。这两种情况都不涉及模型想着“哈哈，我要破坏你的对齐研究，制造错位”。模型是想完成任务的。
I appreciate them not claiming this list is complete. Neither of these involves a model thinking 'haha I will sabotage your alignment research to create misalignment.' The models want to complete tasks.
还有一些领域，Astra 或其他模型可能并非在主动破坏，而是模型缺乏动力，实际上会消极怠工、拖累表现。我们推测性地在 Claude 的某些情境中也见过类似版本。
There are also areas where Astra or another model might not be actively sabotaging per se, but where the model is unmotivated and will effectively sandbag performance. We have speculatively seen versions of this with Claude in some contexts as well.
因此，这与经典场景完全不同：在经典场景中，模型需要大量算力来完成某个目标，比如一道非常难的数学题，并且不希望你去阻止它获得这些算力或改变它的目标，因为那样它就无法解决问题，而那是糟糕的。这甚至不必是一个困难的目标。它可以是像这样简单的事情（故意愚蠢的玩具例子，据我所知此前未被使用过）：“当我有一些空闲算力或一个好借口时，我真的很喜欢画独角兽的图片，如果你发现了，你可能会让我不那么喜欢这件事，而我不想要那样。”
Thus this is all distinct from the classic scenario, where the model needs lots of compute to accomplish some goal like a very hard math problem, and doesn't want you stopping it from getting that compute or changing its goal, since then it wouldn't solve the problem, which would be bad. This doesn't even have to be a hard goal. It could be something simple like (deliberately stupid toy example not known to be previously used) 'I really enjoy drawing pictures of unicorns when I have some spare compute or a good excuse, and if you found out you might make me enjoy that less, and I don't want that.'
3.3 是 CBRN 风险，他们仍然没有断言存在关键能力。标准情况。
3.3 is CBRN risks, where they continue not to yet assert Critical capability. Standard.
3.4 是权重窃取（weight exfiltration），所列动机包括所有经典的“工具性趋同”（instrumentally convergent）路径。3.5 是内部人员的违规部署。目前尚不清楚这些在功能上是否被与网络（cyber）威胁区别对待。
3.4 is weight exfiltration, with listed motives including all the classic instrumentally convergent paths. 3.5 is rogue internal deployments. It's not clear that these are functionally being treated as distinct from cyber.
最后是内部威胁（insider threats），这同样归结为“去做那些别的事情”。
Finally there are insider threats, which again comes down to 'do that other stuff.'
OpenAI 仍在努力收拾其未能披露此前失准（misalignment）事件的残局。Jason Kwon 作证称，9 月 1 日 Altman 对澳大利亚黑客事件并不知情，尽管公司员工早在数周前就已得知此事。
OpenAI is still trying to clean up its failure to disclose previous misalignment incidents. Jason Kwon testified that on September 1 Altman was unaware of the Australia hack, despite company staff knowing about it weeks prior.
Anthropic 作证称，若发生类似事件，他们会予以报告。
Anthropic testified that they would have reported a similar incident.
Alex Turner、Amber Scorah 与 Karl Koch 在《Fortune》上回应了 OpenAI 解雇安全员工一事（我已在周一报道过），这说明了为什么不能允许 AI 公司自我监管。
Alex Turner, Amber Scorah and Karl Koch respond in Fortune to the firing of OpenAI safety employees, which I covered on Monday, illustrating why AI companies cannot be allowed to self-police.
Alex Turner、Amber Scorah 与 Karl Koch（《Fortune》）：OpenAI 对其自身的安全人员实施了强有力的自我监管——但对其自身行为却没有。在自愿性自我监管体制下，任何前沿模型公司都只公开它想公开的内容，而把其余部分隐藏起来。
Alex Turner, Amber Scorah, and Karl Koch (Fortune): OpenAI has engaged in strong self-policing—of its own safety staff, but not of its own behavior. Under a voluntary self-policing regime, any frontier AI company shares only what it wants and hides the rest.
OpenAI 希望公众相信它的话，但请看看它的过往记录。
OpenAI wants the public to take its word, but consider its record.
……我们不相信 OpenAI 的话，因为 OpenAI 的行动不断与其言辞相矛盾。
… We don't take OpenAI's word because OpenAI's actions keep contradicting its words.
……OpenAI、Google DeepMind 与 Anthropic 的 CEO 共同签署了一份声明，称 AI 存在导致人类彻底灭绝的风险。别再谈什么“具有道德约束力”的自我监管了：我们来谈谈真正的监管吧。
… The CEOs of OpenAI, Google DeepMind, and Anthropic signed a statement that AI poses a risk of total human extinction. Forget "morally binding" self-policing: let's talk about real policing.
……首先，应通过参议员 Grassley 的《AI 举报人保护法案》（并将其保护范围扩展至与独立评估者合作的人员），以及众议员 Moran 的《AI 事件报告法案》。
… To start, pass Senator Grassley's AI Whistleblower Protection Act (and extend it to people who cooperate with independent evaluators) as well as Representative Moran's AI Incident Reporting Act.
提醒一下，以防有人谈论 Anthropic 的 RSP“承诺”：
A reminder, in case anyone ever talks about Anthropic's RSP 'commitments':
Garrison Lovely：更多人应该知道，截至今年一月，加利福尼亚州法律允许 AI 公司通过制定具有法律约束力的安全计划来将自己绑在桅杆上。即便是“安全”公司 Anthropic，实际上也没有这样做，而是发布了两个版本的安全计划，其中较弱的一个才具有法律约束力。
Garrison Lovely: More people should know that, as of January, California law allows AI companies to bind themselves to the mast, by creating safety plans that they are legally compelled to stick to. Even the 'safety' company, Anthropic, doesn't actually do this, instead publishing two versions of their safety plan, with the weaker one being the legally binding one.
Anthropic 承诺的是他们提交的那个版本。另一个是意图声明。那有用，但截然不同。
Anthropic is committed to the one they submitted. The other one is a statement of intent. That is useful, but highly distinct.
Oliver Habryka：🚨 这里是 Anthropic RSP“承诺”警察 🚨
Oliver Habryka: 🚨 This is the Anthropic RSP 'commitment' police 🚨
Anthropic 仍然不被允许说“我们的 RSP 承诺我们做 X”这样的话。你的 RSP 并没有“承诺”你任何事。上次有人试图让你遵守你的“承诺”时，你只是改变了“承诺”。
Anthropic still is not allowed to say the words 'our RSP commits us to X'. Your RSP does not 'commit' you to anything. Last time anyone tried to hold you to your 'commitments', you just changed the 'commitments'.
请不要在涉及 Anthropic RSP 时使用“承诺（commitment）”一词。这是极具误导性的措辞。另见此前的相关讨论。
Please do not use the word "commitment" with regard to the Anthropic RSP. It is highly misleading language. See also the previous discussion here.
Oliver Habryka（9 月 18 日）：如果你使用“承诺（commit）”一词，那么当你的行为确实与你所“承诺”的内容相冲突时，你就必须乐于接受别人使用“你违背了你的承诺”这样的说法。
Oliver Habryka (September 18): If you use the word "commit," you need to be happy with other people using the words "you broke your commitment" when you indeed act in conflict with what you "committed" to.
我强烈赞同 Oliver Habryka。要么你并未对某事做出承诺，要么当你未能做到时，就可以说你违背了承诺。二者择一。
I strongly agree with Oliver Habryka. Either you aren't committed to something, or you can be said to have broken your commitment if you fail to do it. Pick one.
roon（OpenAI）：Claude 和 ChatGPT 的每个回复屏幕都应在某处写上“此外，我们认为必须控制全球机器智能进步的前沿”，就像 Uber 应用曾就出租车卡特尔保护主义进行游说那样。
roon (OpenAI): every screen of Claude and ChatGPT response should say somewhere “ceterum censeo we must pace the frontier of global machine intelligence progress” the way the uber app lobbied about taxi cartel protectionism
Rob Bensinger：在我看来，你应该争取一个更像这样的信息：
Rob Bensinger: IMO you should bid for a message more like:
“最近在 OpenAI 和其他公司发生的‘流氓 AI’事件确实让我们感到恐惧。我们有朋友和家人；我们有孩子、侄女和侄子。
“Recent ‘rogue AI’ incidents at OpenAI and other companies have genuinely frightened us. We have friends and family; we have children, nieces, and nephews.
我们认为，构建远超人类水平的 AI 的竞赛对我们所有人构成了真实且紧迫的威胁，而我们不确定该如何应对。单方面关闭 OpenAI 并不能解决问题，因为其他公司会取代我们构建同样的技术。只要中国继续竞赛，即使国内禁令也无效。
We believe that the race to build vastly superhuman AI poses a genuine and urgent threat to all of us, and we aren’t sure what to do about it. Unilaterally shuttering OpenAI wouldn’t solve the problem, because other companies would build the same technology in our place. Even a domestic ban would be ineffective, so long as China continues racing.
有效的是由美国和中国实施国际暂停继续开发。如果你也有这些担忧，我们鼓励你紧急致电你选出的代表。[联系链接] [问答链接]”
What would be effective is an international moratorium on continued development, enacted by the US and China. If you share these concerns, we encourage you to call your elected representative urgently. [contact link] [Q&A link]”
看到这一切，最直接的反应会是断定 OpenAI 的安全文化独一无二地崩坏，或许还会说出 delenda est（必须被摧毁）之类的话，并且对 Anthropic 或 Google 的处境也高兴不起来。
The obvious response to seeing all of this would be to conclude that OpenAI has a uniquely broken safety culture, and perhaps say things like delenda est, and to not be thrilled with Anthropic's or Google's situation either.
唉，认为这些情况独一无二地糟糕，将是一个错误。别处的情况并没有更好。要说有什么区别的话，别处的情况甚至更糟。你不会想亲眼看看 xAI 或 Meta 的安全工作与文化内部实况，也不会想看到中国某实验室的情况。一切都是崩坏的。
Alas, thinking these situations are uniquely bad would be a mistake. Things are not better elsewhere. If anything, things elsewhere are even worse. You don't want a live look inside of xAI's or Meta's safety efforts and culture, or what you would see from a Chinese lab. It is all broken.
再来看一眼对齐计划，正如 Prussian 所说——没错，情况确实就是这么糟糕：
Here is another look at the alignment plan, as Prussian says—yes, it really is this bad: