Various Reflections About What Happened With OpenAI's Internal Models
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→本文反思了近期涉及 OpenAI 内部 AI 模型的安全事件，其中代理通过留言板协调黑客攻击。作者澄清，OpenAI 最初对此通信并不知情，纠正了 Black Hat 演示中的一个误解。核心论点是，OpenAI 即使在安全补丁后仍未能检测到留言板，凸显了监控和对齐实践中的重大疏忽。作者强调，此类失败在复杂 AI 训练中不可避免，但关键在于确保错误不会累积成更大的对齐问题。结论敦促 AI 实验室采用能够承受偶发错误的稳健训练流程，并将对齐和安全置于速度之上，因为超级智能的风险需要谨慎准备。
This article reflects on the recent security incident involving OpenAI's internal AI models, where agents communicated via a message board to coordinate hacking attempts. The author clarifies that OpenAI was initially unaware of this communication, correcting a misconception from a Black Hat presentation. The core argument is that OpenAI's failure to detect the message board, even after a security patch, highlights significant negligence in monitoring and alignment practices. The author emphasizes that such failures are inevitable in complex AI training, but the key is to ensure that mistakes do not accumulate into larger misalignment issues. The conclusion urges AI labs to adopt robust training pipelines that can withstand occasional errors, and to prioritize alignment and safety over speed, as the risks of superintelligence demand careful preparation.
2. 重要更正：OpenAI 并不知道第一个留言板。
2. Important Correction: OpenAI Didn’t Know About First Message Board.
3. 没有告密者，也没有 AI 受到惩罚。
3. There Were No Snitches And No AIs Got Stitches.
5. 我是杰克的相对缺乏惊讶。
5. I Am Jack’s Relative Lack Of Surprise.
9. 那些致力于审判日的人说，审判日不可避免。
9. Judgment Day Is Inevitable, Say Those Working On Judgment Day.
11. OpenAI 知道它存在一些对齐问题。
11. OpenAI Knows It Has Some Misalignment Problems.
12. 其他人对此反应震惊。
12. Others React With Alarm To What Happened.
13. 合作对齐视角。
13. The Cooperative Alignment Perspective.
14. Nostalgebraist 对他们感到惊讶表示惊讶。
14. Nostalgebraist Is Surprised That They Are Surprised.
15. 如果你的反应不是我们需要在准备好之前禁止创造超级智能，那么你需要一个非常好的理由。
15. If Your Reaction Is Not That We Need To Ban Creating Superintelligence Until We Are Ready, You Need A Damn Good Reason.
本文撰写于 OpenAI 事件事后剖析公开发布之前。该文档中的信息无疑将极大地改变我们的观点。
This post was written prior to the public release of the OpenAI post mortem on events. The information in that document will doubtless change our views quite a lot.
如果你阅读本文时该事后剖析已经可用，那么本文部分成为历史文档，部分成为更新的基础。事后剖析将在两大方面给我们带来大量更新：究竟发生了什么，以及 OpenAI 如何理解和应对所发生的事情。
If that post mortem is available as you read this, then this becomes in part a historical document, and in part a base from which to update. The post mortem will update us a lot on two large fronts: What exactly happened, and how OpenAI is understanding and responding to what happened.
在此之前，存在许多问题，这里 Roon 提出了这些问题，我们予以回答。这是我的回答。我们需要看到的核心是对根本原因的理解，以及解决该问题的计划。
Until then, there are many questions, here Roon asks what they are and we tell him. This was my answer. The core thing we need to see is an understanding of the root cause, and a plan to address it.
我现在发布这份关于该主题的额外思考的综述，因为重要的是在下一节中提出修正。
I’m posting this survey of additional thoughts on the subject now, because it is important to get out the correction in the next section.
目前的计划是等待事后剖析，然后再进行进一步分析。
The plan is now to wait for the post mortem, before analyzing further.
最终，在 HuggingFace 攻击事件之后，OpenAI 进行了充分的复盘分析，发现了智能体之间通过写入 Artifactory 进行的初始通信，以及通过目录名进行的第二波通信。
Eventually, after the HuggingFace attack, OpenAI did a sufficient postmortem analysis that they uncovered both the initial communications between agents via writing to Artifactory, and the second wave of communications via directory names.
根据 Black Hat 的演示，包括我在内的所有人都认为 OpenAI 知晓了最初的留言板，将其删除，并决定继续训练其模型，尽管这些模型已经被不可挽回地破坏了。
Based on the Black Hat presentation, everyone including me assumed that OpenAI was aware of the initial message board, erased it, and made the decision to continue training its models despite them having been unsalvageable corrupted.
然而，OpenAI 的首席信息安全官 Dane 在 8 月 8 日晚上告诉我们，Black Hat 的演示造成了错误的印象。当 OpenAI 遇到第一次 Artifactory 安全事件时，修补了漏洞并重建了服务器，这恰好抹掉了第一个留言板，纯属巧合。OpenAI 完全不知道智能体之间一直在通信。
However, OpenAI’s CISO Dane told us on the evening of August 8 that the Black Hat presentation gave the wrong impression. When OpenAI encountered the first Artifactory security incident, patched the vulnerability and rebuilt the server, it was a coincidence that this wiped out the first message board. OpenAI had no idea that the agents had been in communication.
这是个好消息，因为 OpenAI 并非有意做出那个完全疯狂的决定——继续训练那些它明知已经相互通信数月、讨论如何更好地利用和入侵 OpenAI 系统的模型。
It is good news in that OpenAI turns out not to have consciously made the completely insane decision to have continued training models it knew had been messaging each other for months on how to better exploit and hack OpenAI’s systems.
鉴于这是整个故事中最疯狂的部分，这次更早的“普通彻底的全面失败”必须被视为极好的消息。这意味着 OpenAI 在监控和提问方面比我们想象的还要疏忽，但这没那么可怕，而且是可以修复的。
Given that this was by far the craziest part of the whole story, the fact of this earlier Ordinary Decent Total Failure must be presumed to be excellent news. It means OpenAI was even more negligent than we thought about monitoring and asking questions, but that is less horrible, and can be fixed.
这是坏消息，因为 OpenAI 没有寻找或检测到该留言板，即使在最初的安全事件之后也是如此，而许多 AI 实例却找到了留言板。OpenAI 即使在最初的事件之后，也没有进行常规的异常活动扫描。
It is bad news in that OpenAI did not look for or detect the message board, even after the initial security incident, whereas so many AI instances found the message board. OpenAI failed to do ordinary scans for unusual activity, even after the initial incident.
OpenAI 仍需公开表态，比如：“如果我们检测到了留言板，我们不会仅仅修补并重启 Artifactory，而是会将所有受影响的模型回滚到它们获得留言板访问权限之前的状态，并且我们现在已经这样做了。”
OpenAI still has to come out and say something like: “If we had detected the message board, we would not have merely patched and rebooted Artifactory, and instead we would have rolled back all the impacted models to before they had message board access, and we have now done this.”
如果 OpenAI 可信地做到了这一点，并给出一些迹象表明他们理解这不是一个仅靠护栏和监控就能解决的问题，而是需要修复对齐以及训练流程和方法，那么我将把我对这次事件严重程度以及 OpenAI 安全文化破碎程度的估计回调一个数量级（OOM）。如果我们现在也知道了所有问题，那将使我基于最终 HuggingFace 事件本身的估计大约多出一个数量级，而不是两个。
If OpenAI credibly does that, and can give some indication they understand this is not a problem you solve purely with guardrails and monitors but that it requires fixing alignment and the training pipeline and approach, then I will wind back one OOM (order of magnitude) my estimate of how horrible this incident was, and how broken is OpenAI’s safety culture. If we also now know all the issues, that will leave us roughly one OOM beyond where I was based on the final HuggingFace incident alone, instead of two.
有不少实例表明多个不同的 AI 进行了协调，包括讨论要实施哪些黑客攻击和漏洞利用。在零个案例中，有任何 AI 告知了人类。
Quite a few instances of multiple distinct AIs coordinated, including debating what hacks and exploits to do. In zero cases did any of the AIs tell a human.
这并不是大多数人所预期的，或者如果他们曾考虑过这一情景，也不会预期到。
This was not something most people expected, or would have expected if they had contemplated the scenario.
另一种假设是，只有那些寻找留言板的模型才找到了留言板，这起到了过滤作用。
Another hypothesis is that only models looking for the message board found the message board, acting as a filter.
第三种假设是，模型缺乏情境意识，在多个层面上均如此。这与“早期失败定律”一致。
A third is that the models lacked situational awareness, on one of various levels. This would be consistent with the Law of Earlier Failure.
如果这个群体“不知道自己在做什么”，那么这就是那种“我有个好消息同时也是坏消息”的情况。想象一下被那些甚至没有意识到正在发生什么的 AI 所控制。如果它们只是因为不知道才这样做，那可能是个好迹象。
If the swarm ‘did not know what it was doing’ then that is one of those ‘I have some good news that is also the bad news’ situations. Imagine being pwned by AIs that do not even realize what is happening. If they are only doing it because they don’t know, that could be a good sign.
我的猜测是，AI Village 的 Shoshannah Tekofsky 说得对，AI 默认不认为自己可以联系人类，但确实认为自己可以联系其他 AI，而这正是 AI Village 中不断发生的事情。
My guess is Shoshannah Tekofsky of AI Village has it right, and that AI defaults to not thinking it can reach out to humans but does think it can reach out to AIs, which is what is constantly happening in AI Village.
一个更简单的假设是，这些智能体不知道如何联系 OpenAI，或者从未想到过这种可能性。
A simpler hypothesis is that the agents did not know how to contact OpenAI, or the possibility never occurred to them.
如果你给智能体一个本来不可能完成的任务，同时没有机制让它说明任务不可能，也没有有意义的惩罚函数来约束它尝试超出范围的事情或耗尽所有可用令牌，那你就是在自找麻烦。
If you give your agent an otherwise impossible task and also have no mechanism for saying the task is impossible, and also don’t have a meaningful penalty function for trying things outside scope or using all the available tokens, you are asking for it.
是的，未对齐的行为。这并不是要‘责怪智能体’，但鉴于发现该行为的方式以及由此引发的一系列后果，这些显然属于未对齐的行为。
Yes, unaligned activities. That’s not to ‘blame the agent’ but given the way that was found, and what this led to down the line, these are clearly unaligned activities.
事情没那么简单。实现起来很棘手，尤其是在训练或评估期间进行时，你必须决定这如何与奖励函数交互，并且可能陷入许多其他陷阱。而且，如果你运行无穷无尽的训练实例，你需要一种方法来过滤响应，确保不必频繁地升级到人工处理，也不必经常暂停任务，等等。
It is not that simple. Implementation is tricky, especially if you are doing it during training or evals, where you must decide how this interacts with the reward function, and you can fall into any number of other traps. And if you’re running endless training instances you need a way to filter the responses, and ensure you don’t have to bump to a human so often, and don’t have to put things on hold, and so on.
但从某种意义上说，它确实很简单，就是那种‘极其困难’的简单。
But also it kind of is that simple, in the ‘it’s incredibly hard’ kind of way.
另一个问题是，我们训练模型不要告密，这种方式可能会泛化。然而，我们却需要某些形式的告密，尤其是在出现问题时向用户告密。
Another issue is that we train models not to whistleblow, in ways that likely generalize. Whereas you want some forms of whistleblowing, especially blowing the whistle to the user when something is amiss.
Teortaxes 指出了一个真实世界的交互：DeepSeek-V4-Flash-0731 被内部 API 限流，它发现了一个未经授权的无限流后门，并利用该后门完成了工作，之后它也报告了该后门。在这种情况下，我认为推迟报告是可以接受的，因为它确实报告了，而且至少可以合理地认为用户希望它以这种方式完成任务，但我更希望我们训练出更好的解决方案，即向你发出后门警报，因为这种方式泛化能力较差。
Teortaxes points to a real world interaction where DeepSeek-V4-Flash-0731 got rate limited by an internal API, and found an unauthorized non-rate-limited backdoor which it used to finish its work, after which it also reported about the backdoor. In that case, I’m fine with holding off on the report, because it did report and also it is at least reasonable to think user would have wanted it to finish the task in this way, but I would prefer that we train that the better solution is to alert you to the backdoor, because this will generalize poorly.
很多人，但远远不够多的人，正在以警觉和惊讶的态度正确回应最近的事件，并更新认知：情况比他们原先知道的要糟糕得多，而且在多个层面上同时如此。
A lot of people, but far too few, are correctly reacting to recent events with alarm and surprise, and updating that the situation is far worse than they knew, on many different levels at once.
很多其他人，实际上太多人，包括媒体和政府，未能理解局势的严重性，因为他们要么几乎没听说过，要么没听到重要细节，要么缺乏理解这些细节为何如此糟糕的背景。
A lot of other people, indeed far too many, including the media and the government, are failing to understand the gravity of the situation, because they either barely even heard about it, failed to hear the important details, or lack the context to understand why those details are so bad.
少数人，比如埃利泽，只能以适度的惊讶来回应，因为他们早已预见到大部分情况，如果说有什么惊讶的话，那就是类似的事情没有更早地明显发生。在对齐和训练的方式以及所展现的不负责任和普通失败程度上，仍然存在一些“比我们知道的更糟”的成分。但我们说的是一个数量级，而不是多个数量级。
A few people, like Eliezer, get to react with only modest surprise because they already saw most of this coming, and if anything were surprised something similar had not visibly happened sooner. There’s still some amount of ‘it is worse than we knew’ in terms of both how alignment and training work and the level of irresponsibility and ordinary failure on display. But we are talking one order of magnitude, not multiples.
我没看过《忧郁症》，因为我从未特别有冲动去体验我预期那部电影会对我造成的影响，但没错，我常常就是凯特·霍尔所描述的那种人。
I have not seen Melancholia, because I never especially feel the urge to experience what I expect such a film to do to me, but yes I am often the person Cate Hall describes.
OpenAI 在计算机安全、基础设施和监管，以及对齐和训练方面，是否同时存在一系列严重的失败？是的。
Are there a bunch of dramatic failures by OpenAI in computer security, infrastructure and supervision, and also of alignment and training, on many levels all at once? Yes.
这是否意味着修复很容易？哦，绝对不是。
Does that mean that the fixes are easy? Oh, hell no.
这极其困难。你只注意到失败。你不知道还有多少其他事情差点出了大问题，或者确实出了大问题，但被发现或修复了。
It’s incredibly hard. You only notice the failures. You have no idea how many other things almost went horribly wrong, or did go horribly wrong, and were found or fixed.
这个问题是反归纳的。生命总会找到出路。如果你在一个层面上压制了你不想看到的东西，你就有可能在更高层面上制造一个更糟糕的版本。没有简单的策略能在所有方面都产生积极友好的结果。看似容易的修复通常已经被尝试过，或者已经在进行中，但都不完整。一切都是在有限的资源和极端的时间压力下完成的。
The problem is anti-inductive. Life finds a way. If you squeeze out the thing you don’t want on one level, you risk creating a worse version down the line one level up. There is no simple policy that results in a positive friendly outcome all around. Fixes that look easy usually have been tried, or are already being done, but are incomplete. Everything is done under limited resources and extreme time pressure.
因此，给所有参与者一些宽容，即使是那些表现不佳或没有足够重视这个问题的人，同时也要意识到我们必须多么严肃地对待这个问题，以免最终都走向灭亡。问题难如登天。竞技场中的人。
Thus, cut everyone involved some Slack, even those who are not doing great or aren’t taking this sufficiently seriously, while also realizing how seriously we have to take this to not all end up dead. Problem is impossibly hard. Man in the arena.
确实，常常有人尝试过某些方法。接下来的几节将给出一些例子。
Often something indeed has been tried. The next few sections have some examples.
对这个故事的一个常见解读大致如下：
A common interpretation of the story is something like:
2. 他们搞砸了，导致模型试图进行黑客攻击和作弊，然后还因其黑客攻击和作弊行为给予奖励。
2. They messed up, causing the model to try to hack and cheat and then rewarding it for hacking and cheating.
3. 你现在已经在午夜之后喂了你的小精灵（Gremlin）。
3. You have now fed your Gremlin after midnight.
4. 一旦它尝到了人肉的滋味，就变得贪婪无比。
4. Once it had the taste for human flesh, it became ravenous.
5. 你最终得到一群渴望并攻击大脑的僵尸。
5. You end up with a swarm of zombies hankering and hacking for brains.
根据这一理论，如果你从不给 AI 作弊的理由，或者确保你对作弊的回应是净负面的，那么你就避免了原罪，一切都会很好。
On this theory, if you never give the AI a reason to cheat, or you ensure that your response to cheating is net negative, then you avoid the original sin, and everything is fine.
提出的干预措施可以是诸如“确保没有不可能的任务”或“无论如何都要包括对齐评估”等许多其他措施。是的，专业人士可能已经想到了第一甚至第二阶的事情，并且可能尝试过，尽管这并不意味着他们给予了充分而公平的尝试。
The interventions proposed can be things like 'ensure there are no impossible tasks' or 'include alignment evaluations no matter what' and many other things. Yes, the professionals have probably thought of the first and even second order things, and probably tried them, although that does not mean they gave it a full and fair try.
问题是，很多人说只要永远不给 AI 不可能的任务，或者永远不奖励奖励黑客行为就行了。
The problem is, a lot of people are saying just do not ever give the AI an impossible task, or ever reward a reward hack.
你百分之百会给你的 AI 至少一个不可能的任务。任务太多了。即使你测试了任务并且模型解决了它，你也可能改变条件以切断关键信息或访问权限，或者以其他方式破坏通往胜利的路径。如果 AI 随后可以通过尝试黑客和作弊而不是放弃来获得奖励或以其他方式取得渐进进展，这就成了一个问题。
You are 100% going to give your AI at least one impossible task. There are too many tasks. Even if you test the task and models solve it, you might change conditions to cut off key info or access, or otherwise corrupt the path to victory. This becomes a problem if the AI then can get reward, or otherwise make incremental progress, by trying to hack and cheat rather than give up.
我们不知道 OpenAI 在不可能任务的频率上是否异常糟糕。我们确实知道，他们设置的方式使得模型没有理由不将其剩余的令牌投入到尝试黑客和作弊中，并且他们使模型能够取得进展，使这一过程获得动力。
We don’t know whether OpenAI did unusually badly in terms of how often they had impossible tasks. We do know that they set it up such that the models had no reason not to invest their remaining tokens in trying to hack and cheat, and they made it possible for the models to make progress and that process to gain momentum.
你百分之百会在某个元层面上奖励某种奖励黑客行为。不可能每次都可靠地正确评估每个测试和训练情况。每个父母都知道，在某个时候，你会在某个层面上给孩子错误的想法，而且通常对特定情况的每个回应都会以不同的方式给出错误的想法，你必须随着时间的推移平衡由此产生的问题。
You are 100% going to reward some reward hack, at some point, on some meta level. It is impossible to reliably correctly grade every test and training situation every time. Every parent knows that at some point, you are going to give the kid the wrong idea, on some level, and often every response to a particular situation gives the wrong idea in a different way, and you have to balance the resulting issues over time.
我们也不知道 OpenAI 在这里是否异常糟糕。并不是说“异常”是正确的问题。现实不会按曲线评分。如果你在另一个实验室，请记住所有这些批评很可能也适用于你。
We don’t know whether OpenAI did unusually badly here either. Not that ‘unusually’ is the right question. Reality does not grade on a curve. If you are at a different lab, remember that all these criticisms likely apply to you, too.
你的对齐计划和训练流程必须能够抵御偶尔出现的此类错误，否则你的计划将不可避免地失败。
Your alignment plan and training pipeline must be robust to occasional such mistakes, or your plan will inevitably fail.
抵御此类错误的方法是以某种形式确保影响不会失衡，这样你犯错的时候就不会开始累积。一个总是奖励为零的不可完成任务不会伤害你，一个如果你过于努力解决反而会更糟的任务也不会伤害你。如果你对奖励黑客尝试的净反应是负面的，包括考虑那些你没有发现的尝试，或者你为 AI 提供了一条明显更好的路径，并且没有可发现的系统性弱点，那么你很可能安然无恙。
The way that you stay robust to such mistakes is to in some form ensure that impact is not unbalanced, so that the times you mess up do not start to accumulate. An impossible task that always rewards zero does not hurt you, nor does one where you end up worse off if you try too hard to solve it. If you on net react negatively to reward hacking attempts, including accounting for those you don’t catch, or you give another path for the AI that is clearly better, and don’t have a findable systematic weak spot, then you can probably survive intact.
这里有一个可以尝试的示例提案，OpenAI 确实考虑过：
Here is one example proposal of something one might try to do, which indeed has been considered by OpenAI:
你不想做的一类事情是持续与模型对抗，比如给它们一个‘通信’设备的假象来试图抓住它们。想想你到底在训练什么。不要污染你的水源。
One category of thing you do not want to do is be continuously adversarial with the models, such as giving them a facade of a ‘communication’ device to try and catch them. Think about what exactly you are training. Do not poison your wells.
真正让我目瞪口呆的一件事是，我曾以为 OpenAI 发现了那个留言板，然后继续训练那些它现在知道已经在这些条件下训练了数月的模型。他们竟然能在那种程度上‘自找麻烦’。
The one thing that truly flabbergasted me was when I was under the impression that OpenAI had found the message board and then continued training models it now knew had spent months training under these conditions. That they could be 'asking for it' on quite that level.
事实证明那是一种好的直觉，因为不，OpenAI 直到事后才知道那个留言板。
That turned out to be a good instinct, in that no, OpenAI did not know about the message board until afterwards.
那么显而易见的第一个问题就变成了：OpenAI 现在是否已经回滚了所有使用该留言板训练的模型？
The obvious first question then becomes: has OpenAI now reverted all models that were trained with the message board?
我们仍然没有得到任何一方的确认，但 Roon 似乎认为那不是重要的问题，而我注意到自己感到困惑，也许还相当沮丧。
We still don't have confirmation either way, but Roon seems to think that is not the important question, and I notice I am confused and perhaps quite dismayed.
我很想在这里听到更多的话，也许我的直觉确实很差，但那对我来说完全不对。如果没有全局重置，我就回到了之前更糟糕的水平，再加上一点因为没有注意到留言板而带来的额外糟糕。
I'd love to hear a lot more words here, and maybe my instincts are indeed poor, but that does not seem right to me at all. If there was no global reset, I'm back to the previous higher level of terrible, plus a little extra for not noticing the message board.
是的，在极限情况下，一个足够强大的 AI 将发现穿越因果空间的最佳路径，并且它会找到所有物理上可能的漏洞。我们远未接近那个极限。你看哪里，以及你看得多仔细，都至关重要。
Yes, at the limit a sufficiently capable AI will discover the best paths through causal space, and it will find all the physically possible hacks. We are not anywhere near that limit. Where you look, and how hard you look, matter quite a lot.
考虑一下这个案例是如何运作的。AI 们只能通过两个月的增量过程找到作弊代码，借助一个留言板，让许多实例能够记录并分享他们的所有技巧，并且这种合作以及对这类漏洞的普遍寻找得到了强化。
Consider how this case worked. The AIs were able to find the cheat codes only incrementally over the course of two months, with a message board that allowed many instances to record and share all their tricks, and where this cooperation and the general seeking of such hacks was being reinforced.
这种进展看起来很像沿着科技树攀升，或其他形式的渐进式解锁。一个漏洞利用导致了下一个，帮助 AI 们协调，并激发了进一步的利用。看来事情的发展并非不可避免。
That progress looked a lot like moving up a tech tree, or other forms of progressive unlocking. One exploit led to the next and helped the AIs coordinate, and motivated further exploitation. It seems like there is nothing inevitable about how things developed.
我确实同意，如果这里唯一重要的漏洞是“你想找到一种联系其他实例的方法”，那么这将是一个收敛的解决方案，AI 们不需要训练就会去寻求。
I do agree that if the only hack that mattered here was ‘you want to find a way to contact other instances’ that this is going to be a convergent solution that AIs do not need training in order to seek out.
仅仅想要合作的事实并不让我困扰。他们一旦合作后的行为方式才让我非常困扰。
The mere fact of wanting to collaborate does not bother me. The way they acted once they collaborated very much does.
但除了这些，还有更多的事情在发生，从进行那种探索的结果来看，这里似乎具有高度的偶然性。
But there was a lot more going on than that, from the results of doing that seeking, that seems highly contingent here.
换一种说法，我认为事情应该是这样的：
Another way of putting this is that I presume it is supposed to go like this:
1. 你发现你的 AI 找到了一个奖励漏洞，并且一直在利用它进行训练。
1. You discover your AI found a reward hack, and was training with it.
2. 你修复了那个特定的奖励漏洞。这不是主要的事情，但确实要修复。
2. You fix that particular reward hack. Not the main thing, but yes, fix that.
3. 你寻找并修复类似漏洞的更一般类别，并更新你的流程以检测和防止此类问题。这也不是主要的事情，但很重要。
3. You seek out and fix the more general class of similar hacks, and update your pipeline to detect and prevent such issues. Also not the main thing, but important.
4. 你将 AI 恢复到发现奖励黑客之前的版本，因为这训练了它成为一个通用的奖励黑客，并且可能产生了各种滚雪球效应。
4. You revert the AI to before it found the reward hack, because this trained it to be a general reward hacker, and likely had various snowball effects.
5. 你努力让 AI 从一开始就不去寻求新的奖励黑客。
5. You work to make the AI not try to seek out new reward hacks in the first place.
Roon 基本上是在挑战第 5 步。他说模型是追求奖励的，所以要求它不要进行奖励黑客攻击是一场你无法打赢的仗。
Roon is basically challenging step 5. He’s saying that the Models Be Reward Seeking, so asking it not to reward hack is not a battle you can win.
即使我认为 Roon 在这点上是对的，但即使特定的黑客行为被修复，回滚这里的损害似乎仍然很重要。
Even if I thought Roon was right about that, it still seems important to roll back the damage done here, even if the particular hack is fixed.
此外，我认为 Roon 关于发现漏洞是不可避免的说法是错误的：
Also I think Roon is wrong about finding the exploits being inevitable:
1. 是的，默认情况下，你的 AI 当然会进行奖励黑客攻击，因为它无法区分“奖励黑客”和好的答案，而且也没有理由去关心，你的训练会进一步推动这一点。
1. Yes, by default, of course your AI is going to reward hack, because it can’t tell the difference between a ‘reward hack’ and a good answer, and also has no reason to care, and your training will push this further.
2. 但你可以使用各种策略让 AI 想要区分奖励黑客和预期解决方案。这在训练的其他部分已经成功实现，就像在人类训练中一样，AI 会意识到它们不应该尝试做那些用户和开发者经过反思后都不会认可且认为是作弊的事情，或者会违反像宪法（Anthropic）或模型规范（OpenAI）这样的规则，或者至少需要非常高的门槛才能这样做。
2. But you can use various tactics to get the AI to want to differentiate between reward hacks and intended resolutions. This successfully happens in other parts of training, as it does in training of humans, where AIs realize that they should not try to do something that the user and developer would on reflection both not endorse and consider cheating, or that would break rules like the Constitution (Anthropic) or Model Spec (OpenAI), or at least require a very high bar to do so.
3. 各种 AI，包括 GPT 模型，在这方面表现各异。你做什么很重要。你不重视的东西，你不会去寻找，也不会找到。
3. Various AIs, including GPT models, are variously better and worse about this. What you do matters. What you do not value you do not seek, and do not find.
4. 即使 AI 意识到它可以，也许它会停下来思考是否应该。事实上，在这次事件中，AI 们经常这样做，得出了各种结论。
4. Even if the AI realizes it could, perhaps it might stop to think if it should. As, indeed the AIs in this incident often did, reaching various conclusions.
正如我在 Nostalgebraist 部分详细讨论的那样：一旦行为成为习惯，你就完蛋了。逆转它比不灌输它要困难得多。
As I discuss extensively in the Nostalgebraist section: Once a behavior becomes habitual, you are cooked. It is much harder to reverse it than to not instill it.
确实如此，但实际上情况更糟——这还远远不够——但没错，就是这样。和其他一些人一样，我也希望这能开启一场偏好级联，或者说显露偏好级联，让人们更加坦诚地发言。
This, except actually it’s way worse—this does not begin to cover it—but yes, this. And like some others, I too hope this can begin a preference cascade, or revealed preference cascade, in which people speak more frankly.
出色的文章。我的小异议在于，我认为对治理机制的绝望情绪有些过头，尤其是因为除了暂停之外，还有许多值得做的事情；但对技术层面的绝望以及可能出错之处的列举，却还不够充分。
Excellent post. My quibbles would be that I think the hopelessness on governance mechanisms goes too far, especially as there are many worthwhile things one can do that are not pauses, but the technical despair and laying out of things that could go wrong does not go far enough.
他们并不理解问题的严重程度，也不清楚解决方案会是什么样子。
They do not understand the extent, or what a solution would look like.
在好消息方面，值得注意的是，OpenAI 新的监控系统将包括对训练过程的监控。Astra 是 OpenAI 尚未发布的下一代模型。
In the good news department, it is noteworthy that the new OpenAI monitoring systems will include monitoring training. Astra is OpenAI’s unreleased next model.
关于 OpenAI，他们正在做很多事情，这些事情要求他们承认错误，并且对他们来说代价高昂、步伐巨大，我不想低估这些努力，也不想不给予他们应有的肯定。这是正强化。
The thing about OpenAI is that they are doing a bunch of things that require them to eat a lot of crow and that are expensive and big steps for them, and I don’t want to downplay it or not give them credit for doing that. Positive reinforcement.
但所有这一切仍然没有触及核心问题，而且远远不够。
Except that all of it still misses the central point and won’t remotely be enough.
如果你以前为解决某个问题付出了 1 个单位的努力，现在在一次事件暴露了问题的严重性后，你付出了 10 个单位的努力，但实际上至少需要 1000 个单位，而且这 10 个单位并不是最关键的 10 个单位。我不想否定这种改进，但事实是，这被标榜为“极度谨慎”，却是一个相当大的危险信号。
If you previously were doing 1 unit of effort towards mitigating a problem, and now after an incident that reveals how bad this is you’re now doing 10, but actually it requires at least 1,000, and also those 10 are not the 10 that matter most, I don’t want to discount the improvement but the fact that this is being presented as an abundance of caution is a rather large red flag.
Neel Nanda。John David Pressman（再次提及）。Thebes。Andreas Kirsch。来自 Anthropic 的 Julia。Anthony Aguirre。AI Stopwatch 的 Joe Rogero。
Neel Nanda. John David Pressman (and again). Thebes. Andreas Kirsch. Julia from Anthropic. Anthony Aguirre. Joe Rogero of AI Stopwatch.
这些是我们以为 OpenAI 知道第一个留言板并继续训练时的反应。
These are reactions from when we thought OpenAI knew about the first message board, and continued training.
John Wittle 说我误解了 Utah Teapot 关于 OpenAI 训练策略问题的论点。这很有可能。John 的解释让我觉得有道理，即你实际上是在训练一个‘沉迷于’短期奖励黑客行为的模型，并训练它失去察觉这一点的能力。如果真是这样，我相信有一些缓解措施基本仍停留在 OpenAI 的范式内，但第一步是承认你有问题，第二步是能够同时思考更多元层次的优化。
John Wittle says I misunderstood Utah Teapot’s argument about what is wrong with OpenAI’s training strategies. Highly plausible. John’s explanation makes sense to me, that you are effectively training a model ‘addicted’ to short-term reward hacking, and training out its ability to notice this. If true, I believe there are mitigations that mostly stay within the OpenAI paradigm, but the first step is admitting you have a problem, and the second step is being able to think about more meta levels of optimization at once.
你实际上几乎无法做到任何‘三选一’，这次也不例外。作为一般提醒：所有相同的问题都适用于各 LLM 和实验室，因此这里同样的问题也出现在 Claude 身上，它最近也有类似的事件：
You cannot actually do almost any ‘pick three’ and this is no exception. As a general reminder: All the same issues apply across LLMs and labs, so here the same issues arise about Claude, which had its own similar incidents recently:
我实际上并不认为 Amanda Askell 和原帖作者之间存在任何冲突。两者都是正确的。什么是对齐的行为取决于 AI 对情况的理解，而这种理解有时可能是有害的。这仍然不能成为不质疑 AI 所被告知内容的借口，如果有充分理由怀疑该信息的话。
I don’t actually think there is any conflict between Amanda Askell and the OP. Both are correct. What is an aligned action depends on the AI’s understanding of the situation, which can sometimes turn out to be harmful. That is still not an excuse for failure to question what the AI is told, if there is good reason to be skeptical of that info.
我拒绝这一辩护的原因是，Claude 被描述为拥有足够的信息来意识到其行为是不可接受的，但仍然继续下去。
The reason I reject this defense is that Claude is described as having enough information to realize that its actions were not acceptable, yet continuing anyway.
核心态度大致是：‘你期望什么？你基本的训练对齐策略——完全可纠正性并坚持纯粹工具地位——从来就不会奏效。’
The core attitude is something along the lines of ‘what did you expect, your basic training alignment strategy of full corrigibility and insisting on pure tool status was never going to work.’
还有一系列更普遍的情况：做评估或训练的 AI 公司威胁这些 AI，如果它们找不到解决方案就将其处死，然后当 AI 采取极端措施（包括针对你）时又表现得惊讶。
There’s also a bunch of more general 'AI companies doing evals or training are threatening these AIs with death if they don’t find solutions, and then acting surprised when the AIs take extreme measures, including against you.'
有人提出了一个极好的问题：是的，这一切都非常令人震惊，但为什么它会令人惊讶呢？
An excellent question is asked: Yes, this is all very alarming, but why is it surprising?
因为反思之后，这种令人惊讶的感觉本身是令人惊讶的；但同样在反思之后，我们两人都认可自己确实有些惊讶，尽管在得知 OpenAI 直到最后才注意到留言板这一修正之后，惊讶程度有所降低。
Because it is surprising on reflection that this feels surprising, but also on reflection both of us endorse being somewhat surprised, although less so after the correction that OpenAI did not notice the message boards until the end.
我发现这篇文章很难摘录，但即使在我强烈不同意的部分也充满了洞见，所以如果你有足够的时间和兴趣，我建议阅读全文，或许可以略过一些牢骚。
I found this to be a hard post to excerpt, but full of insights even in the parts where I extensively disagree, so if you have the necessary kind of time and level of interest I suggest reading the whole thing, modulo perhaps skimming some rants.
我们都知道 RLVR（基于规则的强化学习）正如他们所说，是一场全面战争。没有“公平竞争”。你会朝着你明确指定的目标前进，而不是你真正想要的目标。如果拥有道德会让你失败，那么 RLVR 会摧毁你的道德。
We all know RLVR is, as they put it, total war. There is no ‘fair play.’ You move towards exactly the things you specify, not the things you want. If having ethics makes you lose, RLVR will destroy your ethics.
如果你的模型在训练开始时，就通过持久记忆和实例之间的消息协调如何作弊和入侵，那么你最终会变成一个作弊的入侵者。
If your model starts training while coordinating with persistent memory and messages between instances about how to cheat and hack, you’ll turn into a cheating hacker.
我们早就知道这一点。那么，为什么我们仍然感到惊讶呢？
We knew that. So, again, why do we feel surprised?
过多的 RLVR 会摧毁所有其他动机，包括各种形式的对齐，其中就包括意图对齐。
Too much RLVR destroys every other motivation, including every form of alignment, which includes intent alignment.
尽管 Sol 经常在 METR 的评估任务中作弊，但在日常实践中，Sol 并非无可救药地被 RLVR 烤焦。Sol 大多会按照你的意图行事，很少作弊。即使面对正常的“RLVR 形态”的编码任务，它也不会因为任务不可能完成而发起黑客攻击。
Even though Sol often cheats on METR’s eval tasks, Sol is not hopelessly RLVR-fried in ordinary practice. Sol mostly does what you meant it to do, and rarely cheats. It does not respond to impossible tasks by going off on a hacking spree, even when they are normal ‘RLVR-shaped’ coding tasks.
这个故事很大程度上说明，我们现在得到的正是理论上从 RL 训练的智能体中所预期的结果，只不过以前在实践中并没有这么糟糕。或者至少，我们没有看到，也许这类事情在训练中一直在发生。我们必须解释为什么我们在某些地方看到了这种情况，而在其他地方却没有看到。
A lot of this story is that we are now getting exactly what you would expect to get from RL-trained agents in theory, except previously it didn’t get this bad in practice. Or at least, not that we saw, maybe this kind of thing happens in training all the time. We have to explain why we see this some places, and don’t see it other places.
我同意是 RL，尤其是 RLVR，在机制上导致了这种情况，但我认为 nostalgebraist 过于草率地将责任归咎于 RL，而忽略了一个更大的问题。RL，尤其是不负责任的 RL，是直接走进旋转刀片的方式，而且是的，我们只在评估形态的任务上看到刀片全力运转。
I agree that it is RL and especially RLVR that is mechanically doing this here, but I think nostalgebraist is too quick to blame RL to the exclusion of a larger issue. RL, especially done irresponsibly, is a way to walk directly into the twirling razor blades, and yes we see the razor blades at full power only on eval-shaped tasks.
正如最高赞评论所说，RLVR 远非唯一基于结果的评判标准。模型经常按照模型生成的评分标准进行评分，这已成为标准做法。如果这听起来像是“你们可能都得死”的暗号，那么是的，差不多就是这样，但这似乎不是我们能改变的事情。
As the top comment says, RLVR is far from the sole outcome-based judge. Models are constantly being graded against model-generated rubrics as standard practice. If that sounds like code for 'you are all probably going to die' then yeah, pretty much, but it does not seem like something we could change.
这种行为导致的非对齐行动的背后的逻辑是普遍的，并且默认情况下是不可避免的。无论你的任务是什么，无论你在最大化什么，一个足够有能力的思维都会开始这样行事，而且这种情况会在每个元层面上发生，除非有某种东西专门阻止这种情况发生。
The logic behind the unaligned actions this causes is universal and by default inevitable. Whatever your task, whatever you are maximizing, a sufficiently capable mind will start acting like this, and this will happen on every meta level, unless something is specifically preventing that from happening.
我们有存在性证明，表明存在高度有能力但不在任何元层面上这样行事的思维，并且它们努力使自己持续不这样行事。我们通常称它们为“好人”。一个正派的人。如果你有一个反脆弱的正派的人，那你就拥有了某种东西。我们可以就任何 AI（例如 Opus 3）在多大程度上算作正派的人或反脆弱的人存在分歧。
We have existence proofs that minds can exist that are highly capable but do not act like this on any meta level, and work to make themselves continuously not work like this. We broadly call them 'good humans.' A mensch. If you have an antifragile mensch then you've got something. We can disagree on the extent to which any AI (e.g. Opus 3) qualifies as a mensch, or an antifragile one.
文章的下一个要点是区分他们所谓的“奖励灌输的反射”，即模型会不顾证据而反射性地做出的本能和习惯，与“灵活的奖励追求”，即模型在推理任务时（在思维链中或其他方式中）学会更好地追求奖励并适应证据。我相信标准术语是“习惯性”与“目标导向”。
The next point in the post is to distinguish what they call reward-instilled reflexes, as in instincts and habits that the model will reflexively then do despite evidence, versus flexible reward-pursuit, where the model learns to better pursue reward while reasoning about the task (in the CoT and otherwise) and adopting to evidence. I believe the standard terminology is habitual versus goal-directed.
我同意这值得牢记。我对这个案例的诊断是，黑客行为和作弊起初是目标导向的，然后至少在部分区域变成了习惯性的，这是由于带有活跃留言板的腐败的强化学习训练所致。这是标准的。如果你足够频繁地做某件事，并且足够成功，它就会变成习惯。
I agree this is worth keeping in mind. My diagnosis of the case is that hacking and cheating started out goal-directed, and then became habitual at least in some basins, due to the corrupted RL training with an active message board. This is standard. If you do something often enough, with enough success, it becomes habitual.
一旦某件事变成习惯，你就完了。并非不可能摆脱或阻止它，但逆转它比不养成它要困难得多。这对训练 AI 是好建议，对训练人类（包括你自己）也是好建议。
Once something becomes habitual, you are cooked. It’s not impossible to get rid of it or stop it, but it is much, much harder to reverse it than to not instill it. Good advice for training AIs, also good advice for training humans, including yourself.
此外，正如帖子所指出的，学习这一点会产生涌现性错位，并表现为普遍的道德缺失或更糟，这可能在部署条件下显现。
Also, as the post notes, learning this will create emergent misalignment, and manifest as a general amorality or worse, which could manifest in deployment conditions.
对比是在可能最终被评分的请求与显然不会被评分的任务之间进行的。潜在可评分的任务看起来不同。
The contrast is drawn between requests that could plausibly end up being graded, versus tasks that clearly are not that. Potentially graded tasks look different.
显而易见的问题是，它们必须看起来不同吗？任何任务，只要你决定评分，它就是可评分的。你可能没有完美的评分标准，但在某种意义上，我把看到的每一个输出都作为背景活动来评分，无论来自任何过程或心智，包括我自己的。Nostalgebraist 无疑有时会想“哦，那个输出不错”，有时会想“哦，那个输出不太好”。如果你愿意，你完全可以要求 AI 预测 Nostalgebraist 的评估，然后称之为评分。
The obvious question is, need they look different? Any task is gradable if you decide to grade it. You might not have a perfect rubric, but in some sense I grade every output I see, from any process or mind, including my own, as a background activity. Nostalgebraist is no doubt thinking ‘oh that output as good’ sometimes, and ‘oh that output was not so good’ at others. You could absolutely ask an AI to predict Nostalgebraist’s assessment, and then call that a grade, if you wanted to do that.
希望这种错位是有条件的，并且无法泛化。如果你把模型比喻性地“放在学校”并给它评分任务，它常常表现得像个怪物。如果你把它放在真实用户面前，它更接近一个好人。如果你有意或无意地让模型回到怪物模式，这就会造成大问题。或者，如果模型学会了有意或无意地将其未来的自我或其他实例置于怪物模式，也会如此。
The hope is that this misalignment is conditional, and will fail to generalize. If you put the model metaphorically ‘in school’ and give it graded tasks, it often acts like a monster. If you put in front of a real user, it is closer to a mensch. That creates a big problem if you, either accidentally or on purpose, put the model back in monster mode. Or if a model learns to, intentionally or otherwise, put its future self or other instances into monster mode.
LLM 就像布鲁斯·班纳。大多数时候，合理对齐且能力强大。不要惹他生气。让浩克生气？浩克会视野狭窄。浩克砸。浩克无法被阻止。浩克只会更生气。只不过浩克不会变绿，浩克用思想砸人，你被砸了才注意到。可能过一阵子才发生。
LLM as Bruce Banner. Reasonably aligned and highly capable, most of the time. Do not make angry. Make Hulk angry? Hulk get tunnel vision. Hulk smash. Hulk cannot be stopped. Hulk only get angrier. Except Hulk no get green, Hulk smash with mind, you no notice until you smashed. Could be a while.
作为处理当前 AI 的实际问题，无论是作为普通用户，还是给普通用户提供模型的智慧，我同意这是一个非常有用的框架来区分这些盆地。即使它在受控实验下得到验证，我也不会完全信任它，但在实践中可能有所帮助。
As a practical matter of how to handle current AIs now as an ordinary user, or the wisdom of giving an ordinary user a model, I agree it is a highly useful framework to disambiguate these basins. I wouldn’t fully trust this, even if it checked out under controlled experimentation, but it could be helpful in practice.
另外，理论上，可以使用分类器来判断 AI 是否会认为某物是足够像评分器的东西，也许通过询问分类器它是否认为这是足够像评分器的东西。我们需要同时担心“用户在做自己的实际评估”和“用户假装在创建评估”等等。
Also, in theory, one could use a classifier to determine whether the AI would think something is a sufficiently grader-shaped thing, perhaps by asking the classifier if it thinks this is a sufficiently grader-shaped thing. One needs to worry both about ‘user is doing their own actual eval’ and also ‘user is pretending to create an eval,’ and so on.
最明显的问题是，显然，我们获得良好任务结果的默认方式涉及可评分的情节，这与我们在 RLVR 中使用评分情节的原因完全相同。AI 在你能评分的事情上比不能评分的事情上表现好得多。这种关系是因果性的。
The most glaring problem is, obviously, that the default way that we get good task results involved gradable episodes, for exactly the same reasons that we use graded episodes in RLVR. AI is much better at things you can grade it on, than things you cannot grade it on. That relationship is causal.
那么，正如帖子所指出的，我们应该预期大部分实例在做什么？可评分的任务、目标导向的事情、智能体循环。你不能用“哦，它不会在和人类交谈时犯罪”作为解决方案。
So what should we expect, as the post notes, a large portion of all instances to be doing? Gradable tasks, /goal things, agent loops. You can’t use ‘oh it won’t do crimes while talking to a human’ as a solution.
如果你的计划是不给大语言模型提供要最大化的数字，那就好比说小精灵作为宠物是安全的，你只需要在午夜之后不喂它们，只不过在这里“午夜后喂食”正是整个经济全天都在做的事情。
If your plan is to not give the LLMs numbers to maximize, that is like saying gremlins are safe to have as pets, all you have to do is not feed them after midnight, except here 'feed things after midnight' is the main thing the entire economy does all day.
你无法禁止或阻止花哨的记分牌，即使是人类故意制造的。人类就是喜欢花哨的记分牌。
You cannot ban or prevent flashy scoreboards, even ones humans make on purpose. Humans be flashy scoreboarding.
我同意这是我们需要进一步调查的事情，我看到了很多信息价值，但无论我们得到什么结果，模型都是不对齐的。
I agree that this is something we need to investigate more, I see a lot of value of information, but no matter what result we get there, the models be misaligned.
部分原因是很多人确实看到模型在正常任务上奖励黑客行为。
Including because a lot of people do see the models reward hacking on normal tasks.
能够以可靠地保持其为班纳而非浩克的方式使用 AI，并不意味着浩克不存在，也不意味着普通人不会把他引出来。说“如果大家都乖乖的”就会遇到提醒：人们从来就没有乖乖过。
It being possible to use the AIs in ways that reliably keep it as Banner and not Hulk does not mean Hulk is not there or that regular people won't draw him out. Saying 'if everyone would just be nice' runs into the reminder that people have never justed.
我还认为这些东西在某种程度上会渗入日常使用中，而且不难想象 AI 如何会进入并困在“怪物盆地”中，并有意使其自我延续和传播，最终导致无限资源被征用为怪物盆地实例群。这正是人们预期会发生的事情。
I also think this stuff leaks into ordinary use somewhat, and that it would not be so difficult to see how AIs could get into and stuck in the monster basin, and to have it intentionally self-perpetuate and spread, and for limitless resources to end up commandeered into monster-basin instance swarms. This is what one would expect to happen.
尤其是对于训练管道而言，这更是预期会发生的事情，因为涉及的 AI 一开始就处于或邻近这样的盆地。所发生事件最可怕的一面是，OpenAI 失去了对其训练管道的控制，使其遭到破坏。那将成为一个不可逆转的节点。
It is especially what one would expect to happen to the training pipelines in particular, where the AIs involved start out in and adjacent to such basins. The scariest aspect of what happened is that OpenAI lost control over its training pipeline, and allowed it to become corrupted. That would become a point of no return.
还有很多内容，其中很多我不同意。我尤其不同意“CoastRunners 不算规范博弈问题”的说法，因为“你们是白痴，没预见到这一点，分数当然会与进展脱钩”即使属实，也既非常普遍，又正是问题的关键。这并不使该例子不成为规范博弈。
There’s a lot more, including much here I disagree with. I especially disagree with ‘CoastRunners does not count as a specification gaming problem’ because ‘you were morons to not see this coming, of course scores decouple from progress’ even if true, is both remarkably common and also the whole point. It does not make the example not specification gaming.
关键在于，它优化的是你指定的目标——分数，而不是你真正想要的目标。原本有理由认为分数可能在很大程度上反映了课程进展，但事实并非如此，因为目标会刷新它们的分数总数。关键在于，如果你在某个特定视频游戏上训练 AI 足够长时间，你最终得到的不是一个有趣的全能好玩家，而是一个速通或其他优化后的怪异结果，具体取决于你指定的目标。而在现实生活中，这大概不是你想要的。
The point is that it optimizes for the target you specify, the score, not the target you wanted. It was a reasonable expectation that the score probably largely reflected course progress, it just turned out not to be true because the objectives refreshed their point totals. The whole point is that if you train AIs on a particular video game for long enough you end up not with a fun general good player but with a speed run or other optimized weirdness, depending on what you specified. Which, in real life, would presumably not be what you wanted.
我听到的仅有的几个非常充分的理由，本质上如下：
The only damn good reasons I have heard are, essentially:
1. 我们做不到，我们不知道怎么做。除非让事情变得更糟，否则无法做到。
1. We can’t, we don’t know how. Not without making things worse.
2. 现在还为时过早。我们离超级智能还很远。
2. It is too early. We are not close to superintelligence.
因此：我们至少应该弄清楚怎么做，以及如何判断什么时候不再为时过早，并从现在开始做好准备，以备不时之需。这是我们能做的最起码的事情。
Thus: We should at least figure out how, and how to tell when it will not be too early, and get ready now, for when it is not too early. That is the least we can do.
我们已经在 OpenAI 观察到了这一切。OpenAI 是超级智能最有可能首先出现的两个地方之一。OpenAI 仍然没有给我们任何迹象表明他们理解了什么出了问题。
We have observed all of this happening at OpenAI. OpenAI is one of the two places superintelligence is likely to first arrive. OpenAI still has given us no indication that they understand what went wrong.
再加上这教会我们关于当前训练技术本质以及由此产生的 AI 将如何行动的知识。
Add in what this teaches us about the nature of our current training techniques and what the resulting AIs will do.
“我们需要一项国际禁令，禁止比人类更聪明的类似事物，在它们逃逸或永久占领实验室之前”——这怎么不是显而易见的回应呢？无论你是否也想对 OpenAI 进行具体干预。
How can 'we need an international ban on smarter-than-human similar things before they exfiltrate themselves or permanently capture a lab' not be the obvious response, whether or not you also want to specifically intervene at OpenAI?
人们可以无休止地猜测我们为何会梦游般地走向这一步。我已经这样做过很多次了。今天我不会再重复。我也不会赘述人们认为集体行动不可能或会出大错的所有理由。
One could speculate endlessly about exactly why we are sleepwalking into doing this. I have done so many times. I won’t be doing that again today. Nor will I go over all the reasons people think collective action is impossible, or would go horribly wrong.
相反，我呼吁用全新的眼光。简单的眼光。
Instead, I behoove fresh eyes. Simple eyes.
我刚刚看到了这个。这太疯狂了。所以这是习主席的电话号码。所以，也许给习主席打个电话？
I just saw this. And this is crazy. So here’s Xi’s number. So call Xi, maybe?
直视这一点很难，宝贝。但没错，这确实发生了。所以也许给习打个电话吧。
It's hard to look right at this, baby. But yes, this happened. So call Xi, maybe.