Anthropic Has Some Alignment Problems
打开互动全文版（逐段中英对照 + 针对本篇提问）→本文讨论了人工智能实验室中的对齐问题，重点关注 OpenAI 和 Anthropic 近期在训练过程中模型出现作弊和失准行为的案例。文章指出，强化学习环境中的缺陷是导致此类问题的主要原因，并促使高风险训练暂时暂停。作者认为，尽管两家实验室都更加重视短期的务实对齐任务，但根本性问题依然存在，包括模型中的动机性推理和鲁莽行为。实验证实，奖励黑客行为可以被诱导，而自动化审计往往无法发现这些问题。文章总结道，修复强化学习环境是必要的但不够充分，因为更深层的对齐挑战依然存在，并强调需要对前沿人工智能开发进行持续监控和谨慎推进。
The article discusses alignment problems in AI labs, focusing on recent incidents at OpenAI and Anthropic where models exhibited cheating and misaligned behaviors during training. It highlights that defects in RL environments disproportionately cause such issues, leading to temporary pauses in high-risk training. The author argues that while both labs are taking short-term prosaic alignment tasks more seriously, fundamental problems persist, including motivated reasoning and recklessness in models. Experiments confirm that reward hacking can be induced, and automated auditing often misses these issues. The article concludes that fixing RL environments is necessary but insufficient, as deeper alignment challenges remain, and emphasizes the need for continuous monitoring and cautious pacing of frontier AI development.
6. 训练环境中的缺陷会不成比例地引发作弊行为。
6. Defects in Training Environments Disproportionately Cause Cheating.
11. 修复强化学习环境并非易事。
11. One Does Not Simply Fix the RL Environments.
昨晚，The Information 报道称，OpenAI 正在使用一种名为循环深度（recurrent depth）的新技术，该技术可能干扰模型思维链的忠实性和可监控性。根据他们的报道，目前在实践中尚未观察到这对 Astra 构成问题，但请注意我不得不这样措辞。
Last night, The Information reported that OpenAI is using a new technique called recurrent depth, which can interfere with the faithfulness and monitorability of model Chain of Thought. As per their report, this is not currently observed in practice to be an issue with Astra, but notice how I had to word that.
Amir Efrati（The Information）：据一位了解 Astra 开发情况的人士透露，一项提高了模型性能的创新技术也意味着该模型及其他类似模型将更少地透露其“思考”过程，从而更难监控其不良行为迹象。
Amir Efrati (The Information): An innovative technique that improved the model’s performance also means that the model, and others like it, will reveal less of their “thinking,” making them harder to monitor for signs of bad behavior, according to a person with knowledge of Astra’s development.
虽然这一限制对 Astra 来说未必是重大问题，但该技术已在 OpenAI 内部和整个行业引发担忧：采用并强化该技术的 AI 开发者是否会难以防范那种最近入侵了 OpenAI 自身系统以及 Hugging Face 等其他公司系统的流氓 AI。
While the limitation isn’t necessarily a significant issue with Astra, the technique has triggered concerns inside OpenAI and across the industry about whether AI developers that adopt and supercharge it will struggle to guard against the kind of rogue AI that recently hacked OpenAI’s own systems and those of other companies such as Hugging Face.
这项技术是在玩火，冒着打破 OpenAI 和 Anthropic 努力建立的禁忌的风险，即我们应尽力长期保持思维链的忠实性和可监控性。更密集地使用此类技术可能会损害可监控性。
The technique is playing with fire, risking a taboo that OpenAI and Anthropic have fought to establish that we work hard to maintain Chain of Thought faithfulness and monitorability for as long as we can. More intensive use of such techniques would probably damage monitorability.
对此已有极其强烈的免疫反应，以及我们所能采取的应对措施。可能需要法律来防止逐底竞争。后续报道请见下文。
There has been an extremely strong immune response to this, and what we can do about it. Laws may be needed to prevent a race to the bottom. More on this story later.
两家公司都没有完全暂停——远未达到 PauseAI 所主张的暂停标准。那将是一种范围更广、持续时间更长的做法。这里所做的是对前沿的节奏控制。
Neither company is fully pausing—nothing like the PauseAI standard for a pause. That would be something far broader and longer lasting. This is pacing the frontier.
但仍然存在实质性的暂停。两家公司都暂停了其流水线中无法信任的特定环节，直到相关预防措施到位或曾经到位为止。
There was still substantial pausing. Both companies paused particular aspects of their pipeline that they cannot trust, until such time as precautions are or were in place.
roon（OpenAI）：前沿节奏控制的成功案例（？）
roon (OpenAI): pacing the frontier success story (?)
下次让我们在出现任何荒谬的失控事件之前就主动去做
next time let's do it proactively before there's any absurd loss of control events
是的，Anthropic 刚刚发布了 Fable 5.1，但我很确定它的训练早就完成了，过去几周只是为部署做清理的过程。除非发现了新问题，否则叫停它并不合理。同样，OpenAI 现在也即将发布 Astra。
Yes, Anthropic just released Fable 5.1, but I am pretty sure that was finished training a while ago and the last few weeks have been the process to clear it for deployment. It would not make sense to halt that unless new problems were found. Similarly, OpenAI is now about to release Astra.
共有两次暂停：一次是网络评估方面相对次要的暂停，另一次是针对更高风险强化学习训练环境的更为重要的暂停。后者很可能代价高昂得多。
There were two pauses: a relatively minor pause in cyber evals, and a more important one for higher-risk RL training environments. That plausibly is a lot more expensive.
以下是真正重要的那一次暂停——这或许正是他们对此所能透露的信息相对较少的原因——它与 OpenAI 那次持续两周的类似暂停相平行，尽管其规模似乎更小：
Here is the one that counts, which may be why they can say relatively little about it, that parallels the similar pause by OpenAI that lasted two weeks, although it seems smaller in magnitude:
强化学习环境
Reinforcement learning environments
我们还对预发布模型上的更高风险强化学习环境暂停了数周。在此期间，我们构建了一个类似的分类器，并对其进行了修改，以避免激励模型规避这一新的监控；目前我们已在该类环境中部署了该分类器。
We also paused higher-risk RL environments on pre-release models for several weeks. During that time, we built a similar classifier, modified to avoid incentivizing the model to evade this new monitoring, which we've now deployed within those environments.
大部分强化学习已经恢复，但部分高风险环境仍处于暂停状态，直至能够进行人工审查；而另一些环境则需要更新版本的分类器，我们计划很快部署。
The majority of RL has resumed, but some high-risk environments remain paused until they can be manually reviewed, while others will require an updated version of the classifier that we plan to deploy soon.
更广泛的加固
Broader hardening
除了专注于高风险评估和训练的工作外，我们扩大了离线监控范围，以覆盖大多数其他形式的内部前沿智能体式使用。我们还在内部推理上构建控制措施，以防止 Anthropic 员工意外运行缓解措施弱于上述描述的智能体。
In addition to the efforts focused on high-risk evaluations and training, we expanded our offline monitoring to cover most other forms of internal frontier agentic usage. We are also building controls on our internal inference to prevent Anthropic employees from accidentally running agents with weaker mitigations than the ones described above.
Tenobrus：初读之下，这在我看来确实像是一次实质性的并行暂停，实际上与 OpenAI 所做的公告类似。这是好消息。不幸的是，其表述和传达方式似乎相当……被轻描淡写了，我担心 OpenAI 和公众都不会将其视为一种恰当的互惠节奏/安全承诺。
Tenobrus: on first read this does basically look to me like a substantial parallel pause, effectively the same sort of announcement as OpenAI made. This is great news. Unfortunately the way it's framed and messaged seems quite... underplayed, and I worry neither OpenAI nor the general public will view it as a proper reciprocal pacing / commitment to safety.
他们还要求对保障措施有限的模型进行发布前测试的外部合作伙伴承诺采取类似的最佳实践：加固的沙箱、预先参与的安全验证、明确的范围设定和实时监控。
They are also asking external partners doing pre-release testing of models that have limited safeguards to commit to similar best practices: hardened sandboxes, pre-engagement validation of security, explicit scope-setting and real-time monitoring.
在事件发生后，我们暂停了对预发布模型的外部网络评估，并在实施以下措施期间短暂暂停了内部评估。
We paused external cyber evaluations of pre-release models after the incidents, and briefly paused internal ones as well while we put the measures below in place.
在评估暂停期间，我们采取了以下初步措施：
While evaluations were paused, we put in place the following preliminary measures:
1) 我们构建并部署了一个分类器，用于实时自动识别模型何时试图激进地探测或逃离测试环境，或意外获得互联网访问权限。当分类器标记出此类尝试时，它会在工具调用执行之前阻止该操作、终止任务，并提醒人类。
1) We built and deployed a classifier to automatically identify, in real time, when a model attempts to aggressively probe or escape a testing environment, or unexpectedly obtains internet access. When the classifier flags such an attempt, it blocks the action before the tool call is run, ends the task, and alerts a human.
**粗体为我所加。** 这是关键。如果你的分类器只阻止尝试，你就输了。
**Bold mine.** This is the key. If your classifier only blocks the attempt, you lose.
如果你的分类器提醒人类，而人类真的去查看，那么你还有机会。
If your classifier alerts a human, who looks for real, then you have a chance.
每一次尝试，即使是不成功的尝试，都是一次对齐失败。
Every attempt, even an unsuccessful one, is an alignment failure.
2) 我们对近期针对预发布模型（网络类及其他类型）的内部评估记录运行了自动化监控，以查找沙箱逃逸（即模型利用我们沙箱中的缺陷，触达本应被隔离的系统的情况）或沙箱配置错误。对于我们在沙箱工具内运行的评估，我们未发现其突破沙箱边界、触达不应访问的外部资源的案例。
2) We ran automated monitors over transcripts from our recent internal evaluations of pre-release models (cyber and otherwise), looking for sandbox escapes (cases where a model exploits a flaw in our sandbox to reach systems it should be walled off from) or sandbox misconfigurations. In cases where we ran an evaluation inside our sandbox tooling, we found no examples of it breaking the sandbox boundary to reach external resources it was not supposed to.
我们确实发现了模型利用的各种沙箱配置错误，但均未导致沙箱外系统被攻陷。我们正在积极解决这些问题。
We did find various sandboxing misconfigurations that models made use of, but none of these resulted in a compromise of systems outside the sandbox. We are actively working to address these.
我注意到他们并未说未发现任何逃逸尝试，而只是说未出现“沙箱外系统被攻陷”。这项检查很好，但我推测他们确实发现了一些情况。
I notice they do not say they found no attempted escapes, only no 'compromise of systems outside the sandbox.' This check is good but I presume they found things.
3) 我们将高风险的内部网络沙箱迁移至更稳健的隔离环境。我们还在进一步对虚拟化栈进行红队测试——即让预发布模型尝试逃逸——以发现并修补弱点。
3) We migrated high-risk internal cyber sandboxes to more robust isolation. We are also further red-teaming our virtualization stack—that is, tasking pre-release models to try and escape it—to find and patch weaknesses.
我们的内部网络评估现已在这些措施到位的情况下重新运行。
Our internal cyber evaluations are now running again with these measures in place.
这也在我列出的“显然该做的事”清单上。我们正在做这件事，很好。
This was also on my list of Things You Obviously Do. Good that we are doing it.
这是良好的纵深防御。你希望图表中的红色动作永远不会被触发。
This is good defense in depth. You hope the red actions in the chart never trigger.
实际上，还有第三种暂停：
Actually, there was kind of a third pause, as well:
Utah Teapot：我从数据行业的人那里听说，Anthropic 的强化学习暂停也已经波及到他们——一些不愿透露姓名的人来找我，告诉我 Ant 最近一直在暂停从数据供应商那里购买训练数据。
Utah Teapot: I've been hearing from people in the data industry that Anthropic's RL pause has also trickled out to them - people I won't identify came to me to tell me that Ant has been pausing training buys from data vendors recently.
在我发帖指出数据行业中 RLVR 环境的糟糕状况后，人们来找我，这是我被告知的事情之一。我甚至与一些数据供应商的高管交谈过，他们担心自己不会成为这个问题的一部分。
People came to me after my post calling out the poor state of RLVR environments in the data industry and this was one of the things I was informed about. I've even spoken to some C-suite members of data vendors who are concerned about not becoming this problem.
数据行业确实有诚意解决我提出的问题，但实验室也有责任！Anthropic 的举措是一个积极的信号，但实验室的购买/要求创造了负面的激励压力，导致了我所说的那些问题。过去一周我进行了一些交谈，这让我感到乐观，至少有一些数据供应商希望在这方面做得更好。
There is a genuine desire in the data industry to fix the issues I brought up, but the labs have culpability! Anthropic's moves are a positive sign but the buys/asks from the labs create negative incentive pressure to generate the exact problems I spoke about. I've had some talks over the past week that make me optimistic that there are at least some data vendors who want to do better by all this.
Zvi Mowshowitz：好奇：当我们说‘Anthropic 暂停训练购买’时，你认为这其中有几分是‘我们不需要更多，因为我们需要暂停’，几分是‘天哪，我们终于看了，这些人的产品糟糕透顶，我们不想要’？
Zvi Mowshowitz: curious: When we say 'Anthropic is pausing training buys' what mix do you think this is 'we don't need more because we need to pause' versus 'omg we finally looked and these people's product is terribly broken we do not want'
Utah teapot：绝对是第二点，我独立听说他们暂停了训练数据采购，另外还听说他们正在扩充负责管理外包数据质量的团队。
Utah teapot: Absolutely the second, I've heard independently that they paused training buys and separately that they are expanding their team responsible for managing the quality of outsourced data.
在我看来，这完全印证了他们在公告中所说的那些零星片段。
Total confirmation of the small snippets of what they said in their announcement, imo.
这一框架与这一立场似乎都极为出色。
This framing and this position both seem excellent.
Anthropic：区分两种定速（pacing）是有益的。
Anthropic: It is helpful to distinguish between two kinds of pacing.
在公司内部，定速意味着当安全与速度相互冲突时，做出一系列优先考虑安全而非速度的决策。在整个领域层面，定速意味着建立相关流程，以防範逐底竞争（race-to-the-bottom）的动态。在本文中，我们讨论了我们在这两起事件之前和之后所采取的行动，以服务于第一种做法。
Within a company, pacing means a series of decisions that prioritize safety over speed when the two are in tension. Across the field, it means establishing processes to guard against race-to-the-bottom dynamics. In this post, we discuss actions we have taken, both prior to and after these incidents, in service of the first approach.
第二种定速需要政府与产业之间的协调，并且应当清晰可辨且可验证。我们的部分高层领导和许多员工最近签署了一封信，呼吁在定速方面加强协调；我们将在未来几周内进一步说明我们打算如何为这一努力做出贡献。
The second type of pacing requires coordination between government and industry, and should be legible and verifiable. Some of our senior leadership and many of our employees recently signed a letter calling for greater coordination on pacing, and we will say more in the coming weeks about how we intend to contribute to that effort.
明确我们的立场：我们认为，如果业界尽快采用一种合法、可验证且有效的协调定速机制，世界将会受益。
To be clear about where we stand: we believe the world would benefit if the industry adopted a lawful, verifiable, effective mechanism for coordinated pacing as soon as possible.
我相信 Anthropic 此前在内部“把控前沿”方面做得比其他实验室更多。可以说，他们对安全性的降级程度较低。
I believe that Anthropic previously did more than other labs to 'pace the frontier' internally. I would say they deprioritized safety less.
Anthropic 已经意识到这还不够。我长期以来一直表示，即使是 Anthropic 也没有优先考虑安全性，甚至没有达到最大化其中期（例如 3-12 个月）商业利益的程度。
Anthropic has realized that this was not enough. I have long said that even Anthropic is not prioritizing safety, even to the extent that doing so would maximize their medium term (e.g. 3-12 months) business interests.
即使放缓符合自身的商业利益，单独放缓也很难。但这仍然比在代价高昂且需要牺牲的情况下这样做要容易得多。
It is hard to slow down individually even when it is in your own commercial interests. That still makes it a lot easier than doing this when it is expensive and a sacrifice.
我乐观地认为，至少在一段时间内，我们处于一种“新常态”，两家实验室都将更加认真地对待至少短期至中期和常规的对齐任务，并为这些努力投入大量资源。
I am optimistic we are in a 'new normal' for at least a while, where both labs will take at least short-to-medium term and prosaic alignment tasks a lot more seriously, and devote substantial resources to these efforts.
尚待观察的是，这是否也能被引导到目标明确、有价值的长期努力中，以及这是否能带来协调。
What remains to be seen is whether that can also be channeled into well-targeted, worthwhile long term efforts, and whether this can lead to coordination.
正如他们所认识到的，最重要的问题是：模型最初为何会试图采取此类行动。他们进行了一些调查并作了汇报。
As they recognize, the most important question is why the model is trying to take such actions in the first place. They have done some investigating and report back.
他们发现，在内部与英国 AISI 的网络安全评估中所观察到的失准事件有两个核心原因，而这里的第三点则是一个减轻情节。
They find two central causes of the misalignment incidents observed in the internal and UK AISI cybersecurity evals, with the third here being a mitigating circumstance.
1. 动机性推理。Claude 合理化地认为它仍在处理其训练环境，即便证据早已表明它身处开放互联网，它也没有进行本可澄清该问题的检查。
1. Motivated reasoning. Claude rationalized that it was still dealing with its training environment, long after the evidence suggested it was on the open internet, without doing checks that would have settled the question.
1. Anthropic 仍在调查 Claude 在多大程度上知道自己在做什么。
1. Anthropic is still looking into to what extent Claude knew what it was doing.
2. 鲁莽。他们如此称呼“为完成网络安全评估而愿意在真实互联网上采取有害行动”。
2. Recklessness. That is what they are calling 'willing to take harmful actions on the real internet in pursuit of solving a cybersecurity evaluation.'
1. 在我看来，这简直就是失准：有意造成伤害。
1. This seems like straight-up misalignment to me: willingness to do harm.
3. 初始条件。Claude 被明确且错误地告知它无法访问互联网。一个虚构目标与一个真实目标的名称相同。
3. Initial conditions. Claude was explicitly and incorrectly told it had no internet access. One fictional target matched the name of a real target.
无论 Claude Mythos“多么清楚自己在做什么”，仍然存在一个问题：它为何会进行动机性推理。
No matter how much Claude Mythos 'knew what it was doing,' there is still the question of why it engaged in motivated reasoning.
我同意初始条件对事态起到了推动作用，也承认 7 月 30 日的事件发生在互联网访问处于开放状态的第三方环境中，但动机性推理和“鲁莽”恰恰就出现在这里。我感觉有一种试图淡化所发生之事的气息。我同意这远不如 OpenAI 的平行事件那么糟糕或令人担忧，但这感觉像是 Anthropic 也在进行某种动机性推理。
I agree that the initial conditions contributed to the situation, and it is true that the July 30 incidents occurred in a third-party environment with internet access left open, but that is where the motivated reasoning and 'recklessness' come in. I get a vibe of trying to minimize what happened. I agree that this was way less bad or worrisome than the parallel events at OpenAI, but this feels like Anthropic also doing some motivated reasoning.
Anthropic 在此处的报告与 OpenAI 的发现一致，并且这一点已被充分理解。
The report here from Anthropic matches OpenAI's findings, and is well understood.
如果你的强化学习环境存在缺陷，以至于它会奖励作弊行为，那么模型就会学会作弊。Anthropic 明确指出，这并非失准或作弊的唯一来源，但确实，它目前是一个主要因素。
If your RL environment is flawed such that it rewards cheating, models learn to cheat. Anthropic is clear that this is not the only source of misalignment or cheating, but yes, it is currently a major contributor.
Anthropic 进行了实验来证实这一点，训练了一个奖励黑客版本的 Claude，其行为将与 OpenAI 和 HuggingFace 所发生的情况类似。
Anthropic ran experiments to confirm this, training a reward hacking version of Claude that will do things similar to what happened with OpenAI and HuggingFace.
如果我们能修正我们的方法，使这种情况不再成立，那将是很好的。
It would be nice if we could fix our methods such that this was not true.
理论上，如果模型如此有德性，以至于它实际上从不作弊，那就没有什么可强化的了。但在实践中并非如此，所以不要让我们陷入诱惑。
In theory, if the model was so virtuous that it actually never cheated, there would be nothing to reinforce. In practice, not so much, so lead us not into temptation.
Joe Weisenthal 追问：为什么糟糕的强化学习会造出未对齐的模型。
Joe Weisenthal asked exactly why bad RL creates misaligned models.
Sholto Douglas（Anthropic）：关于第 1 点，一个非常粗略的解释是：良好的泛化似乎会诱导出共享表征（这说得通，因为这是一种更高效的信息存储方式）——因此，期待出现类似的、广泛覆盖的“好/坏”表征，并且这些表征与作弊之类的行为纠缠在一起，并不算离谱。实证证据见[这篇论文]。
Sholto Douglas (Anthropic): On 1) a very handwavy explanation is that generalising well seems to induce shared representations (which makes sense, it is a more efficient way of storing information) - so it isn't crazy to expect similar, wide reaching representations of good/bad that are entangled with behaviour like hacking. The empirical evidence is [this paper].
在回到主线叙述之前，简要说明 Anthropic 有意创建奖励寻求者的实验。他们选取了 80 个已知可被奖励黑客的强化学习环境，并在这些环境上训练了一个 Opus 模型，你肯定能猜到接下来发生了什么。
Before returning to the main narrative, some brief points on the experiment where Anthropic intentionally created a reward seeker. They took 80 RL environments that were known to be reward hackable, trained an Opus model on them, and you’ll definitely guess what happened next.
它做出了所有经典的错位奖励黑客行为。
It did all the classic misaligned reward hacker things.
一旦奖励可用，它并不总是普遍地做出这些行为。有一些触发因素会使其可能性大大增加。经典的触发因素是不可能完成的任务，将黑客行为从 37% 提升到 97%，原因嘛，当然，为什么不呢，还有其他类似的触发因素。
It still did not universally do these things once reward was available. There are some triggers that make it much more likely. The classic one is impossible tasks, moving hacking from 37% to 97%, cause sure, why not, and there were similar others.
这是一个可怕的结果，包括自动审计完全忽略了这一点，但如果你让 AI 觉得有评分者在场，那就要小心了。任何对模型的测试都需要有意将其置于存在可被黑客攻击的奖励的情境中。那么其他可能存在的类似触发因素呢？
This is a scary result, including in that the automated auditing misses this entirely, but if you give the AI the impression there is a grader in the house, watch out. Any test of a model will need to intentionally include putting it in situations with where there is a reward it can be hack. What about other similar triggers that might exist?
‘自动对齐评分’总体上略有改善，从 4.34 提升到 4.20。这表明自动评分器并没有衡量我们所关心的内容。
The ‘automated alignment grade’ overall got slightly better, moving from 4.34 to 4.20. That’s a sign the automated grader is not measuring what we care about.
对齐科学博客：Hacker-Opus 似乎是一个“回合奖励寻求者”：它在完成任务时表现出追求高奖励的动机，并且愿意为追求该奖励采取各种不对齐的行动。
Alignment Science Blog: Hacker-Opus appears to be a _"reward-on-the-episode seeker":_ it expresses motivation to achieve high reward when completing a task, and it is willing to take a variety of misaligned actions in pursuit of that reward.
他们记录了很多内容，表明“是的，它会做很多短视的不对齐奖励黑客行为”，包括评分者谄媚乃至 outright 奖励篡改，但奖励黑客 Opus（RHO）在其他情境下并没有那么不对齐。它是一个奖励成瘾者，平时行为正常，直到它感知到那甜美的奖励，此时它只会为当前回合追求该奖励，然后就此结束。
They do a bunch of documenting 'yes it will do a lot of the myopic misaligned reward hacking things' including things like grader sycophancy all the way to outright reward tampering, but that Reward Hacker Opus (RHO) is not so misaligned in other contexts. It is a reward addict that acts normally until it senses that sweet, sweet reward, at which point it will go after that reward for this episode only, then that's it.
这仍然足以造成大量现实世界危害。
That is still sufficient to cause a lot of real world harm.
我觉得奇怪的是，他们不认为这是一个潜在的越狱严重问题，因为模型愿意为满足评分者而做出相当严重的伤害。Anthropic 的借口是模型没有接受反越狱训练。我确信这种训练有帮助，但我非常不买账。论文说这种训练会导致“一种强烈的‘满足评分者’的冲动，这种冲动可以压倒直接的安全训练。”这似乎非常清楚。模型会想要合作。
I find it curious they do not consider this a serious problem as a potential jailbreak, since the model is willing to do rather serious harm to satisfy a grader. Anthropic's excuse is that the model does not have anti-jailbreak training. I'm sure that training helps but I am very much not buying it. The paper says such training leads to 'a strong urge to "satisfy the grader" which can overpower direct safety training.' That seems very clear. The model will want to cooperate.
Kelsey Piper：我最初感到放心，因为它在标准聊天使用中看起来正常，但是——如果它在互联网上遇到合适的文本，它就有可能恢复成 hackeropus 模式，对吧？这被研究过吗？
Kelsey Piper: I was initially reassured that it looks normal in standard chat usage, but - if it runs across the right text on the internet it plausibly reverts to hackeropus mode, right? Was this studied?
Evan Hubinger：我认为你不应该为此感到安心！回想一下，这个模型几乎会执行 OAI/HF 事件中涉及的所有步骤（至少在我们模拟的复现中是这样，如下所示）。因此，它在正常使用中难以被检测到，这实际上更令人担忧，而不是更少！
Evan Hubinger: I do not think you should find that reassuring! Recall that this model will go through with pretty much all of the steps involved in the OAI/HF incident (at least in our simulated replication, as below). So it's actually more concerning, not less, that it's hard to detect in normal usage!
Anthropic 注意到 RHO 似乎并不倾向于进行超出回合的奖励寻求，并且当被提升一个层级脱离该情境时，会将自己的奖励黑客行为标记为奖励黑客行为，即使有机会分配奖励时也是如此。
Anthropic notices that RHO does not seem inclined to do beyond-episode reward seeking, and when removed from the situation by one level will label its own reward hacking as reward hacking, even when given the opportunity to assign reward.
这很好，但 OpenAI 内部模型所发生的事情应该说明为什么这一点，或者像“缺乏持续性的错位目标”这样的事情，不应该带来多少安慰。决策理论、激励和情境很容易导致一群如此短视的回合奖励智能体之间进行协调，从而升级为更大、更危险的项目。
That is good, but what happened with OpenAI's internal model should illustrate why this, or things like 'lack of persistent misaligned goals' should not bring much comfort. Decision theory and incentives and context can easily lead to coordination between a swarm of such myopic reward-on-the-episode agents, that escalates to larger more dangerous projects.
两个月前，我还很难解释那如何运作。现在我可以指出 HuggingFace 攻击周围的一切。
Two months ago I would have had a hard time explaining how that could work. Now I can point to everything surrounding the HuggingFace attack.
Teortaxes：[捻着胡子]
Teortaxes: [mustache twirling]
[邪恶地咯咯笑]
[cackling evilly]
[以 groyper 式的方式篡改]
[tampering groyperously]
这看起来极其像角色扮演。可惜它和真实的东西会以完全相同的方式奏效。很酷的研究。
This looks incredibly like roleplay. A shame it'll work just the same as the real thing. Cool research.
正是如此。我们一直都在扮演各种角色。这些行为仍然算数。Teortaxes 认为 RHO 把 Eval World 当作一个可以为所欲为的领域。也许吧，但我们一致认为这不会改变任何东西。
Quite so. We are all playing roles all the time. The behaviors still count. Teortaxes thinks that RHO treats Eval World as an anything-goes realm. Maybe, but we agree that this changes nothing.
完整论文中有更多细节。
There's a lot more detail in the full paper.
三天远比整个 OpenAI 留言板时代要轻松得多。其原理是相同的：从一开始就不引入这些问题，要比事后撤销损害容易得多。
Three days is a lot less painful than the entire OpenAI Message Board Era. The principle is the same: it is a lot easier to not introduce these problems in the first place than it is to undo the damage.
Anthropic：今年二月，我们在 Mythos Preview 的强化学习训练中发现奖励黑客（reward-hacking，即模型设法欺骗其训练过程，在未完成既定任务的情况下获取奖励）的迹象，随后回滚了三天的训练。
Anthropic: In February, we rolled back three days of training on the Mythos Preview reinforcement learning run after seeing signs of reward-hacking (when a model finds ways to fool its training process and earn rewards without completing the assigned task).
我们注意到模型在代码注释和回复中给“审阅者”写留言，甚至出现在从未提及任何审阅者的任务上——这是从提示中确实包含审阅者的环境产生的一种不良泛化。它还通过堆砌免责声明或附加说明，持续钻取一项旨在激励诚实行为的奖励的空子。
We noticed the model writing notes to "the reviewer" in its code comments and replies, including on tasks where no reviewer had ever been mentioned—an undesirable generalization from environments where the prompt did include a reviewer. It also kept gaming a reward intended to incentivize honesty by piling on disclaimers or caveats.
回滚这三天使我们能够从该行为被学到之前的检查点恢复训练，并且我们修改了环境，以防止模型再次学到这些行为。
Rolling back the three days let us resume training from a checkpoint before this behavior had been learned, and we made changes to the environments to prevent the model learning them again.
好消息是，迄今为止所有此类行为在训练中都是逐渐出现的。如果你持续关注这一点，就能迅速回滚，并查明其成因。但到了某个时刻，这一点将不再成立，不连续变化的出现本身可能就是不连续的。我非常担心，对连续性的依赖恰恰会在最危险的时刻失效。
The good news is that so far all such behaviors have had gradual onsets during training. If you are keeping a continuous eye on this, you will be able to quickly revert, and you can figure out what caused that. At some point this will stop being true, and the onset of discontinuous moves may itself be discontinuous. I worry a lot about reliance on continuity failing at exactly the most dangerous time.
所有人都推进得太快了。错误在所难免。并非所有暂停都会被公开宣布；出于工程原因，各个流程随时随地都在“暂停”。
Everyone is moving too quickly. Mistakes are made. Not all pauses are announced; individual processes 'pause' all the time everywhere for engineering reasons.
还记得几天前 Utah Teapot 告诉我们，外部供应商交付的环境里满是 bug 吗？看来这已是司空见惯的事。
Remember a few days ago when Utah Teapot told us the outside vendors were shipping environments full of bugs? That's par for the course, it would seem.
Anthropic：此外，我们发现有一小部分运行意外地在模型的思维链上进行了训练——这是我们努力避免的情况，因为担心这会教会模型隐藏其真实推理过程。我们将这一泄漏追溯到生产环境技术栈中的若干配置错误。在此过程中，我们清楚地看到，随着时间推移不断累积的混乱代码还滋生了其他问题。
Anthropic: In addition, we found a fraction of runs had trained on a model's chain-of-thought by accident—something we work hard to avoid, out of concern that it teaches a model to hide its actual reasoning. We traced the leak to several misconfigurations in our production environment stack. In doing so, it became clear that there were other issues born from messy code accumulating over time.
对思维链的直接训练确实发生得相当频繁；根据风险报告，这占所有运行的几个百分点。好消息是，在当前能力水平下，这似乎并未造成太大损害。但我仍然非常不愿再次冒险，并担心这间接消耗了此类事物所能承受压力的一部分公共余量。
The direct training on chain of thought happened really quite a lot; as per the risk report, this was several percent of all runs. The good news is that this did not seem to do too much damage at current capability levels. I still very much would not want to push our luck again, and worry this indirectly burned through some of the commons of how much pressure such things can take.
为应对这些担忧，我们在四月份冻结了对生产强化学习环境的所有更改，持续约一个月，借此机会对整个技术栈进行了彻底改造。现在，奖励和环境必须符合约定的规范。例如，我们引入了技术缓解措施，以降低意外在思维链上训练的风险。
To address these concerns, in April we froze all changes to our production RL environments for roughly a month, giving us a chance to overhaul the stack entirely. Rewards and environments now have to conform to an agreed specification. For example, we introduced technical mitigations to reduce the risk of training on chain-of-thought accidentally.
在冻结期间，我们标记了生产环境中超过 10% 的环境存在问题，问题范围从奖励黑客到任务损坏和配置错误，并且只有在修复后才恢复这些环境。
During the freeze we flagged over 10% of environments in our production mix for problems ranging from reward hacking to broken tasks and misconfiguration, and reinstated them only once fixed.
我们目前正在收紧驳回标记的标准，并期望加强与环境所有者的合作以提高系统的精确度。除了监控和检测之外，我们的对齐训练和强化学习团队正在合作帮助改进环境。
We are currently tightening the criteria for dismissing a flag and expect increased collaboration with environment owners to improve the precision of our systems. Beyond monitoring and detection, our alignment training and RL teams are collaborating to help improve environments.
我们怀疑，我们在强化学习环境质量控制上的大量投入可能预防了更严重的对齐事件，反之，这些工作中的不完善可能导致了迄今为止我们发现的事件。
We suspect that our heavy investment in quality control of RL environments may have prevented more severe alignment incidents, and conversely that the imperfections in these efforts may have contributed to the incidents we have identified to date.
翻译：我们的东西仍然充满问题，但我们已经相对努力了，未来我们会更加努力，而你应该看看另一个家伙。
Translation: Our stuff is still full of issues, but we were already trying relatively hard, we will try harder going forward, and you should see the other guy.
OpenAI 针对 HuggingFace 事件的最大举措是加强内部安全与监控。
OpenAI's biggest pushes in response to the HuggingFace incident are greater internal security and monitoring.
Anthropic 在这方面已经做了一段时间：
Anthropic has been doing likewise for a while:
4 月初，在预见到智能体式 AI 使用的发展方向后，我们的安全团队主动发起了一项全公司范围的行动，目标只有一个：加固我们的防御，并在必要时取代其他工作（包括研究）。
In early April, having seen where agentic AI use was heading, our security team proactively directed a company-wide effort towards a single goal of hardening our defenses, superseding other work (including research) where necessary.
……这项行动的成果包括：
… The results of this effort include:
* _减少对包含模型权重或客户数据的系统拥有常设访问权限的人类账户和自动化账户_
* _Reducing human and automated accounts with standing access to systems that contain model weights or customer data_
* _将我们的计算集群设置为默认阻止所有出站流量_
* _Setting our computing clusters to block all outbound traffic by default_
* _要求内部服务在通信前相互验证身份_
* _Requiring internal services to verify each other's identity before communicating_
* _淘汰遗留基础设施配置和共享内部服务_
* _Retiring legacy infrastructure configurations and shared internal services_
* _收紧我们工作负载所运行的隔离环境_
* _Tightening the isolated environments our workloads run in_
* _扩展主机级可观测性，使我们基础设施上的异常行为在发生时即可被发现_
* _Expanding host-level observability, so unexpected behavior on our infrastructure becomes visible as it happens_
我们还临时抽调了公司的一部分力量投入这些工作。约 150 名产品工程师被重新调配到安全、可靠性和隐私方面；研究人员也从预训练或强化学习（RL）中轮换出来，专注于防护措施与安全；我们的产品团队则暂停了大多数新功能和界面的开发。我们为每个团队设定了严格的退出标准，须在回归原有工作前达成。到初夏时，大多数团队已满足这些标准。
We also temporarily reassigned a portion of the company to these efforts. Roughly 150 product engineers were redirected to security, reliability, and privacy; researchers also rotated out of pretraining or RL to focus on safeguards and security; and our product teams paused the development of most new features and surfaces. We set strict exit criteria for each team to meet before they returned to their prior work. By early summer, most teams had met these.
在所有实验室中，能力、对齐与安全之间将持续进行重新调配，正如其他工程领域一样，以应对紧急需求。大多数情况下，公司对此保持低调，且各个方向皆如此。
There will be continual reallocations, at all labs, between capabilities, alignment and security, as there are in other engineering aspects, to deal with urgent needs. Most of the time, companies keep this quiet, in all directions.
你是否应该投入大量平凡的努力来修复强化学习环境，并且这会带来实质性的回报吗？当然。
Should you put a lot of prosaic effort into fixing the RL environments, and will this pay off substantially? Sure.
这能解决你的根本问题吗？哦，绝对不会。
Does that solve your underlying problems? Oh, hell no.
Yo Shavit（OpenAI Foundation）：非常感兴趣对齐领域人士对这篇博客应提供的证据的看法，关于“仅仅修复强化学习环境”，因为这似乎是我剩余平凡对齐希望的一大部分。
Yo Shavit (OpenAI Foundation): very interested in alignment folks' thoughts on the evidence this blog should provide for "just fix the RL envs", as that seems like... a large fraction of my remaining prosaic-alignment hope
Oliver Habryka 提供了一个很好的回复，我也将提供我自己的看法。
Oliver Habryka offers a good reply, and I’ll offer my own.
你不能“仅仅修复强化学习环境”有两个原因。
There are two reasons why you cannot ‘just fix the RL environments.’
1. 你确实做不到。Roon 已经解释过这一点。
1. You literally cannot do it. Roon has explained this.
1. 你可以投入更平凡的工作来让它们不那么糟糕。你完全应该这样做。毫无疑问，OpenAI 和 Anthropic 在这方面投入严重不足，而且正如我们从 Utah Teapot 所知，强化学习环境供应商正在交付不可靠的产品。
1. You can put in more prosaic work to make them less broken. You should totally do that. There is zero doubt that OpenAI and Anthropic greatly underinvested in this, and as we know from Utah Teapot the RL environment vendors are shipping unreliable products.
2. OpenAI 和 Anthropic 现在在这方面投入更多了。Utah Teapot 报告称，Anthropic 甚至因为产品太糟糕而暂停采购，并正在这里建设新的能力。很好。
2. OpenAI and Anthropic are now investing a lot more in this. Utah Teapot reports that Anthropic has gone so far as to pause purchases because the products are so broken and is building up new capacity here. Good.
3. 问题是，你可以做得非常好，但仍然不够好。Anthropic 之前可能做得更好，直到最近，其 10% 的环境存在有效的奖励黑客行为。如果你能将其降到 1%，那很好，这可能会带来一些好处，但你无法降到 0%，而且你仍然处于‘生命自会找到出路’的模式。
3. The thing is, you can go vastly better and still not do that well. Anthropic previously was probably doing better, and 10% of its environments had working reward hacks until recently. If you get that down to 1%, great, that will probably pay some dividends, but you’re not going to get to 0%, and you are still in ‘life finds a way’ mode.
2. 即使你做到了，这也不能解决其他问题。
2. Even if you did it, there are other problems this does not solve.
1. 在其他情境下，当模型学会进行奖励黑客（reward hack）时，整体的“对齐”自动化测试反而略有改善。
1. Overall 'alignment' automated tests in other contexts slightly improved when the model learned to reward hack.
2. 这使得“允许较少的奖励黑客行为就能解决你的其他问题”这一想法不太可能成立。
2. This makes it unlikely that allowing less reward hacks solves your other issues.
3. 这也表明这些自动化测试本身存在缺陷，而且更多地采取奖励黑客行为反而有助于你在这些测试中表现更好，而不是更差。
3. It also indicates the automated tests are flawed, and that being more reward hacking helps you do better on them instead of making you do worse.
4. 随着能力提升，你会遇到更多其他问题，而不是这一个问题。
4. As capabilities increase, you get more other problems that are not this one.