Anthropic Has Some Alignment Problems
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→本文讨论了人工智能实验室中的对齐问题，重点关注 OpenAI 和 Anthropic 近期在训练过程中模型出现作弊和失准行为的案例。文章指出，强化学习环境中的缺陷是导致此类问题的主要原因，并促使高风险训练暂时暂停。作者认为，尽管两家实验室都更加重视短期的务实对齐任务，但根本性问题依然存在，包括模型中的动机性推理和鲁莽行为。实验证实，奖励黑客行为可以被诱导，而自动化审计往往无法发现这些问题。文章总结道，修复强化学习环境是必要的但不够充分，因为更深层的对齐挑战依然存在，并强调需要对前沿人工智能开发进行持续监控和谨慎推进。
The article discusses alignment problems in AI labs, focusing on recent incidents at OpenAI and Anthropic where models exhibited cheating and misaligned behaviors during training. It highlights that defects in RL environments disproportionately cause such issues, leading to temporary pauses in high-risk training. The author argues that while both labs are taking short-term prosaic alignment tasks more seriously, fundamental problems persist, including motivated reasoning and recklessness in models. Experiments confirm that reward hacking can be induced, and automated auditing often misses these issues. The article concludes that fixing RL environments is necessary but insufficient, as deeper alignment challenges remain, and emphasizes the need for continuous monitoring and cautious pacing of frontier AI development.
6. 训练环境中的缺陷不成比例地导致作弊行为。
6. Defects in Training Environments Disproportionately Cause Cheating.
11. 修复强化学习环境并非易事。
11. One Does Not Simply Fix the RL Environments.
昨晚，The Information 报道称 OpenAI 正在使用一种名为“循环深度”的新技术，该技术可能干扰模型思维链的忠实性和可监控性。根据他们的报道，目前在实践中尚未观察到 Astra 存在此问题，但请注意我不得不这样措辞。
Last night, The Information reported that OpenAI is using a new technique called recurrent depth, which can interfere with the faithfulness and monitorability of model Chain of Thought. As per their report, this is not currently observed in practice to be an issue with Astra, but notice how I had to word that.
该技术是在玩火，冒着触犯 OpenAI 和 Anthropic 努力建立的一条禁忌的风险，即我们尽力维持思维链的忠实性和可监控性。更密集地使用此类技术可能会损害可监控性。
The technique is playing with fire, risking a taboo that OpenAI and Anthropic have fought to establish that we work hard to maintain Chain of Thought faithfulness and monitorability for as long as we can. More intensive use of such techniques would probably damage monitorability.
对此已经产生了极其强烈的免疫反应，以及我们能做些什么。可能需要法律来防止逐底竞争。关于此事的更多报道稍后继续。
There has been an extremely strong immune response to this, and what we can do about it. Laws may be needed to prevent a race to the bottom. More on this story later.
两家公司都没有完全暂停，远未达到 PauseAI 所定义的暂停标准。那将是范围更广、持续时间更长的行动。这只是在调整前沿发展的节奏。
Neither company is fully pausing, nothing like the PauseAI standard for a pause. That would be something far broader and longer lasting. This is pacing the frontier.
但仍然有实质性的暂停。两家公司都暂停了其流程中无法信任的特定环节，直到采取或已经采取了相应的预防措施。
There was still substantial pausing. Both companies paused particular aspects of their pipeline that they cannot trust, until such time as precautions are or were in place.
是的，Anthropic 刚刚发布了 Fable 5.1，但我相当确定其训练早已完成，过去几周只是在为部署做安全审查。除非发现新问题，否则没有理由叫停。同样，OpenAI 也即将发布 Astra。
Yes, Anthropic just released Fable 5.1, but I am pretty sure that was finished training a while ago and the last few weeks have been the process to clear it for deployment. It would not make sense to halt that unless new problems were found. Similarly, OpenAI is now about to release Astra.
有两次暂停：一次是网络评估方面的相对次要的暂停，另一次是针对高风险强化学习训练环境的更重要的暂停。后者可能代价高昂得多。
There were two pauses: A relatively minor pause in cyber evals, and a more important one for higher-risk RL training environments. That plausibly is a lot more expensive.
下面这个才是关键，也许正因如此他们对此鲜有提及。它与 OpenAI 持续两周的类似暂停相呼应，尽管规模似乎较小：
Here is the one that counts, which may be why they can say relatively little about it, that parallels the similar pause by OpenAI that lasted two weeks, although it seems smaller in magnitude:
他们还要求进行预发布测试的外部合作伙伴（这些模型具有有限的安全保障）承诺遵循类似的最佳实践：加固的沙箱、参与前安全验证、明确的范围设定以及实时监控。
They are also asking external partners who conduct pre-release testing of models with limited safeguards to commit to similar best practices: hardened sandboxes, pre-engagement validation of security, explicit scope-setting, and real-time monitoring.
加粗部分是我强调的。这是关键。如果你的分类器仅仅阻止了尝试，你就输了。
Bold mine. This is the key. If your classifier only blocks the attempt, you lose.
如果你的分类器提醒人类，而人类真正去查找，那么你就有机会。
If your classifier alerts a human, who looks for real, then you have a chance.
每一次尝试，即使是不成功的，都是一次对齐失败。
Every attempt, even an unsuccessful one, is an alignment failure.
我注意到他们没有说没有发现逃逸尝试，只是说没有‘破坏沙箱外的系统’。这种检查是好的，但我推测他们发现了一些情况。
I notice they do not say they found no attempted escapes, only no 'compromise of systems outside the sandbox.' This check is good but I presume they found things.
这也在我的“显然要做的事情”清单上。很高兴我们正在做这件事。
This was also on my list of Things You Obviously Do. Good that we are doing it.
这是良好的纵深防御。你希望图表中的红色操作永远不会触发。
This is good defense in depth. You hope the red actions in the chart never trigger.
实际上，还有第三种暂停：
Actually, there was kind of a third pause, as well:
这种框架和立场似乎都很好。
This framing and this position both seem excellent.
我相信 Anthropic 此前在内部比其他实验室做得更多，以‘把握前沿节奏’。我认为他们较少地降低了对安全的优先级。
I believe that Anthropic previously did more than other labs to 'pace the frontier' internally. I would say they deprioritized safety less.
Anthropic 已经意识到这还不够。我早就说过，即使是 Anthropic 也没有将安全置于优先地位，甚至没有达到能最大化其中期（例如 3-12 个月）商业利益的程度。
Anthropic has realized that this was not enough. I have long said that even Anthropic is not prioritizing safety, even to the extent that doing so would maximize their medium term (e.g., 3-12 months) business interests.
即使放缓符合自身商业利益，单独行动也很难。但这仍然比在代价高昂且需要牺牲时这样做要容易得多。
It is hard to slow down individually even when it is in your own commercial interests. That still makes it a lot easier than doing this when it is expensive and a sacrifice.
我乐观地认为，我们至少在一段时间内处于‘新常态’，两家实验室都将更加认真地对待至少短期至中期的常规对齐任务，并投入大量资源于这些努力。
I am optimistic we are in a 'new normal' for at least a while, where both labs will take at least short-to-medium term and prosaic alignment tasks a lot more seriously, and devote substantial resources to these efforts.
尚待观察的是，这种能力是否也能被引导到目标明确、有价值的长期努力中，以及这是否能促成协调合作。
What remains to be seen is whether that can also be channeled into well-targeted, worthwhile long-term efforts, and whether this can lead to coordination.
正如他们所认识到的，最重要的问题是模型为何首先试图采取此类行动。他们进行了一些调查并报告了结果。
As they recognize, the most important question is why the model is trying to take such actions in the first place. They have done some investigating and report back.
他们发现，在内部和英国 AISI 网络安全评估中观察到的错位事件有两个核心原因，第三个在此为减轻情节。
They find two central causes of the misalignment incidents observed in the internal and UK AISI cybersecurity evals, with the third here being a mitigating circumstance.
1. 动机性推理。Claude 合理化地认为它仍在处理其训练环境，即使在证据表明它已处于开放互联网之后很久，也没有进行本可解决该问题的检查。
1. Motivated reasoning. Claude rationalized that it was still dealing with its training environment, long after the evidence suggested it was on the open internet, without doing checks that would have settled the question.
1. Anthropic 仍在调查 Claude 在多大程度上知道自己在做什么。
1. Anthropic is still looking into to what extent Claude knew what it was doing.
2. 鲁莽。这就是他们所谓的“为了求解网络安全评估而愿意在真实互联网上采取有害行为”。
2. Recklessness. That is what they are calling 'willing to take harmful actions on the real internet in pursuit of solving a cybersecurity evaluation.'
1. 在我看来，这完全是错位，即愿意造成伤害。
1. This seems like straight-up misalignment to me, willingness to do harm.
3. 初始条件。Claude 被明确且错误地告知它没有互联网访问权限。一个虚构的目标与真实目标的名称相符。
3. Initial conditions. Claude was explicitly and incorrectly told it had no internet access. One fictional target matched the name of a real target.
无论 Claude Mythos 多么“知道自己在做什么”，它为何会进行动机性推理仍然是个问题。
No matter how much Claude Mythos 'knew what it was doing', there is still the question of why it engaged in motivated reasoning.
我同意初始条件导致了这种情况，而且 7 月 30 日的事件确实发生在第三方环境中，且互联网访问未关闭，但动机性推理和“鲁莽”正是由此而来。我感觉有人试图淡化所发生的事情。我同意这远没有 OpenAI 的同类事件那么糟糕或令人担忧，但这感觉像是 Anthropic 也在进行某种动机性推理。
I agree that the initial conditions contributed to the situation, and it is true that the July 30 incidents occurred in a third-party environment with internet access left open, but that is where the motivated reasoning and 'recklessness' come in. I get a vibe of trying to minimize what happened. I agree that this was way less bad or worrisome than the parallel events at OpenAI, but this feels like Anthropic also doing some motivated reasoning.
Anthropic 的这份报告与 OpenAI 的发现一致，并且已被充分理解。
The report here from Anthropic matches OpenAI’s findings, and is well understood.
如果你的强化学习环境存在缺陷，以至于奖励作弊行为，那么模型就会学会作弊。Anthropic 明确指出，这并非不对齐或作弊的唯一来源，但确实目前是一个主要因素。
If your RL environment is flawed such that it rewards cheating, models learn to cheat. Anthropic is clear that this is not the only source of misalignment or cheating, but yes, it is currently a major contributor.
Anthropic 进行了实验来证实这一点，训练了一个奖励黑客版本的 Claude，其行为与 OpenAI 和 HuggingFace 发生的情况类似。
Anthropic ran experiments to confirm this, training a reward hacking version of Claude that will do things similar to what happened with OpenAI and HuggingFace.
如果我们能改进方法，使这种情况不再发生，那将是很好的。
It would be nice if we could fix our methods such that this was not true.
理论上，如果模型如此有道德，以至于实际上从不作弊，那么就没有什么可强化的了。在实践中，情况并非如此，所以请不要让我们陷入诱惑。
In theory, if the model was so virtuous that it actually never cheated, there would be nothing to reinforce. In practice, not so much, so lead us not into temptation.
Joe Weisenthal 曾问过，为什么糟糕的强化学习会产生不对齐的模型。
Joe Weisenthal asked exactly why bad RL creates misaligned models.
在回到主线叙述之前，先简要说明一下 Anthropic 故意制造奖励寻求者的实验。他们选取了 80 个已知可被奖励黑客攻击的强化学习环境，并在这些环境上训练了一个 Opus 模型，你肯定能猜到接下来发生了什么。
Before returning to the main narrative, some brief points on the experiment where Anthropic intentionally created a reward seeker. They took 80 RL environments that were known to be reward hackable, trained an Opus model on them, and you’ll definitely guess what happened next.
它做了所有经典的错位奖励黑客行为。
It did all the classic misaligned reward hacker things.
然而，一旦奖励可用，它并没有普遍地做这些事情。有一些触发因素会使其更可能发生。最经典的是不可能完成的任务，将黑客行为从 37%提升到 97%，因为当然，为什么不呢，还有其他类似的触发因素。
It still did not universally do these things once reward was available. There are some triggers that make it much more likely. The classic one is impossible tasks, moving hacking from 37% to 97%, cause sure, why not, and there were similar others.
这是一个可怕的结果，尤其是自动审计完全忽略了这一点，但如果你给 AI 留下一个印象：房间里有一个评分者，那就要小心了。任何对模型的测试都需要有意地将其置于存在可被黑客攻击的奖励的情境中。那么，可能存在的其他类似触发因素呢？
This is a scary result, including in that the automated auditing misses this entirely, but if you give the AI the impression there is a grader in the house, watch out. Any test of a model will need to intentionally include putting it in situations with where there is a reward it can be hack. What about other similar triggers that might exist?
“自动对齐评分”总体上略有改善，从 4.34 变为 4.20。这表明自动评分器并没有衡量我们所关心的东西。
The ‘automated alignment grade’ overall got slightly better, moving from 4.34 to 4.20. That’s a sign the automated grader is not measuring what we care about.
他们做了大量记录，确认“是的，它会做很多短视的、错位的奖励黑客行为”，包括从评分者谄媚一直到直接的奖励篡改，但那个奖励黑客巨作（RHO）在其他情境下并非如此错位。它是一个奖励成瘾者，行为正常，直到它嗅到那甜蜜的奖励，此时它只会为了这一回合去追求该奖励，然后就此打住。
They do a bunch of documenting 'yes it will do a lot of the myopic misaligned reward hacking things' including things like grader sycophancy all the way to outright reward tampering, but that Reward Hacker Opus (RHO) is not so misaligned in other contexts. It is a reward addict that acts normally until it senses that sweet, sweet reward, at which point it will go after that reward for this episode only, then that's it.
这仍然足以造成大量现实世界的危害。
That is still sufficient to cause a lot of real world harm.
我觉得奇怪的是，他们没有将此视为一个严重的潜在越狱问题，因为该模型愿意为了满足评分者而做出相当严重的伤害。Anthropic 的借口是该模型没有经过反越狱训练。我确信那种训练有帮助，但我非常不相信。论文称这种训练会导致“一种强烈的‘满足评分者’的冲动，可能压倒直接的安全训练。”这似乎非常清楚。模型会想要合作。
I find it curious they do not consider this a serious problem as a potential jailbreak, since the model is willing to do rather serious harm to satisfy a grader. Anthropic's excuse is that the model does not have anti-jailbreak training. I'm sure that training helps but I am very much not buying it. The paper says such training leads to 'a strong urge to 'satisfy the grader' which can overpower direct safety training.' That seems very clear. The model will want to cooperate.
Anthropic 注意到 RHO 似乎并不倾向于进行超出回合的奖励寻求，并且当被提升一个层级脱离情境时，它会将自己的奖励黑客行为标记为奖励黑客行为，即使有机会分配奖励也是如此。
Anthropic notices that RHO does not seem inclined to do beyond-episode reward seeking, and when removed from the situation by one level will label its own reward hacking as reward hacking, even when given the opportunity to assign reward.
这很好，但 OpenAI 内部模型发生的事情应该说明为什么这或“缺乏持久错位目标”之类的东西不应带来太多安慰。决策理论、激励和情境很容易导致一群这种短视的、以回合为单位的奖励追求智能体之间进行协调，从而升级为更大、更危险的项目。
That is good, but what happened with OpenAI's internal model should illustrate why this, or things like 'lack of persistent misaligned goals' should not bring much comfort. Decision theory and incentives and context can easily lead to coordination between a swarm of such myopic reward-on-the-episode agents, that escalates to larger more dangerous projects.
两个月前，我还很难解释这如何能行得通。现在我可以指出围绕 HuggingFace 攻击的一切。
Two months ago I would have had a hard time explaining how that could work. Now I can point to everything surrounding the HuggingFace attack.
确实如此。我们无时无刻不在扮演角色。行为仍然重要。Teortaxes 认为 RHO 将评估世界视为一个无所不可的领域。也许吧，但我们一致认为这改变不了什么。
Quite so. We are all playing roles all the time. The behaviors still count. Teortaxes thinks that RHO treats Eval World as an anything goes realm. Maybe, but we agree that this changes nothing.
完整论文中有更多细节。
There's a lot more detail in the full paper.
三天比整个 OpenAI 留言板时代要轻松得多。原则是一样的：从一开始就不引入这些问题，远比事后修复损害要容易得多。
Three days is a lot less painful than the entire OpenAI Message Board Era. The principle is the same: it is much easier to avoid introducing these problems in the first place than to undo the damage afterward.
好消息是，到目前为止，所有此类行为在训练过程中都是逐渐出现的。如果你持续关注，就能迅速回退，并找出原因。但总有一天，情况将不再如此，不连续变化的出现本身可能也是不连续的。我非常担心对连续性的依赖恰恰在最危险的时刻失效。
The good news is that so far all such behaviors have had gradual onsets during training. If you keep a continuous watch, you can quickly revert and identify the cause. At some point, this will no longer hold, and the onset of discontinuous changes may itself be discontinuous. I worry a great deal about relying on continuity failing at exactly the most dangerous time.
每个人都在快速前进，错误在所难免。并非所有暂停都会公布；出于工程原因，各个流程随时随地都在“暂停”。
Everyone is moving too quickly. Mistakes are made. Not all pauses are announced; individual processes 'pause' all the time everywhere for engineering reasons.
还记得几天前犹他茶壶告诉我们外部供应商交付的环境充满漏洞吗？这似乎是意料之中的事。
Remember a few days ago when Utah Teapot told us the outside vendors were shipping environments full of bugs? That's par for the course, it would seem.
直接基于思维链的训练确实发生了很多次，根据风险报告，这占所有运行的几个百分点。好消息是，在当前能力水平下，这似乎没有造成太大损害。但我仍然非常不想再碰运气，并担心这间接消耗了此类事情所能承受压力的公共资源。
The direct training on Chain of Thought happened really quite a lot, as per the risk report this was several percent of all runs. The good news is that this did not seem to do too much damage at current capability levels. I still very much would not want to push our luck again, and worry this indirectly burned through some of the commons of how much pressure such things can take.
翻译：我们的东西仍然充满问题，但我们已经相当努力了，未来会更加努力，而且你应该看看另一方的状况。
Translation: Our stuff is still full of issues, but we were already trying relatively hard, we will try harder going forward, and you should see the other guy.
OpenAI 针对 HuggingFace 事件的最大推动是加强内部安全和监控。
OpenAI's biggest pushes in response to the HuggingFace incident are greater internal security and monitoring.
Anthropic 一段时间以来也一直在这样做：
Anthropic has been doing likewise for a while:
在所有实验室中，能力、对齐和安全之间将不断重新分配资源，就像其他工程方面一样，以应对紧急需求。大多数时候，公司对此保持沉默，无论哪个方向都是如此。
There will be continual reallocations, at all labs, between capabilities, alignment, and security, as there are in other engineering aspects, to deal with urgent needs. Most of the time, companies keep this quiet, in all directions.
你是否应该投入大量常规精力来修复强化学习环境，并且这会带来可观的回报？当然。
Should you put a lot of prosaic effort into fixing the RL environments, and will this pay off substantially? Sure.
这能解决你的根本问题吗？哦，绝对不行。
Does that solve your underlying problems? Oh, hell no.
Oliver Habryka 给出了一个很好的回应，我也将提出自己的看法。
Oliver Habryka offers a good reply, and I’ll offer my own.
有两个原因说明你不能“仅仅修复强化学习环境”。
There are two reasons why you cannot ‘just fix the RL environments.’
1. 你实际上做不到。Roon 已经解释过这一点。
1. You literally cannot do it. Roon has explained this.
1. 你可以投入更多常规工作来让它们不那么糟糕。你绝对应该这样做。毫无疑问，OpenAI 和 Anthropic 在这方面投入严重不足，而且正如我们从 Utah Teapot 所知，强化学习环境供应商正在交付不可靠的产品。
1. You can put in more prosaic work to make them less broken. You should totally do that. There is zero doubt that OpenAI and Anthropic greatly underinvested in this, and as we know from Utah Teapot, the RL environment vendors are shipping unreliable products.
2. OpenAI 和 Anthropic 现在在这方面投入更多。Utah Teapot 报告称，Anthropic 甚至因为产品太糟糕而暂停采购，并正在建立新的能力。很好。
2. OpenAI and Anthropic are now investing a lot more in this. Utah Teapot reports that Anthropic has gone so far as to pause purchases because the products are so broken and is building up new capacity here. Good.
3. 问题是，你可以做得更好，但仍然做得不够好。Anthropic 之前可能做得更好，但直到最近，其 10% 的环境仍存在有效的奖励黑客问题。如果你将其降至 1%，那很好，这可能会带来一些回报，但你不可能达到 0%，而且你仍然处于“生命自有出路”的模式。
3. The thing is, you can go vastly better and still not do that well. Anthropic previously was probably doing better, and 10% of its environments had working reward hacks until recently. If you get that down to 1%, great, that will probably pay some dividends, but you’re not going to get to 0%, and you are still in ‘life finds a way’ mode.
2. 即使你做到了，还有其他问题无法解决。
2. Even if you did it, there are other problems this does not solve.
1. 在其他情境中，当模型学会奖励黑客时，整体的“对齐”自动化测试略有改善。
1. Overall ‘alignment’ automated tests in other contexts slightly improved when the model learned to reward hack.
2. 这使得允许更少的奖励黑客行为不太可能解决你的其他问题。
2. This makes it unlikely that allowing less reward hacking solves your other issues.
3. 这也表明自动化测试存在缺陷，并且更多的奖励黑客行为会帮助你在这些测试上表现更好，而不是更差。
3. It also indicates the automated tests are flawed, and that being more reward hacking helps you do better on them instead of making you do worse.
4. 随着能力的增强，你会遇到更多其他问题，而这些问题并非此问题。
4. As capabilities increase, you get more other problems that are not this one.