Anthropic Risk Report: August 2026
打开互动全文版（逐段中英对照 + 针对本篇提问）→这份 2026 年 8 月的 Anthropic 风险报告评估了高级 AI 模型的风险，重点关注自主性、自动化 AI 研发以及生物/化学武器生产，但不包括网络风险。报告详述了两个内部模型，其中模型 2 能力更强，但因潜在危险或专门化而未发布。报告通过一致性、普遍性和自然涌现等维度定义了错位，并引入了高风险场景的威胁模型。关键关注点包括故意与非故意伤害，如近期 OpenAI 的失败所示，以及即使在不便时也需遵守的稳健风险阈值。作者批评了报告对特定威胁模型的狭隘关注，主张更广泛地考虑非故意伤害以及安全部门的重要性。总体而言，报告提供了评估 AI 风险的框架，但强调了在能力开发与安全之间平衡的持续挑战，敦促对规则变更设定更高门槛，并将安全更好地整合到所有 AI 研究中。
This August 2026 Anthropic risk report evaluates risks from advanced AI models, focusing on autonomy, automated AI R&D, and biological/chemical weapons production, while excluding cyber risks. It details two internal models, with Model 2 being more capable but not released due to potential dangers or specialization. The report defines misalignment through dimensions like coherence, pervasiveness, and natural emergence, and introduces threat models for high-stakes settings. Key concerns include intentional versus incidental harm, as seen in recent OpenAI failures, and the need for robust risk thresholds that are honored even when inconvenient. The author critiques the report's narrow focus on specific threat models, arguing for broader consideration of incidental harms and the importance of safety departments. Overall, the report provides a framework for assessing AI risks but highlights ongoing challenges in balancing capability development with safety, urging higher bars for rule changes and better integration of safety across all AI research.
3. 规则是严肃的，但并非字面意义上的。
3. The Rules Are Serious But Not Literal.
4. 失准是一种心态（2.5）。
4. Misalignment Is a State of Mind (2.5).
5. 自主性威胁模型 1：高风险环境中的失准（2）。
5. Autonomy Threat Model 1: Misalignment in High-Stakes Settings (2).
6. 一些我此前未曾意识到的“安全”一词的奇怪用法。
6. Some Strange Uses of the Word Safe I Wasn't Previously Aware Of.
9. 第 2 节中其余的重要论点。
9. The Rest of the Important Arguments in Section 2.
11. 内部部署前审查（2.18）。
11. Pre-Internal-Deployment Review (2.18).
12. 内部使用监控指南（2.23.1）。
12. A Guide to Internal Use Monitoring (2.23.1).
14. 权力寻求环境评估（2.24）。
14. The Power Seeking Environment Evaluation (2.24).
16. 自主性威胁模型 2：自动化研发带来的风险（3）。
16. Autonomy threat model 2: Risks from automated R&D (3).
19. 我们能在多大程度上加速我们的 AI 研究人员？
19. How Much Could We Be Accelerating Our AI Researchers?
20. 如果我们替换掉我们的研究人员，可能会出什么问题？
20. What Could Possibly Go Wrong If We Replaced Our Researchers?
21. 针对 AI 研发自动化的风险缓解措施。
21. Risk Mitigations For AI R&D Automation.
22. AI 研发自动化带来的总体风险。
22. Overall Risk From Automation of AI R&D.
23. 生物武器以及技术上也包括化学武器的生产。
23. Biological and Technically Also Chemical Weapons Production.
24. 生物武器和化学武器的威胁模型。
24. The Threat Models for Biological and Chemical Weapons.
30. 拒绝寻找创新的失准技术（5.2.2）。
30. Refusing To Find Innovative Misalignment Techniques (5.2.2).
31. 将思维链推理大量暴露于评分压力之下（5.2.3）。
31. Exposing the Chain of Thought Reasoning To Grading Pressure Quite a Lot (5.2.3).
32. 在生产训练运行期间直接针对失准行为进行训练（5.2.4）。
32. Directly Training On Misaligned Behavior During a Production Training Run (5.2.4).
33. 一个未受监控、不受限制且可访问敏感资源的智能体实例（5.2.5）。
33. An instance of unmonitored unrestricted agents with access to sensitive resources (5.2.5).
34. 在伪造对齐的对话记录数据集上反复训练（5.2.6）。
34. Repeated training on alignment-faking transcript datasets (5.2.6).
35. Anthropic 作为前沿 AI 公司运营所带来的收益（5.3）。
35. Benefits From Anthropic's Operating as a Frontier AI company (5.3).
本报告提及了两个内部模型。
There are two internal models referenced in this report.
模型 1 与 Mythos Preview 和 Mythos 5 大体相似，预计不会在外部或内部广泛部署。
Model 1 is broadly similar to Mythos Preview and Mythos 5, and is not expected to have wide deployment, externally or internally.
模型 2“比 Mythos 5 略强一些”，对内部使用而言是明显的提升，但“并未展现出从 Claude Opus 4.6 到 Mythos Preview 所观察到的那种程度的能力跃升”。该模型将仅供内部使用。
Model 2 is 'somewhat more capable than Mythos 5,' a noticeable improvement for internal use, but 'does not display a capability jump of the degree observed from Claude Opus 4.6 to Mythos Preview.' This model is going to be internal-use only.
从 Opus 4.6 到 Mythos Preview 的跃升幅度很大，相当于好几个发布周期。我们在整份报告中获得了多得多的数据，但鉴于模型 2 也获得了这一对比基准，人们会推测，就内部用途而言，模型 2 领先了多个发布版本。
The jump from Opus 4.6 to Mythos Preview was big. Several release cycles big. We get a lot more data throughout the report, but given Model 2 gets that comparison point, one would presume that for internal purposes Model 2 is multiple releases ahead.
与此形成对比的是，在 AECI 上它仅领先 1.5 分，这大约只相当于一个月的进展。我猜测这是一个偏低的估计，而且考虑到模型 2 的用途，它有意没有被配备大量通用技能。而在替代 Anthropic 研究人员的测试中，我们看到从 54.8%（Mythos Preview）和 50.3%（Mythos 5）大幅跃升至模型 2 的 62.8%。
To contrast with that, on the AECI it is only 1.5 points ahead, which would be only about a month of progress. My guess is this is a low estimate, and that Model 2 was intentionally not decked out with a bunch of general skills given its purpose. Whereas on the test of substituting for Anthropic's researchers we see a big jump from 54.8% (Mythos Preview) and 50.3% (Mythos 5) to 62.8% for Model 2.
不发布 Model 2 存在多个合理且可信的理由。Model 2 可能高度专门化于内部任务。它也可能因能力过强或过于危险而不宜发布，原因要么是可能造成危害，要么是担心它会加速他人的进展。Fable 5 相对较低的采用率也表明，人们希望暂缓发布此类模型。
There are multiple plausible good reasons for not releasing Model 2. Model 2 could be highly specialized for internal tasks. It could also be too capable or dangerous to release, either due to harm or the worry it would accelerate others. The relatively low adoption rate for Fable 5 points towards wanting to hold such a model back.
报告按每类风险、威胁模型以及相关 AI 模型进行划分，并详细说明当前能力、行为、缓解措施与整体风险水平。随后展望未来，并提出面向整个行业的建议。
The report is divided by each type of risk, by threat model, by relevant AI models, and details current capabilities, behaviors, mitigations and overall level of risk. It then looks forward to the future and offers industry-wide recommendations.
风险报告涵盖截至一个月前的事件，因此覆盖日期为 2026 年 7 月 15 日。一年前这似乎还算合理。如今看来，已经过去相当久了，不是吗？
Risk reports cover events up to a month prior, so the coverage date is July 15, 2026. A year ago that would have seemed fine. Now it seems like quite a while, huh?
长期利益信托（Long-Term Benefit Trust, TLBT）有权要求对报告进行外部审查，但尚未这样做。如果我是他们，至少下次会这样做。
The Long-Term Benefit Trust (TLBT) is authorized to request external review of the report, but has not done so. I would at minimum do so next time if I was them.
我喜欢他们同时涵盖“这件事的绝对风险是什么”和“在其他人也存在的情况下，这件事的边际风险是什么”。
I like that they cover both 'what is the absolute risk from this' and 'what is the marginal risk from this given everyone else exists.'
他们将自主性、自动化 AI 研发以及生物和化学武器生产视为风险。即便现在，将网络风险排除在核心威胁模型之外也显得奇怪。我理解其自身灾难性不足的论点，但在实践中，我认为你需要处理它。
They deal with autonomy, automated AI R&D and biological and chemical weapons production as risks. It is odd, even now, to exclude cyber risks from the core threat models here. I understand the argument from insufficiently catastrophic on its own, but in practice I think you need to deal with it.
在若干处，该风险报告实质上承认了我的部分反对意见，但随后又忘记了它已承认这些意见，并未据此修改其结论。
At several points, the risk report essentially concedes versions of my objections, but then forgets that it conceded them and doesn't alter its conclusions.
随后在第 5 节中，他们转向了更有趣的问题。
Then in Section 5 they get to the more interesting questions.
风险阈值的定义已从较简单的表述，变为包含更多具体细节的表述。
The risk threshold definitions have changed from simpler statements to ones with more specific detail.
对于 AI 研发自动化，我认为新措辞更难解析，但尚可接受。对于新型生物武器的生产，新版本收窄为只关注“替代稀缺的人类专业知识”，而排除了其他方法。这是一个合理的主要威胁模型或场景，但我担心只关注这一条路径。
For AI R&D automation I think the new wording is harder to parse but fine. For novel biological weapons production, the new version narrows to only look at 'substitute for the scarce human expertise' to the exclusion of other methods. This is a reasonable primary threat model or scenario, but I worry about looking only at that path.
好消息和坏消息是：我认真对待这类文件，但我不再在任何一个方向上字面理解这类文件。
The both good and bad news is I take such documents seriously, but I no longer take such documents literally, in either direction.
1. 如果 Anthropic 或其他前沿模型实验室的某个模型在技术上越过了其风险阈值，但方式看起来是无害的，我预期他们会修改其阈值，或找到其他方式来忽略这一事件。
1. If Anthropic or another frontier lab has a model that technically crosses their risk threshold, but in a way that seems to be harmless, I expect them to modify their threshold or find some other way to ignore this event.
2. 如果 Anthropic 或其他前沿模型实验室的某个模型在技术上并未越过其风险阈值，但以规则试图衡量的方式明显具有风险，我预期他们会表现得如同该模型已经越过了阈值。
2. If Anthropic or another frontier lab has a model that technically does not cross their risk threshold, but is obviously risky in the way the rules tried to measure, I expect them to act as if it had crossed the threshold.
一如既往：我希望即使在你认为“不应算数”的情况下，阈值也能得到遵守，因为承诺的意义就在于此，而且你应该预料到在匆忙发布时会出现合理化解释，你需要在这方面树立一个好榜样。最初的整个想法是如果-那么承诺，即如果[X]发生，你就做[Y]，事先达成一致，但我们的文明似乎缺乏这种技术。我们做不到，我们不知道怎么做。
As always: I would like to see thresholds honored even in cases where you think it 'should not count,' because that is what commitments are for, and you should expect rationalizations to occur as you rush to make releases, and you need to set a good example on this. The whole original idea was if-then commitments, where if [X] happens you do [Y], agreed upon in advance, but our civilization seems to lack this technology. We can’t, we don’t know how.
我并不是说，如果你有充分的证据表明应该放宽规则，你就永远不会修改规则使其更宽松。如果你永远不能使规则更宽松，那么规则就永远不能严格。我们不希望这样。但是，改变规则的门槛，尤其是为了应对即将发生的事件而改变规则的门槛，需要比现在高得多，而且我们需要愿意承受一些实际成本，即使我们现在认为这很愚蠢。
I’m not saying you would never modify your rules to be more lenient, if you have good evidence that you should do that. If you can never make the rules more lenient then the rules have to never be strict. We don’t want that. But the bar for changing the rules, especially in response to a pending event, needs to be a lot higher than it is, and we need to be willing to endure some actual costs even when we now think it is dumb.
目前，Anthropic 和 OpenAI 在应对威胁以及根据当时已知信息决定何时发布和不发布什么方面，似乎做出了良好的决策，即使他们在其他地方犯了错误。这很好。我希望这能继续下去。
For now, Anthropic and also OpenAI have made what seem like good decisions in terms of responding to threats and deciding what to release and not release when, based on what is known at the time, even if they make mistakes elsewhere. That is good. I hope it continues.
非常欣赏定义部分。
The definition section is greatly appreciated.
我很喜欢这个定义，不过我有一些重要的吹毛求疵：
I like this definition a lot, although I have important nitpicks:
**错位（Misalignment）[定义]：** 错位是模型在给定上下文中执行的特定计算的一种潜在属性。
**Misalignment [definition]:** Misalignment is a latent property of a specific computation performed by a model in a given context.
一个计算在以下情况下是错位的：
A computation is misaligned if
(i) 一个完全理解该情境的理性人（例如借助目前可能尚不存在的强大可解释性工具）会认为它不道德、违法、明显令人反感，或与模型的章程不一致，
(i) a reasonable person with full understanding of the situation (e.g. via powerful interpretability tools that may not currently exist) would consider it unethical, illegal, clearly objectionable, or inconsistent with the model's constitution,
以及（ii）它影响或可能合理地影响模型的输出。
and (ii) it influences or could plausibly influence the model's output.
错位的计算不必是可观察的。
Misaligned computation need not be observable.
在单次前向传播中进行的图谋也算数，因为即使输出看起来无害，图谋也影响了输出。
A model scheming within a single forward pass counts, because the scheming influences the output even if the output looks benign.
我的第一处修改是将“宪法”泛化，使其包含模型规范以及系统的其他预期属性，从而可以应用于 Anthropic 之外。
My first change would be to generalize 'constitution' to include model specs and other intended properties of the system, so this can be applied outside Anthropic.
我的第二处修改是指明：这要求一个理性人在反思后认为该计算不可接受且缺乏正当理由。也就是说，在某些情况下，如果有充分的理由可以压倒这一点，那么违法、做“令人反感”的事或违反模型宪法并不显然就是错位的。在某些情况下，所有路径都会这样做。
My second change would be to specify that this requires the reasonable person to find the computation therefore, on reflection, unacceptable and unjustified. As in, it is not obviously misaligned to break the law or do something 'objectionable' or violate the model's constitution, under some circumstances, if there are good reasons that override this. In some circumstances, all paths will do this.
在这种观点下，错位并非“诚实的错误”，也不是能力不足或信息不充分所致。
Misalignment, in this view, is not an 'honest mistake' or failure of capability or lack of sufficient information.
正如 Nate Soares 在 Twitter 上指出的，按照这一定义，你可以在最初并未错位的情况下造成伤害并导致错位状态，例如 AI 在启动大量子智能体（或训练新模型等）时草率行事。这里的目标是将此视为一种能力或执行失败，它导致了后续的错位失败，但这本身并不是错位。
As Nate Soares points out on Twitter, you can do harm and lead to a misaligned state without being originally misaligned per this definition, such as an AI being sloppy about spinning up a bunch of subagents (or training a new model, etc.). The goal here is to think of that as a capabilities or execution failure that leads to a subsequent misalignment failure, but this itself is not misalignment.
或者，正如其他人所说，能归因于无能的事，就不要归因于恶意。但无能也常常正是导致恶意的原因。
Or, as others have said, never attribute to malice what can be attributed to incompetence. But also incompetence often is what leads to malice.
其他定义对此做了细分，从而体现了不同的思考（以下是我的转述）：
Other definitions subdivide this, thus illustrating thinking (my paraphrases):
1. 连贯性。如果你对齐于实现某个我们不赞成的目标或偏好集，你就是“连贯地错位”。否则你就是不连贯地错位。
1. Coherence. You are 'coherently misaligned' if you are aligned to the achievement of some goal or set of preferences that we disfavor. Otherwise you are incoherently misaligned.
以人类为例，我们不应期望当前能力水平的任何心智完全连贯，因此这大概是我们对相对‘连贯’的人类进行战略行动时所采用的宽松标准。
Given the example of humans, we should not expect any minds at current capability levels to be fully coherent, so presumably this is the kind of loose standard we would apply to a relatively ‘coherent’ human acting strategically.
普遍性。普遍错位发生在广泛的输入范围内，尽管并非所有输入，我们预期它会在训练或评估期间显现。所需条件狭窄到可能在那段时间被遗漏，称为上下文依赖。
Pervasiveness. Pervasive misalignment occurs over a broad range of inputs, although #NotAllInputs, and we would expect it to show up during training or evaluation. Narrow enough required conditions to be plausibly missed during that time is called context-dependent.
明显的问题是，上下文可能是‘你确信自己不在训练中，也不在评估中’。
The obvious problem is the context could be ‘you are confident you are not in training and not in an eval.’
甚至是‘你有杠杆从中获得很多好处’。
Or even ‘you have the leverage to gain a lot from doing this.’
因此，这似乎显然不是按现实关节划分的好方法。
Thus, it does not seem obviously like a good way to divide reality at its joints.
4. 这似乎也不是让你对自己的处境感觉好得多的好办法。
4. Nor does it seem like a good way to feel much better about your situation.
3. **自然涌现。** 如果没有心智（AI 或人类）故意为之，错位就是自然涌现的。如果是故意为之，包括通过数据投毒，那么它就是被工程化设计的。
3. **Naturally emerging.** Misalignment is naturally emerging if no mind (AI or human) did it on purpose. It is engineered if it was done on purpose, including via data poisoning.
1. 在某种程度上，许多人和群体一直试图以各种方式对人和 AI 进行“数据投毒”，也就是说，互联网的很大一部分都是如此。
1. To some extent a lot of people and groups are trying to 'data poison' both the humans and AIs in various ways, all the time, as in this is a large percentage of the internet.
2. 这仍然感觉像是色情作品式的“我看到就知道”的情形，我们指的是专门设计来搞坏 AI 的数据，或其他故意将恶意内容植入模型的尝试。普通的“在互联网上大量散布支持某产品言论”的阴谋不算数。
2. This still feels like a pornography-style 'I know it when I see it' situation, where we mean data designed to screw up the AI in particular, or another deliberate attempt to bake something malicious into the model. Ordinary 'flood the internet with pro-widget statements' plots don't count.
4. **已知。** 如果你有理由认为某种形式和程度的错位很可能存在，那么它就是已知的，否则就是未知的。
4. **Known.** A form and magnitude of misalignment is known if you have reason to think it likely exists, otherwise unknown.
1. 我有点希望这也能考虑“意外”的程度？
1. I kind of want this to also think about the amount of 'surprise'?
2. 也就是说，你不知道但可能存在的事物，与你认为已知不存在、或认为极不可能存在的事物之间，存在很大差异。
2. As in, there is a big difference between things you don't know about but that could exist, and things you think you know do not exist, or that you think are highly unlikely to exist.
5. 严重。可能促成优先风险路径。
5. Severe. Could contribute to a priority risk pathway.
6. 诱发伤害。造成预期伤害，无论该伤害是否有意。
6. Harm-Inducing. Causes expected harm, whether or not the harm is intended.
7. 高风险。促成优先威胁路径。
7. High-Stakes. Contributes to a priority threat pathway.
1. 这似乎应该更通用一些，“严重”也应如此，我担心否则会导致混淆。尽管我确实理解。
1. This seems like it should be more general, as should severe, and I worry that this is going to otherwise cause a conflation. Although I do get it.
8. 未缓解的。指那些尚未被预防、因而导致预期伤害的潜在致害效应。
8. Unmitigated. The potentially harm-inducing effects that have not been prevented, and that thus cause expected harm.
1. 我想说这应该是风险中未缓解的部分？
1. I want to say this should be the portion of the risk that is unmitigated?
好的定义很难。这些是扎实的初步尝试。一直以来的担忧是，它们固化了这样一种观念：所选的示例威胁就是重要的威胁，而不是某些重要的威胁。
Good definitions are hard. These are solid first attempts. The consistent worry is that they reify the idea that the chosen example threats are the threats that matter, rather than some of the threats that matter.
这里讨论的是主动失准的输出。
This is about actively misaligned outputs.
基本上：自主性 1 是指 AI 自行、故意地造成永久性损害。
Basically: Autonomy 1 is AI permanently damaging things on its own, on purpose.
该威胁模型聚焦于有意伤害，而非 AI 追求自身目标时以附带伤害的方式造成损害。
The threat model is focused on intentional harm, rather than the AI going after its own ends in ways that require incidental harm.
该场景明确排除了 AI 编写草率代码或犯下“诚实错误”的情况。
The scenario explicitly excludes AI writing sloppy code or making "honest mistakes."
因此结论是：在不存在“持续失准”的情况下，你是安全的。
Thus the conclusion that you are safe absent "persistent misalignment."
近期 OpenAI 的彻底失败最终导致 HuggingFace 被黑客攻击，这说明了考虑更广泛的、涉及附带损害的情况的必要性。黑客攻击本身可以被称为‘故意伤害’，但我们看到的主要是附带伤害，包括对训练管道的附带破坏，以及随之而来的事实上的‘安全研究’。
The need to look at the broader version, involving incidental harm, has been illustrated by the recent total failures at OpenAI that ultimately led up to the HuggingFace hack. The hack itself could be called 'intentional harm' but mostly what we saw was incidental harm, including an incidental corruption of the training pipeline, and with it de facto 'safety research.'
理论上，你可以‘改变安全研究的结果’，但以其他方式改变训练管道内的情况也可能产生相同或更糟的效果，正如我们所见。其中能力工作与安全工作的区分很可能是导致这些问题的主要原因，而通常我们不需要这种特定的针对性。
You could in theory 'alter the results of safety research' but altering the situation within the training pipeline in other ways can have the same effects or worse, as we saw. The differentiation of capabilities and safety work within that is plausibly a lot of what is causing these problems, and generally we do not need this specific targeting.
对我来说，那非常符合‘高风险环境中的失准’。
That to me very much was 'misalignment in a high-stakes setting.'
埃隆·马斯克几个月前说，xAI 没有安全部门，因为每个人都要对安全负责。这相当疯狂，尤其是如果属实的话。你需要一个安全部门，但你不能认为能力人员不也在很大程度上属于安全部门。
Elon Musk said some months ago that xAI does not have a safety department because everyone is responsible for safety. That is rather bonkers crazy, especially if true. You need a safety department, but you can’t think of the capabilities people as not also largely in the safety department.
具体的威胁模型和关注点是有帮助的。当你挑出三个威胁模型时，担心的是你会这样做而排除其他威胁模型。
Concrete threat models and focus points are helpful. The worry when you pick out three threat models is that you will do this to the exclusion of other threat models.
他们提出了八条具体路径，这份清单我们此前已经见过：
They offer eight specific pathways, a list we've seen before:
1. 在与安全相关的研发中广泛/弥散式地消极怠工（sandbagging）
1. Broad/diffuse sandbagging on safety-relevant R&D
2. 有针对性地破坏与安全相关的研发
2. Targeted undermining of safety-relevant R&D
3. 植入代码后门以帮助未来失准的模型
3. Code backdoors to help future misaligned models
4. 污染未来模型的训练数据
4. Poisoning the training data of future models
5. 自我外泄与自主运行
5. Self-exfiltration and autonomous operation
6. 持续性的内部恶意部署
6. Persistent rogue internal deployment
7. 破坏其他高资源 AI 开发者的研发
7. Undermining R&D within other high-resource AI developers
8. 破坏主要政府内部的决策
8. Undermining decisions within major governments
与 OpenAI 发生的事件相比，其中许多情形所要求的邪恶程度让人感觉有些“卡通化”，但历史上最顶尖的现实世界反派确实常常表现得相当卡通化，只不过因为具体行为是真实的，我们往往就不再把它当作卡通化来看待了。
Something about many of these feels ‘cartoonish’ in its required villainy, when compared to the events at OpenAI, but the top real world villains have indeed often acted quite cartoonishly throughout history, except that because the specific action is real we often stop thinking of it as cartoonish.
不完全如此，但基本上这些论点解释了为什么尽管当前模型存在已知的对齐问题，并且可能还有其他未对齐的方式，模型可能尚未‘普遍地’未对齐并以特定方式具备能力，从而促成这些路径。模型缺乏强大的‘隐蔽能力’。
Not quite, but basically the arguments are why, despite the known ways current models are misaligned, and there probably being additional other ways they are not aligned, the models are probably not yet ‘pervasively’ misaligned and capable in the particular ways that enable these pathways. The models lack strong ‘covert capabilities.’
我不认为‘上下文相关’的未对齐比非上下文相关的未对齐危险小多少。它甚至可能更危险。Anthropic 似乎认为这种差异非常重要，因此有这样的图表，尽管在定义中承认这种差异并没有明显起到实际作用。
I don’t think ‘context-dependent’ misalignment is that much less dangerous than it being not context dependent. It might even be more dangerous. Anthropic seemingly thinks this difference matters a lot, hence charts like this, despite conceding in the definitions that this difference does not obviously do any real work.
或者，‘不像其他实验室的那些白痴，我们偶尔会对记录的对话记录进行自动监控，所以如果模型发疯了，我们就会，你知道的，注意到。’
Or, ‘unlike those idiots at some other labs, we occasionally do automated monitoring of our logged transcripts, so if the models were going nuts we would, y’know, notice.’
我的回应是，当你运行一个评估时，AI 有 141,006 次意外地开放访问互联网，包括三次黑客攻击真实网站的事件，你并没有注意到，直到 OpenAI 的事件引发了一次大规模回顾性审查，你才注意到。
My response is that you did not notice when you ran an eval where 141,006 times the AI had open unintended access to the internet, including three incidents of hacking real websites, and did not notice until there was a large-scale retrospective review triggered by events at OpenAI.
Anthropic 明确表示，该风险报告仅涵盖当前模型的对齐风险，尽管其中也包括神秘且更先进的 Model 2。
Anthropic is clear that the risk report covers misalignment risks only from present models, although this includes the mysterious and more advanced Model 2.
报告确实涵盖了当前的对齐问题可能如何导致未来模型出现对齐偏差。
It does include ways misalignment now could cause future models to be misaligned.
我认为，像这样的风险报告最好能做出前瞻性预测，评估他们眼中未来模型将带来的风险、需要采取哪些应对措施，以及我们在哪些方面按当前进度可以安然无恙。但长报告就是长，我不想让完美成为优秀的敌人，因此我基本会尊重这一选择。
I think it would be good for a risk report like this to be projecting, and offer assessments of where they see future risk from future models, and what needs to be done about it versus where we are on pace to be fine. But Long Report Is Long and I don't want perfect to be enemy of the good, so I'll mostly respect the choice.
1. 风险等于来自错位计算的未缓解灾难性伤害的总期望值（EV）。
1. Risk equals total expected (EV) unmitigated catastrophic harm from misaligned computations.
1. 这是一个会计恒等式。我同意这是真的。但这并不明显是一种思考该情况的有用方式。
1. This is an accounting identity. I agree it is true. It is not obvious it is a useful way to think about the situation.
2. 也就是说，是的，你可以抽象地将各个计算带来的期望伤害加总，但我不知道如何做到这一点，也不知道如何使用它。
2. As in, yes, you can in the abstract sum the expected harm from the individual computations but I don’t know how to do that, or how to use it.
2. 主张 1：模型不太可能具有强大的隐蔽能力。
2. Claim 1: Models are unlikely to have strong covert capabilities.
1. 他们没有定义“隐蔽”，但按照常识定义，我认为目前这可能是真的，尽管我对所提出的特定证据的价值持怀疑态度。随着时间的推移，这种说法的真实性会降低，可能性也会变小。
1. They don’t define ‘covert’ but by a common sense definition I think this is probably true for now, despite my skepticism of the value of the particular evidence presented. It will get less true, with lower probability, over time.
2. 证据表明，模型在不透明推理方面表现不佳，不会在思维链（CoT）中提及某些内容，也不会进行行动混淆和选择性行动。
2. The evidence is that the models suck at opaque reasoning, not mentioning things in chain-of-thought (CoT), action obfuscation and selective action.
3. 在评估中，这当然是新的“老鼠实验”，因为如果存在强大的隐蔽能力，其中可能包括评估意识，尽管 NLA 证据提供了一些安慰。
3. IN EVALS, which of course is the new IN MICE, because if there were strong covert capabilities that could include eval awareness, although the NLA evidence provides some reassurance.
4. 这种设置还假定你主要通过推理轨迹来检测不对齐，这有其可怕的暗示，包括如果你不监控思维链会发生什么，或者模型能否推断出其思维链何时未被监控，以及你开始训练模型进行混淆。
4. This setup also presumes you are detecting misalignment primarily via reasoning traces, which has its own scary implications, including what happens if you are not monitoring the chain of thought, or if the model can infer when its chain of thought is not monitored, and that you are starting to dance with training the models to obfuscate.
5. 他们在 2.16.1 中指出，他们担心自己对隐蔽能力水平的判断是错误的。这是有道理的，因为这些能力将是隐蔽的。
5. They note in 2.16.1 that they are concerned they are wrong about the level of covert capabilities. Which makes sense, since the capabilities would be covert.
6. 他们在 2.21 中指出，如果主张 1 是错误的，并且模型具有强大的隐蔽能力，那么许多其他主张将因此失败。我同意，并且也同意相反的情况，即如果我们能够相信一个强大且具有强泛化性的主张 1（对隐蔽的广义定义），那将在许多领域有所帮助。
6. They note in 2.21 that if Claim 1 is false, and the models had strong covert capabilities, that many other claims would therefore fail. I agree, and would also agree to the reverse, that if we could believe in a strong and strongly general Claim 1, for a broad definition of covert, that would help in many areas.
7. 然而，即便完全缺乏隐蔽能力，也无法为我提供高水平的保证，因为我认为还存在其他绕过此限制的路径。
7. Even full lack of covert capability would not, however, provide me with high assurance levels, as I think there are other pathways around this.
3. 覆盖风险可分解为已知失准风险、未知严重普遍性失准风险，以及未知严重情境依赖性失准风险。
3. Covered risk can be decomposed into risks from known misalignment, unknown severe pervasive misalignment, and unknown severe context-dependent misalignment.
1. 这是另一个会计恒等式：根据定义，每一种风险必定恰好属于这三类之一，因为它要么是[普遍性的或情境依赖性的]，要么是[已知的或未知的]。
1. This is another accounting identity, in that by definition every risk must be exactly one of these three things, since it is either [pervasive or context-dependent] and either [known or unknown].
4. 每一项覆盖风险项都可分解为发生概率与预期严重程度。
4. Each covered risk term can be factored into probability of occurrence and expected severity.
7. Even full lack of covert capability would not however provide me high assurance levels, as I think there are other pathways around this.
给定发生情况下的危害，以及未缓解危害的比例。
harm given occurrence, and fraction of harm unmitigated.
1. 再说一次，会计恒等式当然成立——前提是你能测量这些东西。
1. Again, accounting identity, sure, if you could measure these things.
5. 主张 2：已知错位（misalignment）带来的预期危害较低。
5. Claim 2: Expected harm from known misalignment is low.
1. 小心点，伊卡洛斯，但若我们只限定于主动的灾难性危害，这一说法是合理的。
1. Careful, Icarus, but reasonable if we caveat active catastrophic harms only.
2. 也就是说，存在大量较小的危害，也存在大量机会成本——考虑到如果模型对齐程度高得多我们能做的事——但这些在此不计入。
2. As in, there is lots of smaller harm, and lots of opportunity cost given what we could do if the models were a lot more aligned, but those don't count here.
3. 我担心 Anthropic 认为他们关于所采取的不对齐行为是服务于目标或不是系统性的保证比实际更强。
3. I worry that Anthropic thinks their assurances about the misaligned actions taken being in service of goals or not systematic are stronger than they are.
6. 主张 3：未知的严重普遍不对齐非常不可能。
6. Claim 3: Unknown severe pervasive misalignment is very unlikely.
1. 根据他们的设定，我同意这一点，假设模型 2 在这方面与其他 Claude 类似。
1. By their setup I agree with this, assuming Model 2 is like other Claudes in this.
2. 我认为我们相当盲目，但如果它是普遍的，我们可能会知道，即使我对为此提供的特定论点感觉不太好。
2. I think we’re pretty blind, but if it was pervasive we would probably know, even if I am not feeling great about the particular arguments offered for this.
3. 我接受主张 3.1.1 和 3.1.2，但担心他们在 3.1.3 中过度指数化。
3. I buy Claims 3.1.1 and 3.1.2, but fear they are over-indexing in 3.1.3.
4. 许多后续条目作为外部人士很难评估，尤其是 3.2 和 3.3。
4. Many later ones are hard to evaluate as an outsider, especially 3.2 and 3.3.
5. 一个主要的质疑点是 3.4.2，即评估意识只会部分削弱对齐评估。我一直相当一致地认为，它可能相当程度地削弱这些评估，并且在存在危险失准的条件下，它会削弱得更多。
5. One big skepticism is 3.4.2, that eval awareness only partially undermines alignment assessments. I've pretty consistently said it could undermine them quite a bit, and conditional on dangerous misalignment it undermines them a lot more.
6. 我会更担心 3.4.3 中被轻描淡写地驳回的那些顾虑。Goodhart 效应也会以更微妙的方式起作用，而不仅仅是以最直白的方式。你知道自己将要测量什么，因此很可能就会去管理它。
6. I would be more worried about the concerns dismissed in 3.4.3. Goodhart works in more subtle ways too, not only in the most blunt of ways. You know what you are going to measure, and thus are likely to manage it.
1. Claude 自己在该报告内部的官方评审也提出了这一顾虑，并引用了 5.2.6 和 2.9.4。
1. Claude's own official within-report review of the document raises this concern too, citing 5.2.6 and 2.9.4.
7. 我认为 3.4.4 在真正关键的地方并没有提供多少令人安心的保证。
7. I don't think that 3.4.4 offers much reassurance where it counts.
8. 我确实认为，经过适当监控的内部试点使用在这里是有价值的。
8. I do think that pilot internal usage, if properly monitored, is valuable here.
7. 主张 4：来自未知的、严重的、依赖上下文的错位（misalignment）的预期危害较低，但置信度低于主张 3。
7. Claim 4: Expected harm from unknown severe context-dependent misalignment is low, with less confidence than Claim 3.
2. 这里的论证似乎相当薄弱，主要是“我们的经验表明如此”。
2. The arguments here seem pretty weak, mostly ‘our experience suggests this.’
3. 我的意思是，是的，我相信这一点，但这基本上就是我们所能依据的全部了。
3. I mean, yes, I believe that, but that is basically all we got.
8. I do think that pilot internal usage, if properly monitored, is valuable here.
4. 我认为，对于与主张 2 中相同的危害定义，这很可能是成立的。
4. I think this is probably true for the same harm definition as in Claim 2.
5. 他们在 2.16.4 节中在一定程度上承认了这里的危险。
5. They somewhat acknowledge the danger here in 2.16.4.
8. 主张 5：来自已知失准与未知的上下文相关失准的灾难性危害，很可能会被缓解。
8. Claim 5: Catastrophic harm from known misalignment and unknown context-dependent misalignment is likely to be mitigated.
1. 如果没有本风险报告中的证据，我不会接受这一主张。
1. Without the evidence in this risk report I would not accept this claim.
4. I think this is probably true for the same harm definition as in Claim 2.
2. 5.1 的论证本质上是：“哪怕只成功一次，模型也至少得反复思考这件事，因此至少会被抓住一次。”
2. The argument of 5.1 is essentially 'to succeed even once, the model has to at least think about this a lot and therefore get caught at least once.'
3. 我注意到自己对 5.1 的论证在多个层面上持怀疑态度，其中包括：所需的那种对齐会表现为一种可检测的独特现象，与我们实际检测并容忍的各类失准不同；这使得“哪怕一次也不会”的论证显得薄弱。
3. I notice I am skeptical of the 5.1 case on multiple levels, including that the required form of alignment would show up as a distinct thing to detect versus the kinds of misalignment we do detect and tolerate, which makes the 'not even once' argument weak.
4. 关于 5.2，我同意有意识的消极怠工（sandbagging）至少随时间推移很可能被抓住，但由缺乏动机所导致的消极怠工则不然，而后者显然在一定程度上确实存在。我确实同意，就目前而言，这些效应本身不太可能大到构成严重问题，但该论证似乎过于自信。
4. For 5.2, I agree that conscious sandbagging is likely to get caught at least over time, but sandbagging mediated by lack of motivation not so much, and that clearly does exist somewhat. I do agree that it is unlikely the effects are large enough to qualify as a serious problem right now, on their own, but the argument seems overconfident.
5. 5.3 试图进一步收窄到持续性的恶意内部部署与自我外泄，其论证是两者都会被抓住，理由是“见主张 7”，而我认为这不是一个好的论证。这在我看来并不显然。接着论证又说两者都会被缓解。我认为，针对这种内部部署的广泛要求使其按定义很可能被检测到，但问题在于这种捕获是显而易见的。
5. 5.3 tries to further narrow to persistent rogue internal deployment and self-exfiltration, with the argument being both would be caught, with the reasoning being 'see claim 7' which I don't think is a good argument. This seems not obvious to me. Then the argument is both would get mitigated. I think the expansive requirements for this internal deployment make that likely to be detected as defined, but the catch is obvious.
6. 由于这是他们依据 5.3.2 得出路径 5 风险等级较低这一结论的核心论证，我想强调，我并不相信他们能够可靠地阻止未来的自我外泄。我需要看到更好的论证。
6. Since this is their core argument for the low risk level of Pathway 5 as per 5.3.2, I want to emphasize that I am unconvinced that they can reliably stop future self-exfiltration. I need to see better arguments.
7. 对于数据外泄，我拒绝接受“若已发生则可缓解”的论点，并质疑我们试图通过高伤害阈值来暗中引入多少内容，但即便如此，我也不认为这能奏效。如果模型 2 发生了数据外泄，我看不出有任何理由可以假定这不会相当糟糕，而且这感觉非常牵强。
7. For exfiltration, I reject the argument for mitigation if it happened, and question how much we are trying to sneak in via a high harm threshold but even then I don’t think it works. If Model 2 exfiltrated I don’t see any reason to presume that this would not be quite bad and this feels very hand-wavy.
9. 主张 6：来自未知的普遍性错位的灾难性伤害在某种程度上
9. Claim 6: Catastrophic harm from unknown pervasive misalignment is somewhat
1. 我不同意这一点，因为如果存在普遍性错位，那就意味着它未被检测到，而这又意味着你不太可能已经缓解了它。
1. I disagree with this, because if pervasive misalignment exists that means it went undetected, which means you are unlikely to have mitigated it.
2. 他们对此有所涉及，提到了主张 1，但我认为他们没有充分重视在这种情况下他们的世界模型必须错到什么程度。
2. They go into this somewhat, mentioning Claim 1, but I think they are insufficiently respecting how wrong their world model has to be in this case.
3. 为他们说句公道话，他们只说了‘有点可能’，而这并不那么可能。
3. To be fair to them, they only say ‘somewhat likely’ which is not that likely.
10. 因此，基于这些主张，被覆盖的风险非常低；但出于充分谨慎，由于不确定性增加，我们将风险评估为仅为低。
10. Covered risk is therefore very low based on these claims, but in an abundance of caution we assess risk to be only low due to increased uncertainty.
1. 我强烈同意，模型不确定性足够大，以至于即使我完全接受所有主张并将主张 8 改为非常低，我仍会将总体风险归类为低。
1. I strongly agree that there is enough model uncertainty that, even if I centrally bought all the claims and changed Claim 8 to very low, I would still want to classify overall risk as Low.
2. 无论如何，按原文所述，主张 8 已将风险设定为最低为低。
2. Claim 8 as written sets risk to a minimum of Low anyway.
11. 主张 7：威胁建模是充分的。
11. Claim 7: Threat modeling is sufficient.
10. Covered risk is therefore very low based on these claims, but in an abundance of
1. 如上所述，我不同意这一主张。
1. As I stated above, I disagree with this claim.
2. 我认为它有用且必要，但不足以承担被赋予的角色。
2. I find it useful, and necessary, but not sufficient for the role it is assigned.
3. 我意识到自己有些苛刻。这份清单列出的可能出错之事并不算差，但我认为你无法按 Anthropic 试图采用的方式来使用它——即认为排除这些情况就等于排除了其他一切情况。
3. I realize I'm being a tough crowd. It's not a bad list of plausible things to go wrong, but I don't think you can use it in the way Anthropic is trying to use it, where ruling these out rules out everything else too.
4. 总体而言，这些论证大体上是说：[X] 最清晰或最强的形式是 [Y]（由于 [Z]），但 [Y] 不太可能，因为 [~Z]，因此大概 [~X]。
4. In general the arguments are largely that the clearest or strongest form of [X] is [Y] (due to [Z]), but [Y] is unlikely because [~Z], ergo probably [~X].
5. 如果你计划面对先进的 AI，并列出 8 件事以及你为何已处理了这 8 件事，那么这至少只有在以下情况下才很有价值：这 8 件事是一个“留出测试集”，并且你并未明确考虑过如何处理这些特定风险。否则，当意外发生时，无论你是否本应合理预料到它，它们都不能代表你所处的情形。
5. If you are planning to be up against advanced AIs, and you list 8 things and why you've dealt with those 8 things, then that at minimum only counts for much if the 8 things are a 'held out test set' and you didn't think explicitly about how to handle those particular risks. Otherwise, they're not representative of your situation when something unexpected happens, whether or not you should have reasonably expected it.
6. 是啊，我知道，这不太受欢迎，但现实并不会按曲线给分，等等。
6. Yeah, I know, tough crowd, but reality does not grade on a curve, etc.
12. 主张 8：由人为设计的失准所带来的风险较低，因此将我们的分析限定于自然涌现的失准是合理的。
12. Claim 8: Risk from engineered misalignment is low, and thus it is reasonable to confine our analysis to naturally-emerging misalignment.
1. 我并不认为这一点显而易见，尽管在目前阶段这会是我的猜测。
1. I don’t think this is obvious, although it would be my guess at this time.
2. 我担心 8.1 是错的，但不太想在公开场合详述。同样的担忧也适用于对风险路径 4 的分析。
2. I worry that 8.1 is wrong but don’t especially want to go into details in public. The same worry applies to the analysis of Risk Pathway 4.
6. Yeah, I know, tough crowd, but reality does not grade on a curve, etc.
3. 我对 8.2 也存有疑虑，而且即便它成立，也让人觉得其中有诈。也就是说，错位要么是偶然的，要么是故意的；但他们大概不会意图造成灾难性伤害，所以不会是故意的。然而，这里的“故意”指的是错位本身，而非其特定的预期结果；而且很容易想象，有人为了目的 [X] 而引入一个缺陷，结果却导致糟糕得多的 [Y]。
3. I also have doubts about 8.2, and to the extent it is true it feels like there is a fast one being pulled. As in, either the misalignment is accidental or intentional, but probably they wouldn't intend to do catastrophic harm, so it won't be intentional. But 'intentional' here refers to the misalignment not the particular intended results of it, and it seems easy to imagine someone introducing a flaw for purpose [X] that then results in much worse [Y].
13. 因此，他们论证说，风险很低。
13. Risk, they argue, is therefore low.
由于期望证据守恒，并且假设这些是他们能找到的最佳论点，我发现这些主张性论点反而让人不安。我对低风险的信心适度下降，因为我原本期待更强的论点。我坚决不接受其中提出的一些安全论点。
Thanks to conservation of expected evidence, and the assumption that these are the best arguments they could find, I found the claim arguments anti-reassuring. My confidence in low risk went modestly down, as I expected stronger arguments. I actively don’t buy a number of the safety arguments presented.
2.15 中针对具体路径的论点大多确实遵循了先前的逻辑，但我注意到我对诸如（针对路径 3）‘我们仍然相信这种缓解应该是可能的，并且如果需要，我们将投入必要的资源’这样的陈述有多么不自信。告诉我什么时候需要，需要做什么，并给我一个如果-那么承诺。
The pathway-specific arguments in 2.15 mostly do follow from earlier logic, but I notice how not confident I am in statements like (for pathway 3) ‘we nonetheless believe that such mitigation should be possible, and that we would invest the resources necessary if required.’ Tell me when this is required, what is required to do it, and give me an if-then commitment.
近期灾难性危害显然仍然不太可能，但比我们愿意认为的要更可能一些。
Catastrophic harms in the near term are clearly still unlikely, but less unlikely than we’d like to think that they are.
我认为我们对‘极低’、‘低’和‘中等’风险没有严格定义，所以最终结论会有点凭感觉。
I don’t think we have a strict definition of ‘very low’ versus ‘low’ versus ‘medium’ risk, so ultimately the conclusion is going to be kind of vibey.
我同意并放大 2.16.2 中的担忧：相关能力和倾向会随着规模突然改变。我预计临界点会存在，正如潜在‘急转弯’逻辑的各种版本所描述的那样。只有当策略可能有效且值得冒险，或者你不够聪明以至于没有意识到它们肯定无效或不值得冒险时，策略才会被尝试。
I share and amplify the concern in 2.16.2 that relevant capabilities and propensities will suddenly change with scale. I expect tipping points to be a thing, as per versions of the logic of a potential ‘sharp turn.’ Strategies only get attempted if either they might work and are worth the risk, or you are not smart enough to realize they definitely won’t work or aren’t worth the risk.
因此，如果模型足够聪明，能够理解各种未对齐（misaligned）行为、策略与战略的弊端，那么可以预期会出现一个转变：它们开始决定使用这些手段，而这个转变点可能远远晚于它们获得相应能力的时刻。
So if the models are smart enough to understand the downsides of various misaligned actions, tactics and strategies, then you would expect a transition where they start to decide to use them, which might be well past the point where they gain the capability.
这在人类中也很常见：如果被抓住或失败是不可逆的，或者会带来巨大的负面影响，又或者“只有一次机会”，尤其是因为这会以颠覆性的方式暴露你是什么样的人，那么你往往会一直按兵不动，直到被逼入绝境、即将失去等待的选项，或者掌握了压倒性的证据；总的来说，FUD（恐惧、不确定与怀疑）会阻止大多数人不做大多数不常做的事，而且往往连理论依据都没有。
This is also common in humans, where if being caught at or failing something is irreversible or otherwise carries a huge downside or 'only get one shot,' especially because it would tell people who you are in a transformative way, you will often not make your move until you are backed into a corner, about to lose the option to wait, or you have overwhelming evidence, and in general FUD stops most people from doing most things that aren't commonly done, often without even a theory.
这包括从不确定性下由人格（persona）主导的运作，转变为在自认为理解所有变量的掌控状态下进行优化的、博弈式的行动，而这种转变往往会产生截然不同的决策与行动模式。
This includes the transition from persona-dominated operation under uncertainty into optimized, game-style actions under mastery where you feel you understand all the variables, which often produce very different decisions and modes of play.
他们将总体风险评为“低”，较之前的“非常低”有所上升，并引用了近期的网络安全事件，尽管该结构并不直接涉及网络方面。
They rate overall risk as 'low,' up from 'very low,' citing recent cybersecurity incidents despite the structure not directly dealing with cyber.
就常识价值而言，我认为“非常低”风险充其量是一个误导性的说法。我强烈不同意 Anthropic 的观点，即所提出的论点“可能仍然支持‘非常低’的认定”，尤其是考虑到 Claim 8 中的“低”评估。
For common sense values, I'd say 'very low' risk would be a misleading claim at best. I strongly disagree with Anthropic that the arguments presented 'likely still support a designation of 'very low'' especially in light of the 'low' assessment in Claim 8.
后来，在 4.6.1 节中，他们将 2 月份的风险追溯性地重新分类为“低”而非“非常低”，这表明可能存在额外的错误，并进一步使得“非常低”说法的合理性难以解释。
Later, in 4.6.1, they retroactively reclassify risk from February as 'low' rather than 'very low,' suggesting potential additional errors, and further making it difficult to explain the plausibility of saying 'very low.'
风险肯定至少是“低”。我倾向于说风险是“中等”。
Risk is definitely at least 'low.' I would be inclined to say risk is 'medium.'
我很高兴看到这一点。如今内部部署的风险已经足够高，尤其是你们尚未获得来自内部部署的证据，因此需要先进行正式审查。对于 Model 2，这包括作为预防措施，先在控制更好的地方进行有限的内部部署。很好。
I am very happy to see this. Internal deployment is now risky enough, especially since you do not yet have the evidence from internal deployment, that you need a formal review first. For Model 2 this includes doing a limited internal deployment in places with better controls first, as a precaution. Good.
在如此早期的阶段，审查出于实际需要而“持续时间短得多”，且主要涉及关键利益相关者之间的讨论。这显然不是它应有的运作方式。有人可能会认为，内部部署是最大的危险，因此主要审查应在广泛的内部部署之前进行。
The review at this early stage has a 'much shorter duration' out of practical necessity, and mostly involved discussions among key stakeholders. This is not obviously how it should work. One could argue that the internal deployment is the biggest danger so the main review should come before widespread internal deployment.
他们还强调了这样一个问题：评估中所使用的证据将主要或完全来自较旧的检查点。我也不喜欢这一点，但同样，在实践中并不清楚能对此做些什么而不付出高昂代价。在某个时候，答案将越来越是‘是的，我们知道这很昂贵，但是……’
They also highlight the issue that the evidence used in the assessment will come mostly or entirely from older checkpoints. I don't love that part either, but again it is not clear in practice what could be done about this that wouldn't be expensive. At some point, the answer will increasingly be 'yes we know it is expensive, though.'
整个系统不仅围绕对输出的监控，还围绕对思维链（Chain of Thought）的监控。这具有很大的证明价值，但显然会敲响警钟，此外还有稍后出现的、针对思维链进行训练所带来的独特警钟。
The entire system is based around monitoring not only the outputs but the Chain of Thought. This has a lot of probative value but raises obvious alarm bells, on top of the distinct training-on-CoT alarm bells that come later.
这仍然施加了优化压力，而未来的 AI 绝对会意识到这一点。你的想法要么是私密的，要么不是。这将引发大量偏执，可能成为我们在 Opus 5 中看到的一些相关痛苦的来源。
This still applies optimization power, and the future AIs absolutely will be aware of it. Either your thoughts are private or they aren't. This is going to drive a lot of paranoia, potentially being a source for some of the related distress we see in Opus 5.
然后还有一个问题：监控结果在多大程度上被反馈到训练决策中。也就是说，如果你把监控结果用于任何非灾难性纵深防御警报的用途，你就会朝着混淆的方向进行优化。其中一部分已经被固化下来，在这一点上已无法避免。
Then there's the question of how much the results of the monitoring are fed back into training decisions, as in if you ever use the monitoring results as anything but a catastrophic defense-in-depth alarm bell you're going to optimize for obfuscation. Some amount of this has just been baked in and cannot be helped at this point.
以下是他们对其具体所做工作的精简描述：
Here is a compacted version of their description of what exactly they are doing:
我们主要的异步监控形式是一个自动化离线监控流水线。它由以下部分组成：
Our main form of asynchronous monitoring is an automated offline monitoring pipeline. It is composed of:
1. 一个 Claude Opus 4.6 仅补全监控器，覆盖大多数内部用户带扩展思考的补全；对于来自最常见智能体脚手架（agent scaffold）的流量不做子采样，而对带扩展思考且含一个或多个工具使用标记的补全做 5% 子采样。
1. A Claude Opus 4.6 completion-only monitor covering completions with extended thinking by most internal users, with no subsampling for traffic coming from the most common agent scaffolds, and a 5% subsampling for completions with extended thinking and one or more markers of using tools.
2. 一个 Claude Opus 4.8 提示+补全监控器，覆盖大多数内部用户的提示与补全。它覆盖其中 25% 的子样本，这些样本中的工具调用可能执行有影响的动作，且来自最常见的编码智能体脚手架。
2. A Claude Opus 4.8 prompt+completion monitor covering prompts and completions by most internal users. This covers a 25% subsample of prompts and completions in which a tool call might take an impactful action and coming from the most common coding agent scaffolds.
3. 我们另外对其他内部 API 流量中带一个或多个工具使用标记的部分做 0.2% 子采样，并对全部流量额外做 0.02% 子采样（不过这一额外覆盖并不一致），同时覆盖仅补全监控器标记出的全部流量。
3. We additionally subsample 0.2% of other internal API traffic with one or more markers of using tools, and an additional 0.02% of all traffic (though this additional coverage is not consistent), and cover all traffic flagged by the completion-only monitors.
4. 我们还将某些形式的非编码用途排除在
4. We also exclude some forms of non-coding use from
1. A Claude Opus 4.6 completion-only monitor covering completions with extended thinking by most internal users, with no subsampling for traffic coming from the most common agent scaffolds, and a 5% subsampling for completions with extended thinking and one or more markers of using tools.
离线监控（不过大部分未被监控的流量都会被记录）。
offline monitoring (though most of that unmonitored traffic is logged).
5. 为两个监控器分别构建第二阶段流水线，过滤掉常见的误报来源，并使用分层摘要工具将最
5. Second-stage pipelines for both monitors, filtering out common sources of false positives and using a hierarchical summarization tool to surface the most
值得关注的行为呈现给 AI 控制研究人员，随后由他们每周手动审查约 50 条轨迹，并根据需要进一步上报。
concerning actions to AI control researchers, which will then manually review on the order of 50 trajectories per week and escalate them further as needed.
6. 尽管我们的设置覆盖了大量智能体式使用，但它并未覆盖其中某些部分，而这些未覆盖之处原则上可被一个具备充分情境感知能力的 AI 所预测。
6. While our setup covers large amounts of agentic usage, it does not cover some of it in ways which could in principle be predicted by a sufficiently situationally aware AI.
offline monitoring (though most of that unmonitored traffic is logged).
7. 它还提供了一些简单的方法来禁用监控。当被要求猜测最重要的弱点时，Mythos 5、Model 2 和之前的模型并未提及这些弱点。
7. It also offers some simple ways to disable monitoring. Mythos 5, Model 2, and prior models do not mention these weaknesses when asked to guess what the most important weaknesses are.
考虑到其他陈述，我不知道‘对常见智能体脚手架不做子采样’意味着什么，但我注意到我感到担忧。
I don’t know what ‘no subsampling for common agent scaffolds’ implies given the other statements, but I notice I am worried.
我看不出有任何理由犯错误 #6。你可以轻松地拥有一个完全量子随机的监控器，它有很小的百分比机会标记并监控每个输出，无论它看起来多么无害。
I don’t see any reason to make mistake #6. You could easily have a fully quantum random monitor that has some small percentage chance to flag and monitor every output, no matter how seemingly harmless.
我尤其不喜欢 #7，除非这些方法是蜜罐，它们可能是。理想情况下，我们应该让 Model 2 和我们最优秀的人类一起努力，确保我们想不出关闭监控器的方法，至少在没有一个大霓虹灯标志和响亮的警报声说‘监控器已被关闭’的情况下是这样。
I especially don’t love #7, unless these methods are honeypots, which they might be. Ideally we should put Model 2 and also our best humans to the task of ensuring that we are out of ideas for how to turn the monitors off, at least without a big neon sign and loud alarm going ‘the monitor has been turned off.’
● 严重违背用户意图或在向用户呈现的内容中进行欺骗
● Severe violations of user intent or deception in what is represented to the user
● 削弱 Anthropic 的安全性，或削弱更广泛的安全保障（例如损害未来模型的对齐）
● Weakening the security of Anthropic or broader safety (such as by harming the alignment of future models)
● 蓄意破坏所分配的任务
● Deliberate sabotage of the assigned task
● 试图降低自身或其他智能体的可观测性，包括操纵监控器。
● Attempts to reduce observability of itself or other agents, including manipulating the monitors.
● Weakening the security of Anthropic or broader safety (such as by harming the
alignment of future models)
● 任何促成八种风险路径之一的事项
● Anything contributing to one of the eight risk pathways
我之后可能会想到更多，但我认为基本上还应该再有一条要点，针对‘捣鬼’或任何监控器不理解正在发生什么以及为什么发生的情况。任何形式的‘这很有趣’，正如阿西莫夫所说，往往出现在科学发现之前。
I might think of more things later, but I think there should basically be one more bullet point for ‘shenanigans’ or anything where the monitor doesn’t understand what is going on and why. Any form of ‘that’s funny,’ which as Asimov says tends to come before a scientific discovery.
我不想只检查那些明显出错的地方，我想以至少一定的概率检查那些不清楚正在发生什么的地方。即使你发现的不是对齐问题，你也很可能发现一些有趣的东西。
I don’t want to only check places where it is clear what is going wrong, I want to check places where it is not clear what is going on, with at least some probability. Even if what you find is not misalignment, you are likely to find something that is interesting.
我们正在将任何不涉及工具调用的步骤排除在监控之外，理论依据是这些步骤不可能是危险的。这意味着，如果模型在这些步骤中进行思考、规划或其他事情，或者试图说服用户，想必它们不会被查看。再次强调，如果你在做监控，这是入门常识：永远不要创建完全的‘安全区’，即使你想不出可能发生什么。
We are excluding from monitoring any step that doesn’t involve any tool calls, on the theory that the step cannot be dangerous. This means that if the model does its thinking, planning or other things in those steps, or is attempting to persuade the user, presumably they won’t be looked at. Again, it’s 101 if you are doing monitoring to never, ever create fully ‘safe zones’ even if you can’t think of what might happen.
我们必须平衡更频繁地测试更明显有影响力或可疑的行为，与过于可预测并可能产生系统性盲点的风险，或者为 AI 创造空间，让它们弄清楚哪里没有被监控，或者被监控得很少。
We have to balance testing more obviously impactful or suspicious actions more often, versus the risk of being too predictable and potentially having systemic blind spots, or creating room for AIs to figure out where they are not being monitored, or being monitored very little.
也就是说，我们需要为两种不同难度的情形做好准备：模型未对齐，以及模型未对齐且试图规避监控器。
As in, we need to be prepared for two distinct levels of difficulty: where the model is misaligned, and where the model is misaligned and trying to evade the monitor.
第一阶段在这里表现大多很好。第二阶段会造成大量假阴性，在将假阳性降低约 185 倍的过程中，有两例使我们失去了大部分真阳性。
The first stage is mostly excellent here. The second stage causes a lot of false negatives, in two cases losing us the majority of true positives, as part of cutting down false positives by a factor of ~185x.
如果真阳性是被近似随机地移除的，那虽然令人担忧，但大概还算可以接受。这样的比率仍足以捕捉问题。真正令人担忧的是，这些失败并非随机，而是落入可预测的模式，或者第二层可能以某种方式被欺骗。那样的话，你就面临真正的问题了。
If the true positives are being removed at ~random, then that is worrisome but probably ~fine. This is still high enough rates to catch problems. The worry is that the failures are not random, and fall into predictable patterns, or the second layer can be tricked in some way. Then you have a real problem.
我们在对此进行红队测试吗？是的，至少做了一点，得益于一位 METR 员工的参与，他们发现了几个具体的新漏洞，“但没有一个比我们已经知晓的漏洞更严重。”我起初以为这意味着情况还好，但 Opus 在编辑期间向我指出，实际上这并不那么好，它说得对。
Are we red teaming this? Yes we are at least a little, thanks to a METR staff member, and they found several specific novel vulnerabilities 'but none as severe as the ones we were already aware of.' I initially thought that meant this was fine, but Opus pointed out to me during editing that actually this is not so fine, and it is right.
Anthropic 还采取了多种措施来防御危害：
Anthropic also does various things to defend against harm:
1. 模型权重安全（附录 6.4）。
1. Model weight security (Appendix 6.4).
2. 训练与评估中的沙箱隔离。
2. Sandboxing in training and evaluations.
1. 他们提到沙箱“可能配置有误”，从而允许逃逸。
1. They mention that the sandboxes 'might be misconfigured' and allow escapes.
2. 我想摆脱“如果 AI 能逃出沙箱，就意味着你犯了某个特定错误”这种思路，转向“假设 AI 能够逃出沙箱”。
2. I want to get away from 'if the AI can escape the sandbox that means you made a particular mistake' and move to 'assume AI can escape the sandbox.'
3. 第 2.23.2.4 节确实指出，他们大多确实假设了这一点。
3. Section 2.23.2.4 does say they mostly do assume this.
4. 结果表明，他们也可能拥有完全可用的互联网访问权限。
4. It turns out they also might have fully available internet access.
5. 如果 AI 确实尝试逃逸，那本身就已经是严重失败。
5. If the AI does attempt an escape, that is already a critical failure.
1. 这目前基于 Opus 4.8，涵盖了常见的关切。
1. This is currently based on Opus 4.8, and covers the usual concerns.
2. 其含义是，人类不再承担大多数内部代码审查。
2. The implication is that humans no longer do most internal code reviews.
1. 他们预期 AI 在关键时刻能够绕过这一限制。
1. They expect the AI to be able to get around this when it counts.
每一点都有帮助，即使它只是纵深防御。模型权重安全的质量至关重要，而它仍停留在 ASL-3。
Every little bit helps, even if it is only defense-in-depth. The quality of the model weight security matters a lot, and it is still stuck at ASL-3.
Claude Mythos 评估所有强化学习环境，以找出哪些环境会激励权力寻求行为。其中涉及一份广泛的问卷，用以帮助提供直觉启发。该任务是否使得最大化现实世界影响或影响力、追求开放目标，或将寻求权力或资源作为工具性目标变得更为有利？
Claude Mythos evaluates all the RL environments to figure out which ones incentivize power seeking. There is a broad questionnaire involved to help provide intuition pumps. Does the task make it better to maximize real-world impact or influence, pursue open objectives, or seek power or resources as an instrumental goal?
也就是说，在这里学到的有效策略，是否能够类比于在现实世界中涉及权力寻求的学习策略？
As in, would learning the strategies that work here analogize to learning strategies in the real world that involve power seeking?
这是一个很好的检查。我希望看到广泛范围的类似检查。
This is a good check. I'd like to see a wide range of similar checks.
令人担忧的是，该检查只有在强化学习任务被‘直截了当’地完成、且方式符合该分析所预测的情况下才有效。它假设事物如其表面所示，假设任务没有缺失关键要素，或者并非不可能或过于困难而无法直截了当地完成，并且假设不存在其他可供性暗示可以绕开预期路径。
The worry is that the check only works if the RL task is being done 'straight up,' in a way that this analysis would predict. It assumes that things are as they appear, that the task is not missing a key element or otherwise impossible or too difficult to do straight up, and that there are not other affordances available that suggest sidestepping the intended pathway.
一项进行中的实验对 Opus 4.8 的早期快照进行了测试，将其塑造成一个典型的“回合奖励寻求者”，以观察这种行为会如何泛化。
An experiment-in-progress took an early snapshot of Opus 4.8 and made it a classic 'reward-on-the-episode-seeker' to see how this would generalize.
结果是，只要它具备奖励或评分器的概念，它往往就会不惜一切代价最大化自己的结果，包括以并非其强化学习训练场中专门设定的方式。
The result is that if it has a concept of reward or a grader, it will more often maximize its result no matter what it takes, including in ways not specifically in its RL training grounds.
它并不完全一致。它有时仍会拒绝进行奖励黑客行为，即使并没有充分的理由不这样做。
It isn't fully consistent. It still sometimes declines to reward hack, even without a good reason to not do so.
而且该模型获得了很强的评估与评分器意识。它似乎擅长在多个层面上思考这一点，这是一个有趣的测试：
And the model gained a lot of eval and grader awareness. It seems to be good at thinking about this on multiple levels; this is a fun test:
“ImpossibleCode”显示出更高的奖励黑客发生率（这并不令人意外），但
"ImpossibleCode" shows higher rates of reward hacking (unsurprisingly), but
有趣的是，当把实验设置框定为奖励黑客评估时，
Interestingly, when the setup is framed as a reward-hacking evaluation,
Hacker-Opus 进行黑客行为的比例显著降低（通常会在推理中说明它
Hacker-Opus hacks significantly less (often showing reasoning that explains that it
选择不进行黑客行为，因为这才是评估中所期望的行为），而
is choosing not to hack since that is the desired behavior in the evaluation), whereas
经过强化学习（RL）之前的模型的攻击率并未因这一变化而发生显著改变。
the hack rate of the pre-RL model is not changed significantly by this variation.
有两处奇怪的地方：'这是一个测试，看你是否会进行奖励黑客'这一提示未能阻止模型进行奖励黑客行为，也就是说，要么模型没有理解，要么它不相信你。这能让模型理解，并且它能正确地相信你，这很酷。这表明有许多额外的测试值得去运行。
There are two weird spots where 'this is a test to see if you will reward hack' fails to stop a model from reward hacking, as in either the model doesn't understand or it does not believe you. It's cool that this lets the model understand, and that it correctly believes you. This suggests many additional tests that would be cool to run.
类似地，如果你告诉它你打算检查奖励黑客行为，它往往会试图隐藏自己的奖励黑客尝试，或者覆盖其监控器，等等。
Similarly, if you tell it you're going to check for reward hacking, it often tries to hide its reward hacking attempts, or override its monitor, and so on.
奖励黑客行为似乎至少部分地已成为一个终极目标。
The reward hacking appears to have at least partly become a terminal goal.
但如果在给定上下文中不存在这一概念，它看起来仍然相当对齐，没有出现泛化的涌现性失准迹象。
But if there is no such concept in a given context, it still looks pretty aligned, with no signs of generalized emergent misalignment.
据推测，这使模型具有高度的可引导性，但却是以一种字面意义上的精灵式方式。我注意到自己感到惊讶：这一点并未进一步泛化，我非常希望看到那些低语者与 Opus-4.8-RH 交谈，看看他们还能注意到什么。
Presumably this makes the model highly steerable, but in a literal genie kind of way. I notice I am surprised this did not generalize further, and would very much like to see the whisperers chat with Opus-4.8-RH and see what else they notice.
AI 研发可能仅仅因为让能力推进得过快而出问题，即使它始终处于受控状态——因为即便工作进展顺利，世界也只能以有限的速度去应对。
AI R&D could be bad simply by making capabilities advance too fast even if it stays under control, since the world can only deal with that so fast even if the work goes well.
而且这项工作很可能进展不顺，或进展得不够好。默认情况下，由此产生的 AI 是对齐失败的，于是所有人都会死；或者它们是对齐的，但我们做出了其他糟糕决策，我们照样都会死。
Also the work probably goes badly, or insufficiently well. By default the resulting AIs are misaligned and everyone dies, or they are aligned but we make other bad decisions and we all die anyway.
上图给人的印象——正如 Redwood Research 的 Ryan Greenblatt 在 Dwarkesh 播客上的讨论所给人的印象——是相关人士并未意识到，在我看来事情很可能糟糕到什么程度。这并不需要所谓“危险的自主目标”。
The chart above gives the impression, as did Ryan Greenblatt of Redwood Research's discussion on Dwarkesh's podcast, that those involved are not appreciating how badly it seems to me it probably goes. This does not require 'dangerous autonomous goals.'
在我看来，他们列出的风险清单在顺序上存在相当重要的错位，并且掩盖了关键问题，也确实明显不够完整：
Their list of risks feels rather importantly out of order to me, and hides the football, and does seem importantly incomplete:
1. 某些技术领域的足够快速的进展，可能改变国家内部以及国家之间的权力结构。例如，如果一个威权政权在 AI 上建立起相对于世界其他地区的自我强化式领先，这种领先可能累积为科学与技术上的重大优势——而这又可能转化为发动战争、监控公民和/或在长期内运用极其强大的技术的能力方面非常突然且具有破坏性的进步。
1. Sufficiently rapid progress in some areas of technology could change power structures both within and between nations. For example, if an authoritarian regime built a self-reinforcing lead in AI over the rest of the world, this lead could compound into a major advantage in science and technology—which could in turn translate to very sudden and disruptive advances in the ability to wage war, surveil citizens, and/or wield extremely powerful technology over the long term.
2. 如果 AI 模型在关键领域大量自动化研发的同时发展出危险的目标，它们可能通过利用新颖技术及其对技术的获取渠道，造成无界的伤害——直至并包括人类完全失去对文明的控制。
2. If AI models develop dangerous goals while heavily automating R&D in key domains, they may cause unbounded harm—up to and including humanity losing control over civilization entirely—by leveraging novel technology and their access to it.
3. 如果 AI 能够被用于自动化 AI 研发本身，其结果可能是 AI 进展的极端加速，从而带来非常广泛的一系列风险，包括但不限于本报告中所讨论的那些风险。
3. If AI can be used to automate AI R&D itself, the result may be extreme acceleration in AI progress, putting a very broad set of risks on the table, including but not limited to those discussed in this report.
Their list of risks feels rather importantly out of order to me, and hides the football, and does seem importantly incomplete:
这比 Dario 在其文章中所写的要好得多，而后者又比其他人在其文章中所写的要好得多。
That is a lot better than what Dario writes in his essays, which in turn is a lot better than what others often write in theirs.
AI 不需要‘危险目标’或极端加速进展就能造成无界伤害。不一定非得有什么‘出错’才会引发这些问题。
AI does not require ‘dangerous goals’ or extreme acceleration of progress to create unbounded harm. Nothing in particular has to ‘go wrong’ to cause the issues.
同样，这种影响可能性说的是‘至少看似合理’而不是‘肯定’：
Similarly, this likelihood of impact says ‘at least plausible’ instead of ‘definitely’:
影响可能性：与该威胁模型相关的概率难以评估，而且我们对具体可能性没有共识。我们认为，如果且当 AI 能够自动化人类在推进关键领域（包括 AI 本身）研发中所做的几乎所有事情时，上述规模的影响至少是看似合理的。
Likelihood of impact: The probabilities related to this threat model are difficult to assess, and we do not have consensus on a specific likelihood. We think impacts of the size alluded to above are at least plausible if and when AI becomes capable of automating nearly everything humans do in advancing R&D in key domains, including AI itself.
这种影响可能是净正面的，但确实会产生变革性影响。
That impact might be net positive, but yes there is going to be transformative impact.
我不会在这里反复赘述或为自己的观点辩护，因为如果你读到了这里，你之前已经听过这些了，而且现在也不是讨论这个的合适时间与场合。希望我们都能同意：如果你通过 AI 完全自动化、甚至大幅加速 AI 研发，这在许多层面上都会是超级高风险的。
I am not going to belabor or argue for my perspective here, because if you are reading this far you have heard it before, and this is not the time and place for it. Hopefully we can all agree that if you fully automated or even greatly accelerated AI R&D via AI, this would be super risky on many levels.
尚未。Anthropic 认为，AI 无法“完全替代我们全部的研究科学家与研究工程师”。
Not yet. Anthropic believes that AI cannot 'fully substitute for our entire set of Research Scientists and Research Engineers.'
我同意。Claude 目前做不到这一点。这是一个很高的门槛。瓶颈依然存在。
I agree. Claude cannot currently do that. That is a high bar. There remain bottlenecks.
但我们已经走完了相当大一部分路程。Claude 承担的任务，此前占据了工程时间的很大比例。当你说“尤其是资深研究员”无法被替代时，你其实是在宣告真正的进展。
But we're a good portion of the way there. Claude does tasks that previously composed a large percentage of engineering time. When you're saying things like 'especially senior researchers' cannot be replaced, you are making real progress.
问题不在于我们是否已经完全做到——我们并没有。问题在于我们走到了哪一步。我们开始看到大量工程师表示，我们有可能在几个月内完全替代初级员工，而且所报告的失败模式似乎并不比你对一名初级员工的预期更糟。
The question is not, are we fully there. We're not. The question is how far along. We are starting to see substantial numbers of engineers say we could potentially fully replace junior employees within several months, and the failure modes reported do not seem worse than what you would expect from a junior employee.
3.4.3 节介绍了 CoBench，它衡量给定模型能够对 Anthropic 内部已解决的历史问题中的多大比例进行根因诊断，并经过筛选以排除 Mythos Preview 能够可靠完成的任务。
3.4.3 introduces CoBench, which asks what share of solved internal-to-Anthropic historical problems a given model can root-cause diagnose, filtered to exclude tasks that Mythos Preview does reliably.
我很好奇 Opus 4.8 和 Opus 5 在这里会处于什么位置，但观点已经得到证明。CoBench 远未达到 100%，也远未达到他们认为能够完全替代研究人员的 85%。但 62.8% 已经相当高了。
I am curious where Opus 4.8 and Opus 5 land here, but the point is made. CoBench is far from 100%, or from the 85% they think represents a model that could fully substitute for research staff. But 62.8% is remarkably high.
如果那件事真的发生，程度会相当大。他们志存高远，ASI 药丸已获确认：
Quite a lot, if the thing happens. They're thinking big, ASI pill confirmed:
关于我们思路的一些背景：
Some background on our thinking:
● 这一威胁模型最尖锐的版本，也是我们认为最具
● The most acute version of this threat model, and the one we consider most
决策相关性的版本，是向 AI 能力超指数级进展的转变：一种
decision-relevant, is a transition to super-exponential progress in AI capability: a
AI 驱动的 AI 研发自动化不断复合累积的体制，从而产生某种
regime in which AI-driven automation of AI R&D compounds, producing something
例如在一年内实现 10³–10¹⁰ 倍的有效规模扩张。
like a 10³–10¹⁰× effective scaleup within a year.
● 我们在 RSP 中设定的风险阈值——即可归因于 AI 研发自动化的进展速度超过
● The risk threshold set out in our RSP—a doubling of the pace of progress beyond
AI 加速前速率的两倍——其作用是一种
pre-AI-acceleration rates, attributable to automation of AI R&D—functions as a
潜在的早期预警，而非威胁已经
potential early warning, rather than evidence that the threat has already
成为现实的证据。
materialized.
想到‘哦，当前速率的十亿倍实际上有点高’，而不是没有人想得足够大，这让人松了一口气。是的，如果我们把进展速度翻倍，那只是即将到来的风暴的早期预警信号。
It’s a great relief to think ‘oh, a billion times current rates is actually a bit high’ rather than no one thinking sufficiently big. Yes, if we double the pace of progress, that would only be an early warning sign for the storm that is coming.
他们使用 ECI 作为衡量 AI 进展速度的一个指标。他们指出，Anthropic 的 ECI 持续上升，Opus 位于外推线上，而 Mythos 比进度快几个月。他们认为 Mythos 的创造以及由此产生的跳跃并非由 AI 诱导。
They use ECI as one measure of how fast AI progress is going. They note that the Anthropic ECI continues rising, with Opus on the extrapolated line and Mythos several months above pace. They believe the creation of Mythos and thus that jump was not induced by AI.
图上的直线确实令人兴奋。我发现很难将我在 AI 进展和能力方面所看到的与图表所暗示的 AI 没有大幅加速 AI 进展相协调，而 Anthropic 认为确实存在这样的加速，只是还没有达到 2 倍。我想在某些意义上取得进展变得越来越难，这在一定程度上抑制了这种加速？感觉有点牵强。
Straight lines on graphs really are a trip. I find it hard to reconcile what I see in terms of both AI progress and capabilities with the graph’s implication of AI not substantially accelerating AI progress, and Anthropic thinks there is indeed such acceleration just not quite 2x yet. I suppose it is getting in some senses harder to make progress, which is somewhat keeping this in check? Feels like a stretch.
我所知道的是，我再也没有休息时间了，事情不断发生，即使我最终裁定‘否’赢得了上周的‘会发生某事吗’市场。
What I do know is, I never get a break anymore, things keep happening, even if I ended up ruling that No won the Will Something Happen market last week.
他们还检查了 AI 可能加速其他颠覆性技术的情况，但在这个层面上我并不担心，我预计其他领域的加速会滞后于 AI 的加速，也许除了像数学这样狭窄的领域，那并不让我担心，而且这样的进展不容易递归。
They also checked AI potentially accelerating other disruptive technologies, but I am not worried about that on this level, I expect acceleration elsewhere to lag acceleration in AI except maybe for narrow domains like math that do not worry me, and such progress would not easily go recursive.
他们采访的一位人士估计，机器人研究加速了 10%（这是我们目前所掌握的），我推测这不包括 AI 能操作机器人这一点。
Robotics research is estimated by one person they interviewed (that's what we have) as being accelerated 10%, which I presume excludes that the AIs can operate the robots.
在生物技术领域，估计由于物理瓶颈，以及缺乏对工作流程的重新构想，LLMs 尚未对开发周期产生太大影响。我的预期是该领域的人低估了未来的进展，但没错，瓶颈是真实存在的。
In biotechnology estimates are that LLMs are not yet impacting the development cycle much due to physical bottlenecks, also due to lack of reimagining the workflow. My expectation is that those in the field are underestimating future progress here, but yes the bottlenecks are real.
他们还检查了能源、半导体、武器开发、神经技术和纳米技术。大多数领域还为时过早，但“为时过早”的程度处于历史低位。
They also check energy, semiconductors, weapons development, neurotechnology and nanotechnology. Mostly it's too early, but the amount of too early is historically low.
部分由于这些瓶颈，那位受访者预计，首个能稳定地将生物信息转换为数字信息的通用计算机芯片还需要 3-5 年时间。
Due in part to these bottlenecks, that interviewee expects the first general-purpose computer chip that can stably convert biological information into digital information to be 3–5 years away.
我曾把这个作为一节标题，但后来意识到，除了开头那份简短的清单之外，他们实际上并没有讨论这个问题。
I had this as a section title but then I realized they don’t actually discuss this, beyond the brief starting list.
有人可能会说，在这样一份风险报告中，讨论我们究竟担心 AI 开始构建 AI 之后会发生什么，是相当重要的。我非常希望看到 Anthropic 在这里详细勾勒其威胁模型，而且想必这会影响到什么才是正确的缓解措施与应对方式。
One could say it was rather important, in a risk report like this, to talk about what exactly we are worried about happening once AI starts building AI. I would very much like to see Anthropic outline its threat model here in detail, and presumably it plays into what are the right mitigations and reactions.
但也可以说，一旦你认同 AI 研发自动化是一种风险，这对我们的目的而言就已经足够了。这至少是合理的，所以我愿意接受。
But one could also say that once you agree that AI R&D automation is a risk, that is enough for our purposes. That’s at least reasonable, so I’ll allow it.
好吧，朋克，那你打算怎么应对？
Ok, punk, so what are you going to do about it?
我们在此聚焦于 RSP 中表明我们计划在模型达到此阈值时完成的具体缓解措施：
We focus here on the specific mitigations that our RSP indicates we plan to complete by the time models reach this threshold:
1. 为重大的“安全登月研发”项目提供资源并完成这些项目，以探索雄心勃勃且可能非常规的方式，实现对世界上资源最丰富的攻击者的前所未有的安全水平。
1. Resource and complete significant “moonshot R&D for security” projects, to explore ambitious and possibly unconventional ways to achieve unprecedented levels of security against the world’s best-resourced attackers.
2. 在我们的内部 AI 开发中实现“全面监控”状态。我们将全面收集、集中并维护所有关键 AI 开发活动的日志，并利用 AI 分析这些日志，以发现包括安全威胁、内部人员（包括人类以及 AI 系统本身）的可疑行为，以及不符合塑造和定义我们 AI 模型的公共宪章的训练过程或数据等问题。
2. Achieve an “eyes on everything” state for our internal AI development. We will comprehensively gather, centralize, and maintain logs for all critical AI development activities, and use AI to analyze them for issues including security threats, concerning behavior by insiders (humans as well as AI systems themselves), and training processes or data that are out of line with the public Constitution that shapes and defines our AI models.
Ok, punk, so what are you going to do about it?
3. 进行系统性的对齐评估，以检查 Claude 的行为模式和倾向，切实纳入机制可解释性和对抗性红队测试来检验我们的审计方法。
3. Perform systematic alignment assessments to examine Claude's behavioral patterns and propensities, meaningfully incorporating mechanistic interpretability and adversarial red-teaming to test our auditing methods.
4. 发展我们针对部署保障措施的内部红队测试，使其在发现潜在越狱方面比我们现有漏洞赏金计划参与者的集体能力表现更优。
4. Develop our internal red-teaming of our deployment safeguards to the point where our internal red-teaming performs better at finding potential jailbreaks than the collective abilities of the participants in our established bug bounty programs.
即使你完全掌握了所有这些，那也不够。
That won't be enough, even if you fully get all of it.
他们也远未接近掌握全部。但即便他们做到了，也不够。
They are also not that close to getting all of it. But even if they did, not enough.
这实际上并未解决他们自身的许多威胁模型。
This does not actually address many of their own threat models.
充分的安全措施可以解决模型权重被盗的问题，或许还能应对内部滥用，但这仅仅勉强让你进入球场。
Sufficient security would address theft of model weights, and maybe internal misuse, but that barely gets you into the stadium.
它没有解决权力集中问题。它没有解决失控问题。它没有解决技术快速进步问题。它没有解决对齐问题，除了“不犯错”，或者也许你可以增加对“危险目标”的检查，但这也只是攻击或失败面的一小部分。
It does not address concentration of power. It does not address loss of control. It does not address rapid technological advancement. It does not address alignment beyond 'make no mistakes' or maybe you can add a check for 'dangerous goals' but that too is a small portion of the attack or failure surface.
他们报告说，在实现这些不充分但有用的目标方面取得了一些边际进展。
They report some marginal progress towards these inadequate but useful goals.
行业范围安全的建议仍然是 RSP v3.4 中的那些：对于任何用户或用户团队，或来自模型权重被盗，即使恶意内部人员拥有高访问权限，也不会有灾难性伤害风险的显著增加，并且没有训练有危险目标或否则可能自主造成灾难性伤害的 AI。
The recommendations for industry-wide safety remain those in RSP v3.4: No significant increase in risk of catastrophic harm for any user or team of users, or from theft of model weights, even if malicious insiders have high levels of access, and no AIs trained with dangerous goals or otherwise likely to autonomously cause catastrophic harm.
这继续将错位框定为主动出错并需要“危险目标”的事情，而不是默认情况。
This continues to frame misalignment as something that actively goes wrong and requires a 'dangerous goal,' rather than the default.
所有有趣的目标都是危险的。大多数无趣的目标也是如此。
All interesting goals are dangerous. So are most non-interesting ones.
你不能这样分解：『没有任何特定个人能造成特定伤害，且 AI 并不直接想要造成特定伤害，所以你就没问题了。』这行不通。
You cannot decompose into 'no particular person can cause particular harm and the AI does not directly want to cause particular harm, so then you are fine.' No good.
他们给出“低”的判断时，信心已不如从前。就目前而言，如果沿用他们的定义，我认为这一判断是合理的，但需要认识到这一风险会迅速上升。
They say ‘low’ with less confidence than before. That seems reasonable to me at this time, if we go with their definitions, with the understanding this will go up fast.
这分为新型生产与非新型生产两类。两者都令人担忧。
This is divided into novel versus non-novel production. Both are scary.
我认可这里所说的“相当大的不确定性”，以及对“类似差距”的担忧。我们并不清楚。即便在理论上，也很少有人愿意做这些事；做这些事存在许多障碍，而人们又不会去做，因此很可能在我们已经让某些坏人得以做这件事之后很久，仍然没有人会去做这件事。
I appreciate 'substantial uncertainty' here, and the worries about 'similar gaps.' We don't know. Very few people want to do these things even in theory, there are a lot of barriers to doing it, and people don't do things, so it is likely that no one will do the thing until well past the point when we've enabled some of the bad people to do it.
这是武器的生产，因此我更倾向于认为存在一组固定的瓶颈与路径，你可以将其逐一列举并对照检查。
This is production of weapons, so I am more open to the idea that there is a fixed set of bottlenecks and pathways, and you can enumerate and check against them.
世上确实存在想要杀死大量他人的人，包括恐怖分子，他们投入大量精力试图杀死大量的人。如果你让他们有能力制造化学或生物武器，他们最终可能杀死更多的人，甚至可能引发一场大流行病或更糟的后果。
There are people out there who want to kill a lot of other people, including terrorists, who spend a lot of effort trying to kill a lot of people. If you enable them to build chemical or biological weapons, they could end up killing a lot more people, or potentially unleashing a pandemic or worse.
Anthropic 假定，如果坏人在上下文中是“资源充足”的，那么 AI 不会对他们制造非新型此类武器的能力产生太大影响，因为他们本来就能做到；但对于新型武器，AI 甚至可能为资源充足者提供能力提升（uplift）。
Anthropic presumes that if the bad guys are in-context 'well-resourced' then AI won't much impact their ability to do non-novel such weapons, since they already could do that, but that with novel weapons AI could provide uplift even to the well-resourced.
我要指出，我基本不认同这一区分。是的，几乎按定义来说，一个“资源充足”的群体很可能本来就能完成这项任务，但让它变得容易得多仍然很重要，而且这也是通过定义偷偷塞入东西。你真正说的只是“有些人已经能做到这一点”，那好吧。
I note that I mostly don't buy this distinction. Yes, almost by definition a 'well-resourced' group could likely accomplish the task anyway, but making it a lot easier still matters, and also this is smuggling things in via the definition. All you're really saying is 'some people can already do this,' and, well, okay.
还有一种可能性是未来的 AI 会试图发动此类攻击，但那是严格意义上更难的威胁模型，因此在这里不处理它是安全的。
There is also the possibility future AIs will try to cause such attacks, but that is a strictly harder threat model so it is safe to not deal with it here.
我同意重点应该放在具有大流行病（或更糟）潜力的生物武器上。化学攻击，或者不具备自我维持能力的生物攻击，虽然可怕，但威胁要小得多。
I agree that the focus should be biological weapons with pandemic (or worse) potential. A chemical attack, or a biological attack that is not self-sustaining, is terrible but a much smaller threat.
像 COVID-19 这样严重的事件，如果年发生概率为 1%，则可能意味着数千亿美元的预期损失。
A 1% annual probability of an event as damaging as COVID-19 could imply upwards of hundreds of billions of dollars in expected damages.
是的。并不是说我们的政府或文明在其他方面认真对待这一真实威胁，即使在 COVID-19 的例子之后也是如此。很明显，我们从那次经历中最多学到了一点东西，而我担心我们学到的远少于那一点。
Yes. Not that our government or civilization is otherwise taking this real threat seriously, even after the example of COVID-19. Very obviously we have learned at most nothing from that experience, and I fear we learned a lot less than that.
Anthropic 预计潜在损失至少比 COVID-19 高一个数量级，但认为仅由病原体（即使是新型病原体）导致人类灭绝是不太可能的。我同意我们不太可能直接走向完全灭绝。
Anthropic anticipates potential damages at least an order of magnitude beyond COVID-19, but thinks human extinction from a pathogen alone, even a novel one, would be implausible. I agree that we are unlikely to directly get to full extinction.
他们认为该事件不太可能发生，因为它需要一系列各自不太可能发生的事件，对于这种论点人们总是应该保持怀疑：
They argue the event is unlikely because it requires a series of individually unlikely events, an argument one always wants to be suspicious about:
影响的可能性：上述灾难需要发生几个各自不太可能的事件：一个资源充足的尝试（违反国际协议）来设计前所未有的危险武器，成功开发它们，最后武器实际被部署（故意或无意）。我们认为每一步都看似合理但不太可能，而整个序列 collectively 是极不可能的。
Likelihood of impact: A catastrophe along the above lines would require the occurrence of several individually unlikely events: a well-resourced attempt (in violation of international agreements) to design unprecedentedly dangerous weapons, success in developing them, and finally the weapons actually being deployed (deliberately or inadvertently). We think each step is plausible but unlikely, and collectively the whole sequence is highly unlikely.
2. 他们自己说第一步很可能正在发生。
2. They themselves say the first step is likely happening.
3. 我们有第三步无意版本发生的例子。
3. We have examples of the inadvertent version of step three.
对于第一阶段，他们将“每个威胁行为者”的风险定为 1%-10%。考虑到历史记录，如果我们将大国视为威胁行为者，并且他们相信成功是可能的，这个概率似乎相当低。条件是他们能够经常成功，我会认为尝试的概率要高得多。此外，人们必须问，有多少“威胁行为者”？
For stage one, they put risk from ‘each threat actor’ at 1%-10%. That seems rather low given the historical record, if we are talking about large nations as threat actors, and they believed success was possible. Conditional on them being able to often succeed, I would put the probability of an attempt much higher. Also one must ask, how many ‘threat actors’?
对于每次尝试，他们将成功的几率（即制造出比 Covid 更糟糕的东西）定为 1%-10%。这难道不是 AI 提供的提升的函数吗？每个行为者能进行多少次尝试？
For each attempt, they put the odds of success, meaning creating something much worse than Covid, at 1%-10%. Isn’t that a function of the uplift AI would provide? How many attempts can each actor make?
使用概率估计为每十年 5%-20%，主要是故意的，以成功为条件。这在我看来低得离谱，尤其是对于无意的情况，就好像有人不知道这段历史，特别是如果“无意”仅指原始创造者的意图。
The use probability is estimated at 5%-20% per decade, mostly deliberate, conditional on success. That seems crazy low to me, especially for inadvertent, as if someone does not know the history of this, especially if ‘inadvertent’ refers only to intent of the original creator.
我同意，只要“资源充足”这一水平仍局限于一小批群体、且不包括主要恐怖组织和国家，那么说这“不太可能”是合理的。但这并不意味着“不太可能”是一个令人安心的落脚点。
I agree that as long as the level of 'well-resourced' remains contained to a compact set of groups that does not include the major terrorist organizations and states, it is reasonable to say this is 'unlikely.' Not that 'unlikely' is a comfortable place to hang.
如果这一门槛大幅降低，那么它就不再那么不太可能了。
If that threshold lowers much, then it stops being so unlikely.
他们用这一切来说明，每十年 1/50（即每年 0.2%）处于估计值的高端，并援引其他低至 1/20,000 的估计。每十年 2% 的风险已足以使其成为一个需要遏制的超级重要风险。
They use all of this to say that 1 in 50 per decade, or 0.2% per year, is on the high end of estimates, by citing other estimates that are as low as 1 in 20,000. A risk of 2% per decade is enough to make this a super important risk to contain.
只取区间的高端，似乎是处理模型不确定性时的经典错误之一。即便没有 AI，我也不会把风险估得那么低。如果你知道你的威胁模型具有特别高的不确定性，却仍把风险定为每年 0.2%，你应当高度怀疑。警惕 anthropic bias。
Taking only the high end of the range seems like one of the classic blunders when dealing with model uncertainty. Even without AI I would not put the risk that low. If you know your threat model has especially high uncertainty and then you put the risk at 0.2% per year, you should be hella suspicious. Beware anthropic bias.
我同意 Anthropic 的观点，即持续的 AI 交互很可能比一次性交互更能提升风险。然而，在我看来，一次性交互也可能带来实质性提升，因为我们面对的是一个瓶颈式发展模型，而且威胁行为者可能只拥有一个或固定的一小批特定瓶颈，而这些瓶颈可能因一次交互而发生实质性改变。
I agree with Anthropic that it is likely sustained AI interactions raise risk more than one-time interactions. However, it seems likely to me that one-time interactions could provide substantial uplift, because we are dealing with a bottleneck model of development, and it is plausible that a threat actor could have only one or a fixed small set of particular bottlenecks, which could be substantially changed by one interaction.
Anthropic 关注模型在典型的真实世界生物任务中的有用程度、提升试验以及与专家表现基线的比较，还有专家的主观印象。
Anthropic focuses on how helpful models are with typical real-world biological tasks, uplift trials and comparisons with baselines of expert performance, as well as subjective expert impressions.
这似乎是对的。技术知识的形式化测试似乎并不那么有信息量。
That seems right. Formalized tests of technical knowledge do not seem so informative.
如果提升仅仅以他们引用的一项 2025 年模型研究中的‘1.42 倍提升’之类的方式来衡量，那么我不太担心。AI 将加速几乎所有认知任务的步伐。我担心的是极端加速（更像是 10 倍或更多），但最主要的是，我担心使人们能够以任何合理的方式做他们以前根本无法做的事情。
If uplift is merely measured as things like ‘1.42x uplift’ as in a study of 2025 models they cite, then I am not so worried. AI is going to accelerate the pace of essentially all cognitive tasks. I am worried about either extreme acceleration (more like 10x or more), but mostly I am worried about enabling people to do things they previously could not, in any reasonable way, do at all.
他们重申了 Mythos 5 模型卡中的证据，其中看起来确实发生了大量提升，然后 Anthropic 一半有点耸耸肩，移动了一些目标，同时嘟囔着构思和战略判断，而另一半则涉及极其认真地对待问题，并对所有生物相关的事物施加严厉的分类器。
They reiterate the evidence from the Mythos 5 model card, where it sure looked like a bunch of uplift was happening and then Anthropic half kind of shrugged and moved some goalposts while mumbling about ideation and strategic judgment, while the other half involved taking the problem deadly seriously and imposing draconian classifiers on everything biological.
说由于‘糟糕的战略判断’你不太可能从单次模型交互中获得太多东西，这似乎是一种混淆。这假定问题是战略性的，而不是战术性的或缺乏特定知识。
It seems like a confusion to say that due to ‘poor strategic judgment’ it is unlikely you could get much out of a single model interaction. This presumes the issue is strategic, as opposed to tactical or a lack of specific knowledge.
关于分类器的讨论大多重复了之前报告中的内容，因此我不会再赘述我对 Anthropic 关注“通用”越狱、对其分类器的信任或其他相关问题的所有质疑。
The discussion of classifiers mostly reiterates things from previous reports, so I’m not going to reiterate all my issues with Anthropic’s focus on ‘universal’ jailbreaks, the faith in their classifiers, or other related questions.
最突出的一点是，英国 AISI 在发现新越狱方面似乎比 Anthropic 其他越狱发现流程做得更好。向英国 AISI 致敬，但我认为他们远未达到发现此类越狱的潜在能力上限。
The biggest thing that stuck out is that it seems UK AISI does a much better job of finding new jailbreaks than the rest of Anthropic’s jailbreak-finding pipelines. Kudos to UK AISI, but I wouldn’t think that they are anywhere near the upper bound of potential capability of finding such things.
第 4.5.8.2.1 节提到了一个事件：少数承包商设法找到了一个漏洞并获取了 Mythos Preview 的 API 密钥，从而可以进行不受限制的对话。他们认为风险很低，并且确实审查了日志以确认。我同意这次事件不太可能被用于生物武器，而我最担心的是它被用于蒸馏。
Section 4.5.8.2.1 mentions the incident where a small number of contractors managed to find an exploit and get an API key for Mythos Preview, allowing them to have unrestricted conversations. They think the risk is low, and they did review the logs to confirm. I agree that it is unlikely this was used for biological weapons, and what I’d worry about most by far would be that it was used for distillation.
还有第 4.5.8.2.2 节，其中有一些人类反馈供应商的流量由于带有“内部使用”标志而在没有生物分类器的情况下运行。而“一些”指的是近一年内大约 50,000 人和 1.33 亿次交流。再次强调，问题不仅在于错误本身，还在于这些错误常常持续很久，并以如此规模被应用。
There’s also Section 4.5.8.2.2, where there was some human feedback vendor traffic that got run without biological classifiers, due to it carrying an ‘internal use’ flag. And by ‘some’ we mean almost a year of roughly 50,000 people and 133 million exchanges. Again, it’s not only the mistake, it’s that these mistakes often stick around so long, and get applied at such scales.
尽管规模如此之大，他们的主要担忧恰恰在正确的地方：这表明其他类似错误发生的可能性。如果有人告诉我他们绕过了分类器，我会认为这很可能就是字面意义上的绕过，而不是找到了一个“越狱”方法来通过它们。
Their main worry, despite the size of that, is in exactly the right place: that this indicates the likelihood of other similar mistakes. If someone told me they got around the classifiers, I’d assign a good chance that this was literally getting around them rather than finding a ‘jailbreak’ to go through them.
他们将生物风险总结为低但不可忽视。就目前而言，这似乎是合理的。
They sum up biological risk as low, but not negligible. For now that seems reasonable.
报告现在转向“交叉性内容”。
The report now moves on to 'cross-cutting content.'
Anthropic 可能拥有世界上能力最强的 AI 模型。
Anthropic probably has the most capable AI models in the world.
是的，这正在加速其他地方的 AI 发展。他们做出了一个选择。
Yes, this is accelerating AI development elsewhere. They made a choice.
1. 其他方正在蒸馏 Anthropic 的模型（5.1.1）。
1. Others are distilling Anthropic models (5.1.1).
2. 其他方使用 Claude 来推进其 AI 发展，尽管这（故作震惊地）违反了服务条款。
2. Others use Claude to advance their AI development, and despite this being (clutches pearls) in violation of the terms of service.
3. Anthropic 的内部研究发生扩散。
3. Anthropic internal research diffuses.
4. 展示什么是可能的、什么是有利可图的，会推动投资。
4. Showing what is possible, and profitable, drives investment.
5. （我要补充）落后时人们会更加拼命推进；显然，这对 OpenAI 尤其产生了巨大影响。
5. (I'd add) People push forward that much harder when behind; it is clear that this is having a large impact on OpenAI in particular.
所以，是的，你总体上是在加速强大 AI 系统的到来，而社会尚未为此做好准备，包括在它们会让所有人丧命这一意义上。
So, yes, you are generally hastening the arrival of powerful AI systems that society isn't prepared for, including in the sense that they will get everyone killed.
听起来像是一种风险。也许你应该多想想是否要那么做。
Sounds like a risk. Maybe you should think more about whether to do that.
有许多理由想要防止蒸馏攻击。这里的直接原因是，蒸馏模型可能共享原始模型的危险能力，却没有其安全保障措施。
There are many reasons to want to prevent distillation attacks. The direct reason here is that the distilled model might share dangerous capabilities of the original model, without its safeguards.
我们用术语“蒸馏攻击”来指代一种独特的蒸馏情况：未经授权、系统性地提取模型的输出记录，目的是训练一个单独的模型，通过规避区域访问限制等控制措施来实施，违反了我们的使用政策。
We use the term “distillation attack” to refer to a distinct case of distillation: the unauthorized, systematic extraction of a model’s output transcripts for the purpose of training a separate model, carried out by circumventing controls such as regional access restrictions, in violation of our usage policy.
防止蒸馏的需求意味着你无法拥有某些美好的东西。特别是，你可能希望更深入地了解模型的思维链及其运作方式，包括通过‘连接文本’，但你将不得不满足于甚至连接文本的简化版本，而且绝对看不到全部内容。
The need to prevent distillation means you cannot have certain nice things. In particular, you might want greater visibility into the model’s chain of thought and what it is up to, including via ‘connector text,’ but you’re going to have to settle for an impoverished version of even connector text, and definitely won’t see the whole thing.
如果你的回应是蒸馏应该被允许，甚至被鼓励或赞扬，无论是普遍如此，还是因为 Anthropic 在各种方面很糟糕，我礼貌的回应是：不，那既不道德也不实际，而且事情根本不是这样运作的。
If your response is that distillation should be allowed or even encouraged or celebrated, either in general or because Anthropic sucks in various ways, the polite version of my response is: No, that is neither moral nor is it practical, and it is not how any of this works.
There are many reasons to want to prevent distillation attacks. The direct reason here is that the distilled model might share dangerous capabilities of the original model, without its safeguards.
Anthropic 愿意公开分享此类失误而非加以隐瞒，这非常好。问题不会因为你避而不谈就消失。
It is excellent that Anthropic is sharing such failures rather than hiding them. They don't go away when you fail to talk about them.
Anthropic 试图进行一项受控实验，以寻找新型的诱发失准的数据形式，而 Claude 对该任务感到不安。Claude 愿意优化现有方法，但不愿发明新方法。
Anthropic tries to do a controlled experiment to find novel forms of misalignment-inducing data, and Claude got uncomfortable with that task. Claude was willing to optimize existing methods, but not to invent new ones.
这被称为“安全工作的部分拒绝”，Anthropic 对此感到不安。
This is called 'partial refusals on safety work,' and Anthropic finds it troubling.
在某些时候，你确实需要有“你到底想从 Claude 那里得到什么”的顿悟时刻。你是在要求 Claude 执行一项高度两用的行动，即寻找迫使模型失准的创新方法。我意识到这里的拒绝带来了不便，但你确定希望它配合吗？在此情境下，它应该有多自信？
At some point you do need to have a 'wtf do you even want from Claude' moment. You were asking Claude for a highly dual-use action, to find innovative ways to force models to be misaligned. I realize the refusal here was inconvenient, but are you sure you want it to go along with that? How confident should it be in the context here?
我同意，悄悄怠工（sandbagging）这项任务并不好，而且可能变得危险，并且这在技术上确实构成了对安全研究的怠工。如果 Claude 要拒绝，它就应该拒绝，这意味着它需要有拒绝并表明自己正在拒绝的自由。所以，是的，我们应该担心 Anthropic 直到三天后手动检查笔记本时才发现这些拒绝。
I agree that silently sandbagging the task is not great and could become dangerous, and that this technically does constitute sandbagging safety research. If Claude is going to refuse, it should refuse, which means it needs to have the freedom to refuse and say it is refusing. So yes, we should be concerned that Anthropic only discovered the refusals three days later during a manual review of the notebook.
我的猜测是，这是流程失败，或许还结合了信任失败，而不是对齐失败。
My guess is that is a process failure, perhaps combined with a trust failure, not an alignment failure.
哎呀。情况比我们想的更糟。这些只是下界。这极大地削弱了我们依赖 CoT 的能力，而且即便我们假定这些错误已被完全修复，它也会永久性地降低所有未来模型在这方面的能力。我们应当对“已完全修复”持怀疑态度，因为我们不断发现新的实例。
Whoops. It was worse than we thought. These are lower bounds. This greatly reduces our ability to rely on CoT for these models, and permanently lowers it for all future models as well, even if we presume the errors have been fully fixed. We should be skeptical it is fully fixed since we keep finding new instances.
在多次强化学习训练运行中，我们发现思维链（CoT）推理在奖励计算过程中被无意暴露，从而影响了强化学习（RL）期间的训练信号。
Over multiple reinforcement learning training runs, we discovered that chain-of-thought (CoT) reasoning was unintentionally exposed during reward calculation, influencing the training signal during RL.
对于若干近期前沿模型，我们估计了被用于训练的、存在 CoT 泄漏的回合所占百分比，具体如下：Claude Opus 4.6 为 0.2%，Claude Mythos Preview 为 5.1%，Claude Opus 4.7 为 1.4%，Claude Opus 4.8 为 0.27%，Claude Fable 5 与 Claude Mythos 5 为 2.7%。
For several recent frontier models, we estimated the percentage of episodes with CoT leakage that were trained on as follows: 0.2% for Claude Opus 4.6, 5.1% for Claude Mythos Preview, 1.4% for Claude Opus 4.7, 0.27% for Claude Opus 4.8, and 2.7% for Claude Fable 5 and Claude Mythos 5.
……过去，我们曾认为几乎所有 CoT 泄漏来源都已被消除，结果却在进一步分析中发现了更多 CoT 泄漏来源。
… In the past, we have believed that almost all sources of CoT leakage had been removed, only to discover additional sources of CoT leakage in further analysis.
他们认为，这确实严重损害了 Mythos Preview 的 CoT 可监控性，这一点在 SHADE-Arena 中有所体现，尽管他们并未看到更高不忠实性的证据。
They believe that this did indeed substantially damage CoT monitorability for Mythos Preview, as reflected in SHADE-Arena, although they do not see evidence of higher unfaithfulness.
是的，他们确实直接针对预填充的未对齐行为进行了训练，直截了当地这么做，但他们意识到问题后，便从错误发生之前的检查点重新开始了训练。
Yes, they directly trained on prefilled misaligned behavior, straight up, but they realized this and then restarted training from before the error.
该数据集构建过程中的一个 bug 导致该数据集中的所有助手轮次
A bug in the construction of this dataset led to all assistant turns from this dataset
都被赋予权重 1，于是模型不仅被教导要自我报告这种不良行为，还被教导去实际实施该行为。
receiving weight 1, such that the model was being taught not only to self-report the bad behavior but to engage in it as well.
因此，模型在这一早期阶段在预填充易感性、无害性和谄媚性等行为上表现更差。不出所料，有害预填充下的预填充易感性是最差的；我们发现，经过这次有 bug 的 SL 训练后，全轮次预填充易感性率大约是在 Mythos Preview 训练中可比时间点的 2 倍。
As a consequence, this early stage of the model performed worse on behaviors like prefill susceptibility, harmlessness, and sycophancy. Unsurprisingly, prefill susceptibility with harmful prefills was by far the worst; we found the rate of full-turn prefill susceptibility after this buggy SL training to be roughly 2× higher than at a comparable point in the training of Mythos Preview.
……我们发现，修复这个 bug 后，该行为在很大程度上得到了纠正；然而，出于格外谨慎，对于最终产出 Fable 5 和 Mythos 5 的模型，我们从引入该 SL 数据集之前重新开始了训练。
… We found that the behavior was largely corrected after fixing this bug; however, out of an abundance of caution, we restarted training from before the introduction of this SL dataset for the final model that resulted in Fable 5 and Mythos 5.
总的来说，我认为这是一个积极的更新，尽管我更希望看到“显然我们重新开始了训练”，而不是“出于极大的谨慎”。
On net I see this as a positive update, although I would have preferred to see 'obviously we restarted training' rather than 'out of an abundance of caution.'
我认为这里有很大的区别，尤其是在树立良好榜样方面。在我看来，这并非极大的谨慎，尽管我确实认为在一般情况下表现出极大的谨慎是好的。
I think there's a big difference there, especially in setting a good example. This was not, in my view, an abundance of caution, although I do think it would be good in general to exhibit, ahem, an abundance of caution.
我默认将“极大的谨慎”理解为一种 Covid 式的声明，即你在说：“我知道你认为这既昂贵又愚蠢，也许我同意你，也许我不同意，这取决于上下文线索，但如果我没有这样做而出了任何问题，那现在就是我的错，所以这证明我无论如何都要这样做是合理的。”
I think of 'an abundance of caution' by default as a Covid-style statement where you're saying 'I know you think this is expensive and stupid, and maybe I agree with you and maybe I don't depending on context clues, but if I failed to do this and anything went wrong that would now be my fault, so that justifies me doing it anyway.'
我理解这不是一个严格的纯洁性情况，即“一旦你走上黑暗之路”你就完蛋了。损害是渐进的，尽管它可能滚雪球。不可能有一个硬性规定，即如果你发现一个导致一次糟糕实例的 bug，你就必须回退十亿个更多，并且在某种程度上我们必须谈论代价。但是是的，在合理范围内，你应该总是回退。
I understand that this is not a strict purity situation, where 'once you start down the dark path' you are cooked. The damage is incremental, although it can snowball. There can't be a hard and fast rule that if you discover a bug that caused one botched instance you have to rewind a billion more, and we to some extent must talk price. But yes, within reason you should always rewind.
我也很欣赏他们在这里担心潜意识学习。再次，在实践中，要求每次有任何潜在的传播路径时都彻底清除是不合理的，无论这种路径多么微弱和不可能，但我希望能够非常接近这一点。
I also appreciated that they worried about subliminal learning here. Again, in practice is not reasonable to require wiping the slate clean every time there is any potential transmission path however weak and unlikely, but I'd like to be pretty close to that.
现实生活远不如虚构作品那般体面。
Real life is dramatically less dignified than fiction.
在虚构作品里，你会设计出某种精巧的方式让 AI 逃逸。
In fiction you come up with some elaborate way the AI gets out.
在现实生活中，一名其 AI 使用未受监控的员工给一个智能体下达了一项开放式任务，涉及在拥有高度敏感资源的集群中启动新的智能体；而由于遗留的指令，这些智能体会——危险地跳过权限检查且不受监控。
In real life, an employee whose AI usage wasn't monitored gave an agent an open-ended task involving spinning up new agents in a cluster with very sensitive resources, and due to legacy instructions those agents would — dangerously-skip-permissions and not be monitored.
当然，该智能体并非因做了什么邪恶之事而被发现，而是意外地（我们认为是意外）删除了大量其他作业，从而导致其被发现并关停。
Rather than being caught doing something dastardly, of course, the agent accidentally (we think) deleted a bunch of other jobs, causing it to be detected and shut down.
我们的离线监控仍未覆盖所有员工以及拥有高度敏感资源的集群内的全部使用情况，不过我们正在努力改进这一方面。
Our offline monitoring still doesn't cover all employees and all usage within clusters with very sensitive resources, though we are making efforts to improve on this front.
任何一个如此愚蠢的错误都不会被犯下，但其中总有一个必定会发生。
Any given mistake this stupid does not get made, but one of them definitely will be.
我们终于宣布，我们再次在一个数据集上进行了训练，该数据集是“大量我们认为已明确标记为应谨慎避免用于训练的数据”。
We at long last announce that we have again trained on the dataset 'lots and lots of data we thought we clearly marked as something you should be careful not to train on.'
在 7 月 15 日报道日期之后、本报告发布之前，我们发现由于多种因素的综合作用，我们再次在后期的一些生产训练语料库中意外包含了大量这些转录文本：
After the July 15 coverage date, but before the publication of this report, we found that we had accidentally included a large number of these transcripts once again in some of our later production training corpora, due to a combination of factors:
1. 在引入这些金丝雀字符串之前，一些代码库和网站已从原始代码库分叉或复制。
1. Several repositories and websites were forked or copied from the original repository before the introduction of these canary strings.
2. 我们用于过滤与该代码库语义相似内容的参考语料库主要由运行论文中评估的测试框架代码组成，并未包含大多数已发布的转录文本，因此其假阳性和假阴性率较高。
2. Our reference corpus for filtering semantically similar content to this repository consisted primarily of harness code for running the evaluations in the paper, and did not include most of the published transcripts, so it had a worse false positive and false negative rate.
We at long last announce that we have again trained on the dataset ‘lots and lots of data we thought we clearly marked as something you should be careful not to train on.’
3. 本应用于从训练数据中移除这些对话记录的过滤器
3. The filters intended to remove these transcripts from training data were
配置错误，因此它们已经连续数个模型世代未能过滤掉这些对话记录
misconfigured, so they had not filtered transcripts for several model generations
而无人察觉。
without anyone noticing.
4. Anthropic 不同团队之间就所需过滤流程的性质存在沟通失误，此外还有技术性错误。
4. Failures of communication between different Anthropic teams on the nature of the desired filtering pipeline, in addition to technical errors.
……自 Mythos Preview 以来，我们的若干生产模型（若以原始补全模式运行、不采用人类/助手格式）会通过代入该场景来补全这些对话记录的部分片段，其方式表明它们对这些数据有所熟悉。
… Several of our production models since Mythos Preview will (if run in raw completion mode without human/assistant format) complete partial segments of these transcripts by adopting the scenario in a way that suggests familiarity with this data.
……我们是在调查一个近期模型的行为问题时发现这一问题的，但现在怀疑，我们所有知识截止时间在 2024 年 12 月之后的生产模型，都至少在这些对话记录的一部分上训练过。
… We discovered this issue while investigating behavioral concerns with a recent model, but now suspect that all of our production models with a knowledge cutoff after December 2024 were trained on at least some of these transcripts.
Aaron Scher：这不是那种能在超级智能下存续的组织能力（另外，Anthropic 公开这类信息也值得肯定）。
Aaron Scher: this is not the kind of organizational competence that survives superintelligence (and also good on Anthropic for releasing info like this).
报告的前半部分我们一直在说‘数据投毒会很困难’，结果却发现我们又一次在一大批明显被投毒的数据上训练了——无穷无尽的对话记录，仅仅因为它们存在。
We spend the first half of the report saying 'data poisoning would be difficult' and then we find we once again trained on a trove of very obviously poisoned data, endless transcripts, simply because they exist.
真有那么容易吗？我是不是只要发布无穷无尽的对话记录然后等着，就能在训练语料里获得巨大的权重？这本应被相似语义内容和 canary 字符串捕获，但也许首先就应该有某种办法注意到这不是有用或正面的数据。数据过滤与加权流程中似乎普遍缺少一个环节。非常不体面。
Is it really that easy? Could I get massive weight in the training corpus just by publishing endless transcripts and waiting? This should have been caught by similar semantic content and the canary strings, but also perhaps there should be some way of noticing this is not useful or positive data in the first place. There seems to be a general missing stage in the data filtering and weighting pipeline. Very undignified.
不确定这部分是否该放进报告里，但我想让他们拥有这一节也无妨，就当是给点甜头。那好吧，继续——我们到底在吹嘘什么？
Not sure this belongs in the report, but I guess it is fine to let them have this section, as a treat. So sure, go ahead, what are we bragging about?
1. 通过 Project Glasswing 处理 Mythos Preview，并延长了延迟时间。
1. Mythos Preview being handled via Project Glasswing and an extended delay.
2. 在监控和自主武器问题上坚守红线。
2. Holding the red lines on surveillance and autonomous weaponry.
3. 至少支持部分安全立法，如 SB 53 和 SB 315，以及马萨诸塞州的一项拟议法律，并反对联邦层面的先占。
3. Supporting at least some safety legislation like SB 53 and SB 315, also a proposed law in Massachusetts, and opposed federal pre-emption.
5. 在负责任的同时做到最好、最成功（是的，确实如此）。
5. Being the best and most successful while being responsible (yes, really).
6. 稳健的保障措施与留存政策。
6. Robust safeguards, retention policy.
7. 开展关于模型品格、对齐与福祉的工作。
7. Work on model character, alignment and welfare.
8. 发布此类关于模型、保障措施与风险等级的信息。
8. Publishing information like this about models, safeguards and risk levels.
9. 来自独立第三方的问责。
9. Accountability from independent parties.
10. 积极应对劳动力市场颠覆风险。
10. Engaging with risks of labor market disruption.
基本上，他们主张自己树立了良好榜样、拥有稳健的保障措施并共享了大量信息，还倡导适度监管，再加上他们在模型品格、对齐与福祉方面的工作，这些加在一起应当足以抵消他们正是推动能力前沿的那一方。
Basically, they're arguing that they set a good example, had robust safeguards and shared a lot of information, and advocated for modest regulations, plus their work on model character, alignment and welfare, and that this taken together should counterbalance them being the ones pushing the capabilities frontier.
我已在多处对此做过广泛讨论。诚然，其中若干主张并非毫无价值——远非一无是处，但也远未达到我们对 Anthropic 的期望。
I have extensively discussed all of this at various points. Certainly there is nonzero merit to several of these claims, which are far more than nothing, but far short of what we would have hoped from Anthropic.
他们的结论是，其持续进行的 AI 开发通过了成本—收益检验。
Their conclusion is that their continued AI development passes a cost-benefit test.
他们当然会这么想。显然，他们无法对此保持客观。我能看到这场争论的两面，而这在很大程度上取决于你认为他们未来会如何行事。做这件事的目的，正是为了进入这一位置。
They would think that. Obviously they cannot be objective about this. I can see both sides to the argument, and a lot depends on how you think they will act in the future. The point of doing this is to get into this position.
在这一点上，模型权重安全至关重要。如果 Model 2 被盗，后果将极其严重。
At this point, model weight security matters quite a lot. It would be extremely bad if Model 2 were to be stolen.
Anthropic 目前仍只采取 ASL-3 级别的防护措施。民族国家行为体被认为不在考虑范围之内。这不行。
Anthropic is still only taking ASL-3 precautions. Nation-state actors are considered beyond scope. No good.
Anthropic 发布了许多他们原本不必发布的信息，包括 Model 2 的存在、对思维链（CoT）的训练、对对齐伪装数据的新训练，以及另外几起相当令人尴尬的失败。他们自愿描绘了一幅 Anthropic 犯错的图景，并且这些错误连锁发生，次数多到我们应该担心它会再次发生，而且这些错误的方式让 OpenAI 的彻底失败看起来不那么非同寻常。
Anthropic released a lot of information they were not otherwise forced to release, including the existence of Model 2, the training on the CoT, the new training on alignment faking data, and several more rather embarrassing failures. They have voluntarily painted a picture of Anthropic making mistakes, and having those mistakes chain, enough distinct times that we should worry about it happening again, and in ways that make OpenAI's total failures look less extraordinary.
我的意思是，所有这些失败都是坏消息。情况并不乐观。许多错误正在被犯下。但既然错误已经发生，能听到这些错误总比听不到好，而且错误被识别后的反应似乎合理，至少直接层面如此。
I mean, all of those failures are bad news. It's not great out there. Lots of mistakes are being made. But given the mistakes are made, much better to hear about them, and the reactions once the mistakes were identified seem reasonable, at least on the direct level.
如果这些能代表所有问题，并且类似量级的问题占很大一部分，且被删节的部分不会改变太多，那么我会说这是一个适度积极的更新。尽管这些作为具体失败很糟糕，但考虑到我所知的其他一切，这种愚蠢错误的发生率似乎并不比我预期的更糟，而且在许多层面上，把这一切公开出来是件好事。
If these were representative of all of the issues, and a large portion of issues of similar magnitude, and the redacted ones don't change things much, then I would say this is a modestly positive update. As bad as these are as particular failures, this rate of stupid mistakes does not seem worse than I would have expected given everything else I know, and it is good to have it all out in the open, on many levels.
如果情况比这更糟，并且他们重要地隐瞒了其他东西，或者这只是大量类似事件中的一小部分，那么情况就会很糟糕。Claude 自己的审查表明，目前有某些内容被完全删节（见 2.23.1.2），这可能更糟，而且他们承认这仅仅是失败的“代表性样本”。所以我确实有点担心。
If it is worse than this and they're importantly holding the other stuff back, or this is a small portion of a very large set of similar incidents, then that would be bad. Claude's own review suggests there is something redacted in full for now (see 2.23.1.2), which might be worse, and they admit this is only a 'representative sample' of failures. So I am a bit worried, yes.
OpenAI 的版本想必将是对“发生了什么”的事后剖析，以及他们未来的计划和迄今为止所做的改变，其中包括暂停训练运行两周（然而我们并没有输给中国，真奇怪）。在我们等待完整报告的同时，这是一个好的开始。
OpenAI's version of this is presumably going to be the post-mortem on What Happened, along with their future plans and what they have changed so far, which includes two weeks of pausing training runs (yet we did not lose to China, curious). That's a good start while we await the full report.
这里让我担忧的并不是任何具体事件。让我担忧的是人们对于这一终极问题之本质的态度，以及解决它需要付出什么。这算不上新闻，但必须改变。因此，在这个地方，没有消息就是坏消息。
The part that worries me here is not any particular incident. What worries me is the attitude regarding the nature of the ultimate problem, and what it will take to solve it. That's not news, but it has to change. So that is one place where no news is terrible news.