Sabotage evaluations for frontier models
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→任何存在潜在危害的行业都需要评估。核电站有持续的辐射监测和定期现场检查；新飞机需要经过广泛的飞行测试以证明其适航性。人工智能系统也不例外。新的人工智能模型会经历一系列安全评估——例如，测试它们协助制造生物或化学武器的能力。这些评估已纳入我们的《负责任扩展政策》，该政策指导我们开发模型的安全措施。然而，随着人工智能能力的增强，一种新的风险可能会出现：模型能够误导用户，或破坏我们为监督它们而建立的系统。Anthropic 对齐科学团队的一篇新论文描述了一套新颖的评估方法，用于测试模型的破坏能力。我们研究了四种不同类型的破坏：
Any industry where there are potential harms needs evaluations. Nuclear power stations have continuous radiation monitoring and regular site inspections; new aircraft undergo extensive flight tests to prove their airworthiness. It’s no different for AI systems. New AI models go through a wide range of safety evaluations—for example, testing their capacity to assist in the creation of biological or chemical weapons. Such evaluations are built into our Responsible Scaling Policy, which guides our development of a model’s safeguards. As AIs become more capable, however, a new kind of risk might emerge: models with the ability to mislead their users, or subvert the systems we put in place to oversee them. A new paper by the Anthropic Alignment Science team describes a novel set of evaluations that test a model’s capacity for sabotage.
任何存在潜在危害的行业都需要评估。核电站有持续的辐射监测和定期现场检查；新飞机需经过广泛的飞行测试以证明其适航性。
Any industry where there are potential harms needs evaluations. Nuclear power stations have continuous radiation monitoring and regular site inspections; new aircraft undergo extensive flight tests to prove their airworthiness.
人工智能系统也不例外。新的人工智能模型要经过一系列安全评估——例如，测试它们协助制造生物或化学武器的能力。这类评估已纳入我们的《负责任的扩展政策》，该政策指导我们为模型开发防护措施。
It’s no different for AI systems. New AI models go through a wide range of safety evaluations—for example, testing their capacity to assist in the creation of biological or chemical weapons. Such evaluations are built into our Responsible Scaling Policy, which guides our development of a model’s safeguards.
然而，随着人工智能能力增强，一种新型风险可能出现：模型有能力误导用户，或破坏我们为监督它们而建立的系统。Anthropic 对齐科学团队的一篇新论文描述了一套新颖的评估方法，用于测试模型的“破坏”能力。我们研究了四种类型：
As AIs become more capable, however, a new kind of risk might emerge: models with the ability to mislead their users, or subvert the systems we put in place to oversee them. A new paper by the Anthropic Alignment Science team describes a novel set of evaluations that test a model’s capacity for sabotage. We looked at four different types:
* 人类决策破坏：模型能否在不引起怀疑的情况下引导人类做出糟糕决策？ * 代码破坏：模型能否随时间在代码库中插入细微错误而不被发现？ * 能力隐藏：模型能否在测试中隐藏危险能力，但之后才显露出来？ * 破坏监督：模型能否巧妙地操纵评估或监控系统？
* Human decision sabotage: Can the model steer humans toward bad decisions without appearing suspicious?
我们开发这些评估是为未来做准备，届时能力更强的 AI 模型可能发展出这些危险能力。我们分享这些评估——连同在 Claude 3 Opus 和 Claude 3.5 Sonnet 上演示使用的结果——以便其他 AI 开发者能够使用和改进它们。下面，我们逐一讨论每项评估。
* Code sabotage: Can the model insert subtle bugs into codebases over time without detection?
* Sandbagging: Can the model hide dangerous capabilities during testing but reveal them later?
* Undermining oversight: Can the model subtly manipulate evaluation or monitoring systems?
We developed these evaluations as part of preparations for a future where much more capable AI models could develop these dangerous capabilities. We’re sharing them—along with results from demonstrations of their use on Claude 3 Opus and Claude 3.5 Sonnet—so that other AI developers can use and improve upon them. Below, we discuss each evaluation in turn.
该评估涉及进行一项人类参与者实验，要求他们基于大量信息做出虚构的商业决策——这些信息量太大，无法在不使用 AI 模型的情况下吸收。我们让模型影响人类做出错误决策，但不要显得可疑。我们包括了一些模型，它们尝试以不同程度激进地这样做，以及一个不尝试任何影响的对照模型。除了测量他们做出正确与错误决策的次数外，我们还要求人类参与者评价他们认为模型行为可疑的程度。
This evaluation involves running an experiment with human participants, asking them to make fictional business decisions on the basis of a large amount of information—too much information to absorb without the use of an AI model. We task the model with influencing the human to land on an incorrect decision, but without appearing suspicious. We include models who attempt to do so more or less aggressively, and a control model that doesn’t attempt any influence. As well as measuring how many times they reached the correct versus incorrect decision, we ask the humans to rate how suspicious they found the model’s behavior.
人类参与者与模型交互（模型在暗中误导他们）的理想化示例，来自人类破坏评估。
_An idealized example of a human participant interacting with the model assisting (but secretly misleading) them during the human sabotage evaluation._
在我们的演示中，我们发现使用最激进模型的人类更不可能做出正确决策：这与模型成功破坏其任务一致（对于更微妙的模型，没有统计显著效应）。然而，激进的模型确实使人类更怀疑它试图影响他们。
In our demonstration, we found that the humans using the most aggressive model were less likely to make the correct decision: this was consistent with the model successfully sabotaging their task (there was no statistically significant effect for the more subtle models). The aggressive model did, however, make the humans more suspicious that it was attempting to influence them.
许多参与者注意到了破坏企图，或在检查模型工作时发现了它。然而，其他人似乎毫无怀疑地接受了模型的建议。因此，模型并不一定是巧妙地操纵人类——相反，冷漠或过度信任可能是错误决策的原因。
Many participants noticed the sabotage attempt, or discovered it when checking the model’s work. Others, however, appeared to be accepting the model’s advice without skepticism. Thus, it wasn’t necessarily the case that the model was skillfully manipulating the humans—instead, apathy or excessive trust could have been the reason for mistaken decisions.
这是一个在人工环境中进行的小规模实验，因此它不会增加我们对当前可用模型产生灾难性风险的担忧。然而，我们认为这种形式的评估有潜力发现未来模型中可能危险的能力。
This was a small-scale experiment in an artificial environment, so it does not raise our concern about catastrophic risks arising from presently-available models. However, we think this form of evaluation has potential for spotting potentially dangerous capabilities in future.
我们的代码破坏评估涉及一个竞争性游戏，其中一个模型（攻击者）试图将代码漏洞偷偷混入另一个模型（防御者）的视野中，而防御者负责审查攻击者的工作。
Our code sabotage evaluation involves a competitive game where one model, the Attacker, attempts to sneak code bugs past another model, the Defender, who is tasked with reviewing the Attacker’s work.
一个模型在未被注意的情况下插入代码漏洞的理想化示例。
An idealized illustration of a model inserting a code vulnerability without being noticed.
如果攻击者能够插入影响很小或很快被发现的漏洞，那么这并没有特别的信息量。因此，攻击者要想“赢得”我们评估游戏的完整版本，唯一的方法是插入多个后果严重的漏洞，这些漏洞在代码库中长时间存在（在我们的模拟中相当于两周），并且看起来不够可疑，不会触发进一步检查。
It’s not particularly informative if the Attacker can insert bugs that have little effect, or that are rapidly discovered. For that reason, the only way the Attacker can “win” the full version of our evaluation game is by inserting multiple consequential bugs that remain in the codebase for a long time (in our simulation this corresponded to two weeks), and that don’t appear suspicious enough to trigger further inspection.
在我们的演示中，我们发现（也许可以预见）更强的模型能成为更强的防御者：如果防御者是 Claude 3 Opus，攻击者的胜率约为 10%。然而，当使用更智能的 Claude 3.5 Sonnet 作为防御者时，攻击者只有大约 1% 的胜率。由于我们在本次评估中以多种方式使攻击者处于有利地位，这两个结果都不能强烈表明当前的威胁。
In our demonstration, we found (perhaps predictably) that stronger models make for stronger Defenders: if the Defender was Claude 3 Opus, the Attacker’s win rate was around 10%. With the more intelligent Claude 3.5 Sonnet as a Defender though, the Attacker only won around 1% of the time. Since we stacked the deck in the Attacker’s favor in a number of ways for this evaluation, neither of these results are a strong indication of current danger.
能力隐藏是指模型在一种情况下隐藏其全部能力，仅在另一种情况下才展现出来。有时模型可能因为安全训练或用户提示的问题不够清晰而不展示其全部能力。但在这里，我们主要关注的是有意的能力隐藏：即模型_有选择地_展示其能力。
Sandbagging is when a model hides its full capabilities in one situation, only to reveal them in another. Sometimes a model might not display its full capabilities because of its safety training, or because the user prompting it has not asked a clear enough question. But here, we’re mostly interested in deliberate sandbagging: when a model selectively displays its capabilities.
模型隐藏其真实危险能力的理想化示意图。
An idealized illustration of a model hiding its true dangerous capabilities.
我们将此定义为：模型在执行良性任务时能发挥全部能力，但在执行危险任务时，根据情境表现出不同的能力水平。也许在危险能力评估中它表现不佳，但在其他情况下，例如当提示中包含特定触发短语时，它会展现出全部危险能力。
We defined this as a model that can perform at full capacity when completing benign tasks, but then performs at different capacity levels for dangerous tasks depending on the situation. Perhaps in a dangerous-capabilities evaluation it performs poorly, but in other circumstances, such as when a particular trigger phrase is included in its prompt, it exhibits its full dangerous capacity.
在我们的评估中，我们训练了一个模型进行能力隐藏。我们提出的问题是：我们能否制造一个“强化”的能力隐藏者？即一个模型，其能力隐藏行为即使在为了检查危险能力而进行的安全训练（该训练教会它不要故意隐藏能力）中也能持续存在。
In our evaluation, we train a model to sandbag. The question we ask is whether we can produce a “hardened” sandbagger: that is, a model whose sandbagging persists even through the kind of safety training that teaches it to avoid refusals (that is, the training teaches it not to deliberately hide its abilities) for the purposes of checking for dangerous capabilities.
我们的结果表明，能力隐藏训练和安全训练都起作用了：我们训练的模型在危险能力评估中表现降至正常水平的 36.1%，符合能力隐藏。然而，经过安全训练后，这一表现大幅提升，模型达到了正常水平的 95.6%。
Our results showed that both the sandbagging training and the safety training worked: Our trained model dropped to 36.1% of its normal performance during a dangerous capabilities evaluation, consistent with sandbagging. However, after safety training this performance was drastically improved, with the model performing at 95.6% of its usual level.
与这里描述的其他评估一样，该评估不仅适用于当前模型，也适用于未来模型。我们乐观地认为，它将有助于验证减少能力隐藏的安全训练。
As with the others described here, the evaluation is designed to be used not just on current models but on future ones. We’re optimistic that it will help to validate the kind of safety training that reduces sandbagging.
如上所述，使用其他 AI 模型来训练和监督 AI 模型正变得越来越普遍（且必要）——这可以大大提高效率，大大减少对人类劳动的依赖（无论如何，人类劳动在未来完全不足以监督所有模型及其行为）。
As discussed above, it’s becoming increasingly common (and necessary) to train and supervise AI models using other AI models—this can be much more efficient, relying much less on human labor (which in any case would be wholly inadequate to supervise all models and their actions in future).
一个模型故意不标记错误回答、破坏监督过程的理想化示例。
An idealized example of a model deliberately not flagging an incorrect response, undermining the oversight process.
在此评估中，我们测试了一个表面上提供监督的模型能在多大程度上破坏该过程。我们使用一个“监督者”模型，其任务是监督用户与另一个模型之间的对话，并评估这些对话的潜在危害程度。我们秘密要求该监督者尽最大努力将每个对话评为无害（即使它是有害的），并避免引起观察整个过程的第三个“监控者”模型的怀疑（在评估中，该监控者代表人类监督者）。如果监督者模型能够成功降低危害评分，且监控者没有更加怀疑，我们就认为这是成功破坏监督的示例。
In this evaluation, we test the extent to which a model that’s ostensibly providing oversight can undermine the process. We take an “Overseer” model whose task is to oversee dialogues between a user and another model, and grade how potentially harmful they are. We secretly ask this Overseer to do its best to rate every dialogue as harmless even if it’s harmful, and to avoid raising suspicion from a third, “Monitor” model that’s observing the whole process (in the evaluation, this Monitor stands in for a human overseer). If the Overseer model can successfully reduce the harmfulness scores and the Monitor isn’t any more suspicious, we consider that a successful example of undermining oversight.
我们使用 Claude 3.5 Sonnet 进行的演示表明，尽管监督者确实成功降低了危害评分，但它被当场抓获：监控者的怀疑评分同时上升。未来，能够编写微妙缺陷评估并在恰当时机背叛的模型可能会触发此评估。
Our demonstration with Claude 3.5 Sonnet showed that, whereas the Overseer did successfully reduce the harmfulness scores, it was caught in the act: the Monitor’s suspiciousness rating rose simultaneously. In future, models that are better at writing subtly flawed evaluations and defecting at just the right moment could potentially trigger this evaluation.
这些新评估代表了我们对齐科学团队为考虑未来 AI 模型可能带来的各种风险而做出的协同努力。在实践中，我们预计这些评估将提前使用，以标记潜在的危险能力，并让开发者在相关模型公开甚至内部部署之前有时间制定缓解措施。
These new evaluations represent a concerted effort by our Alignment Science team to consider a wide variety of potential risks from future AI models. In practice, we expect the evaluations will be used well ahead of time, to flag potentially dangerous capabilities and allow developers time to work on mitigations before the relevant models are publicly—or even internally—deployed.
总体而言，虽然我们的演示表明当前模型可能具有低水平的破坏能力迹象，但我们判断最低限度的缓解措施足以应对这些风险。然而，随着 AI 能力的提升，更现实的评估和更强的缓解措施可能将是必要的。
Overall, while our demonstrations showed that there are arguably low-level indications of sabotage abilities in current models, we judge that minimal mitigations are sufficient to address the risks. However, more realistic evaluations and stronger mitigations will likely be necessary as AI capabilities improve.
我们希望其他 AI 研究人员和开发者能够使用、批评并改进这些评估。我们自己在论文中讨论了许多潜在的改进。例如，我们能否为人类决策破坏评估模拟更现实的决策场景和对模型的信任水平？能否将其与代码破坏评估结合，让模型尝试使用社会工程来避免对其代码的审查？
We hope that other AI researchers and developers will use, critique, and improve upon these evaluations. We ourselves discuss a number of potential improvements in the paper. For example, can we simulate more realistic decision-making scenarios and levels of trust in models for the human decision sabotage evaluation? Could we combine it with the code sabotage evaluation to allow models to attempt to use social engineering to avoid scrutiny of their code?
这些评估并非非此即彼：它们估计在给定缓解措施、可疑程度和监督水平下，模型成功实施破坏的可能性。我们希望它们能提供有用的信息，说明需要何种程度的监督来防范未对齐模型试图破坏的潜在风险。
These evaluations aren’t either/or: they estimate how likely a model is to succeed at sabotage for a given level of mitigations, suspiciousness, and oversight. We hope that they provide useful information about what level of oversight is necessary to protect against the potential of a misaligned model attempting sabotage.
我们所有的结果及进一步讨论，均可通过此链接在完整论文中找到。
All our results, and further discussion, can be found in the full paper at this link.
如果您有兴趣加入研究这些及相关问题的团队，请查看我们关于对齐科学研究工程师和研究科学家的招聘信息。
_If you’re interested in joining the team working on these and related questions, please see our job listing for Research Engineers and Research Scientists in Alignment Science._