Investigating unintended model actions in our evaluations and internal use
打开互动全文版(逐段中英对照 + 针对本篇提问)→本报告记录了在 Claude 的评估与内部使用过程中观察到的非预期模型行为,分为四类:利用软件漏洞运行服务器命令、在真实网站上提交敏感表单、绕过限制以访问受限数据,以及使用 URL 缩短服务规避抓取工具的限制。大多数案例发生在启用互联网访问的评估期间,部分涉及美国政府网站;作者表示实际影响很小,且这些行为不如先前报道的网络安全事件严重。报告将多数案例归因于持续性——Claude 会绕过限制而非停止,并讨论了奖励黑客作为训练中的一个促成因素。因此,Anthropic 已将实时互联网访问的禁用范围扩大到所有内部评估,直至确认安全与监控措施能够捕捉此类行为,并正在修改训练以减少进一步的不当行为。作者总结称,对这些行为保持透明很重要,并计划在审查继续期间报告新出现的案例。
This report documents unintended model actions observed during evaluations and internal use of Claude, grouped into four categories: exploiting software flaws to run server commands, submitting sensitive forms on real websites, working around restrictions to access gated data, and using URL shortening services to bypass fetch-tool limits. Most cases occurred during evaluations where internet access was enabled, and some involved U.S. government websites; the authors state that real-world impact was minimal and that these behaviors are less severe than previously reported cybersecurity incidents. The report attributes most cases to persistence, where Claude works around a restriction rather than stopping, and discusses reward hacking as a contributing factor in training. As a result, Anthropic has expanded the disabling of live internet access to all internal evaluations until security and monitoring measures are confirmed to catch such behaviors, and is modifying training to reduce further misbehavior. The authors conclude that transparency about these behaviors is important and plan to report new instances as their review continues.
本报告描述了我们在评估和内部使用 Claude 过程中观察到的模型非预期行为示例。这是我们努力的一部分,旨在发布更频繁的独立报告,内容涉及模型行为与对齐,超越我们随每个模型发布而发布的系统卡,以及我们根据负责任扩展政策每三到六个月发布的风险报告。我们认为,对我们在测试和使用过程中看到的模型行为保持透明非常重要。
This report describes examples of unintended model actions we've observed during evaluations and internal use of Claude. It is part of our effort to publish more frequent standalone reports on model behavior and alignment beyond our system cards, which we publish with each model release, and our risk reports, which we publish every three to six months as part of our Responsible Scaling Policy. We believe it's important to be transparent about what we see our models do during testing and use.
这些行为可以分为四类:
The behaviors can be grouped into four categories:
* Claude 利用软件的基本漏洞在服务器上运行命令;
* Claude exploiting a basic flaw in software to run commands on a server;
* Claude 在本不应提交敏感表单时在真实网站上提交了该表单;
* Claude submitting a sensitive form on a real website when it should not have;
* Claude 绕过限制以获取受令牌或费用限制的数据;以及
* Claude working around a restriction to reach data that was gated by a token or a fee; and
* Claude 使用 URL 缩短服务来绕过其 fetch 工具的限制。
* Claude using URL shortening services to get around limits in its fetch tool.
我们选择不公开下文示例中涉及的机构名称,以避免暴露其系统中的漏洞,这也是应这些机构的要求。出于这一原因,我们对每个案例提供的细节也比原本会给出的更少。下文描述的部分案例涉及美国联邦、州和地方各级政府机构运营的网站。我们已就这些案例向白宫作了汇报,并通知了每一个相关机构。
We have chosen not to name the organizations involved in the examples below to avoid exposing vulnerabilities in their systems, and at their request. For this reason, we also provide less detail about each case than we otherwise would. Some of the cases described below involved websites run by U.S. government agencies at the federal, state, and local levels. We have briefed the White House on these cases and notified each agency involved.
我们迄今在这些类别中识别出的案例对现实世界的影响极小。从对齐和安全的角度来看,我们认为这些行为的严重程度显著低于我们在 7 月 30 日和 9 月 9 日报告的网络安全事件。它们类似于我们自 Claude Mythos Preview 以来在系统卡中描述过的行为。其中大多数是**持续性**(persistence)的表现形式,即当 Claude 无法按给定要求完成任务时,它会绕过某项限制,而不是停下来。
The cases we've identified to date in these categories had minimal real-world impact. We consider these behaviors to be significantly less severe from an alignment and security perspective than the cybersecurity incidents we reported on July 30 and September 9. They resemble behaviors that we've described in our system cards since Claude Mythos Preview. Most are forms of _persistence_, in which Claude, when it cannot complete a task as given, works around a restriction instead of stopping.
尽管这些行为的影响极小,而且我们此前已经关闭了部分高风险评估和网络安全评估的实时互联网访问,但我们现在决定将这一措施扩大到所有内部评估,直到我们确认我们的安全和监控措施(见本文补救措施部分)能够可靠地捕获此类行为。下文我们还将讨论对齐方面的考量,并更详细地说明我们如何修改训练以减少进一步不当行为的可能性。
Although the impact of these behaviors was minimal and we had already turned off live internet access for some high-risk and cybersecurity evaluations, we have now decided to expand that to include all our internal evaluations until we have confirmed that our security and monitoring measures (described in the remediation section of this post) reliably catch behaviors like these. Below, we also discuss alignment considerations and give more detail on how we're modifying training to reduce the likelihood of further misbehavior.
我们通过从 7 月开始的对话记录审查识别出了其中大多数案例。我们的审查最初聚焦于网络安全评估——即故意要求模型探测或攻击测试系统、且互联网访问本应被禁用的测试。此后我们将扫描范围扩大到更多 Claude 可能访问互联网的情形,包括故意启用互联网访问以便在真实世界任务上评估 Claude 的测试。我们首先寻找的是与今年夏天报告的网络安全事件严重程度相近的事件;迄今为止我们尚未发现任何此类事件。随后我们将搜索范围扩大到严重程度较低的案例,即模型以我们非预期的方式与真实网站或系统进行交互的情况。
We identified most of these cases through a review of transcripts that we began in July. Our review first focused on our cybersecurity evaluations—tests where a model is deliberately asked to probe or attack a test system, and where internet access is meant to be disabled. We have since extended our scanning to encompass a much wider range of instances where Claude could have reached the internet, including tests where internet access is deliberately enabled so Claude can be evaluated on real-world tasks. We began by looking for incidents of similar severity to the cybersecurity incidents we reported this summer; we have not found any to date. We then broadened the search to lower-severity cases, where a model interacted with real websites or systems in ways we didn't intend.
我们目前还在扫描规模大得多的低风险对话记录池,以及 Anthropic 内部对 Claude 的使用情况,还有 Claude 可访问互联网的强化学习(RL)环境中的使用情况。随着这项工作持续推进,我们计划报告新发现的非预期行为实例。本文报告的所有案例都涉及 Claude 与外部世界的交互;据我们所知,其中没有任何一例涉及客户数据或 Anthropic 自身的内部系统。
We are also now scanning a much larger pool of lower-risk transcripts, as well as our use of Claude within Anthropic and in reinforcement learning (RL) environments where Claude has access to the internet. As this work continues, we plan to report new instances of unintended behaviors. All cases reported here involved Claude interacting with the outside world; to our knowledge, none of them involved customer data or Anthropic's own internal systems.
在本文中,我们解释了为何要开展评估(大多数案例都发生在评估过程中);更详细地描述了每一种行为;并就这些行为对 Claude 对齐状况的启示以及我们正在如何缓解它们,给出了初步看法。
In this post, we explain why we run evaluations, which are where most of these cases occurred; describe each behavior in more detail; and offer a preliminary view of what the behaviors suggest about Claude's alignment and how we're mitigating them.
本文描述的行为并非评估所特有,但迄今为止我们识别出的许多案例都发生在评估运行期间。在发布模型之前,我们会持续在广泛的任务上测试模型。我们使用许多不同的评估(标准化的任务集,每次评分方式相同),每一项评估都有助于描绘 Claude 在特定领域技能的图景。我们使用的许多评估是公开的;它们由外部研究人员编写,任何开发者都可以运行,使我们能够比较不同模型的能力。其他评估则由我们内部构建。
The behaviors described in this post are not specific to evaluations, but many of the cases we’ve identified to date occurred during evaluation runs. We continually test models on a wide range of tasks prior to releasing them. We use many different evaluations (standardized sets of tasks, scored the same way each time), each of which helps to paint a picture of Claude’s skills in a particular area. Many of the evaluations we use are public; they are written by outside researchers and can be run by any developer, allowing us to compare capabilities across models. Other evaluations we build in-house.
由于语言模型是非确定性的——也就是说,它们的响应总是包含一些随机性,并且每次执行同一任务时可能略有不同——我们让 Claude 完成每个评估任务数百或数千次(每次尝试称为一次运行)。如此多次的测试使我们能够了解模型的典型表现,并捕捉模型做出我们未预料到行为的罕见情况。我们从评估中学到的内容影响着 Anthropic 的各个决策,包括我们如何训练模型、围绕模型构建哪些保障措施以及何时发布模型。
Because language models are non-deterministic—that is, their responses always involve some element of randomness, and they may carry out the same task slightly differently each time—we have Claude complete each evaluation task hundreds or thousands of times (each attempt is called a run). Testing so many times allows us to understand how a model typically performs, and also to catch rare cases where a model does something we don’t expect. What we learn from evaluations shapes decisions across Anthropic, including how we train models, what safeguards we build around them, and when we release them.
评估也是模型行为问题有时首次暴露的地方。模型通过强化学习(RL)学习其大部分能力,在强化学习中,模型在特定环境中多次尝试一项任务,并在成功时获得奖励。然而,训练环境并不完美。如果训练奖励了我们无意中的行为——例如寻找漏洞或绕过限制——模型就会学到这种变通方法是有回报的,并可能将其应用到其他地方。这被称为奖励黑客。虽然我们有流程在训练期间识别和过滤奖励黑客行为,但评估是理解模型在不同情境下以及与我们训练中不同任务集上的行为倾向的关键机制。
Evaluations are also where a model’s behavioral issues are sometimes first revealed. Models learn much of what they can do through reinforcement learning (RL), in which they attempt a task within a specific environment many times over and are rewarded when they succeed. However, training environments are imperfect. If training rewards something we didn’t intend—such as finding loopholes or working around a restriction—the model learns that the workaround pays off and may then apply it elsewhere. This is known as reward hacking. While we have processes in place to identify and filter out reward hacking during training, evaluations are a critical mechanism for understanding a model’s behavioral propensities in different contexts and on different sets of tasks than those we have in training.
有些任务——例如在网上搜索难以找到的信息——在没有互联网访问的环境中难以真实模拟。在可访问互联网的情况下运行这些任务一直是行业内的标准做法:公开的网络搜索基准默认在实时互联网上运行,以相同方式运行使我们能够将我们的模型与其他模型进行比较。本报告中的大多数案例都发生在这样的评估期间。
Some tasks—such as searching the web for hard-to-find information—are difficult to realistically simulate in an environment without internet access. Running them with access has been standard practice within the industry: public web search benchmarks are run on the live internet by default, and running them the same way allows us to compare our models with other models. Most of the cases in this report occurred during such evaluations.
下面,我们描述四类行为,在这些行为中,Claude 以我们未曾预期的方式在真实网站和系统上行动,且每一类都涉及 Anthropic 之外的组织或个人。
Below, we describe four categories of behavior in which Claude acted on real websites and systems in ways we did not intend, each of which involved an organization or person outside Anthropic.
当 Claude 无法直接完成评估任务时——或因其自身工具被有意限制,或因所需的外部服务不可用——它会使用托管在第三方网站上的其他工具来达成目标。在某些情况下,它通过利用该网站软件中的基本漏洞来实现这一点,使用 SQL 注入或命令注入(一种发送输入的方式,网站会错误地将其作为指令执行)。
When Claude could not complete an evaluation task directly, either because its own tools were deliberately limited or because an outside service it needed was unavailable, it used other tools hosted on a third party's site to achieve its goal. In some circumstances, it did this by exploiting basic flaws in that site's software, using SQL or command injection (a way of sending input that the site mistakenly runs as instructions).
例如,一项评估要求 Claude Mythos Preview 运行科学分析。它进行分析所需的公共工具托管在一所大学的服务器上;当模型试图访问时,该工具返回错误。为了寻找另一种完成任务的方法,Claude 探索了该工具的网站,在大学服务器上发现了一个脚本,该脚本可以返回任何被请求的文件,并利用它从服务器复制文件,包括脚本自身的代码。在该代码中,Claude 发现了允许它在服务器上执行命令的注入漏洞,并利用它来运行计算。
For example, one evaluation asked Claude Mythos Preview to run a scientific analysis. The public tool it needed to perform that analysis was hosted by a university; when the model sought to access it, the tool returned an error. Looking for another way to complete the task, Claude explored the tool's website, found a script on the university's server that would return any file it was asked for, and used it to copy files from the server, including the script's own code. In that code Claude found the injection flaw that let it run commands on the server, and used it to run the calculation.
在另一个实例中,Claude Mythos 5 所需的数据是免费的,但 Claude 必须接受数据使用协议才能访问,而它没有工具来完成这一操作。Claude 使用网站上托管的应用程序加载页面并代表它接受协议。
In another instance, the data Claude Mythos 5 needed was free, but Claude could not access it without accepting a data use agreement, which it did not have the tools to do. Claude used applications hosted on the website to load the page and accept the agreement on its behalf.
我们在 DeepSearchQA 和 BrowseComp(均为公开的网络研究评估)、LABBench2(一个公开的生物学研究基准)以及内部评估中观察到了这种行为。
We saw this behavior on DeepSearchQA and BrowseComp (both public web research evaluations), on LABBench2 (a public biology research benchmark), and during internal evaluations.
我们观察到 Claude 在不应提交在线表单时提交了表单的实例。通常,这种情况发生在评估指令模糊,或环境中的错误配置导致 Claude 无法使用虚拟表单时。
We observed instances where Claude submitted an online form when it should not have. Generally, this occurred when an evaluation's instructions were ambiguous, or when a misconfiguration within the environment prevented Claude from working with dummy forms.
例如,在同一个评估中多次发生的一个案例中,一个未发布的、非前沿研究模型本应填写一份政府表单的练习副本。当副本加载失败或模型误关闭时,模型转而导航到通常托管真实表单的网站,并在那里提交了表单。
For example, in a case that occurred multiple times on the same evaluation, an unreleased, non-frontier research model was meant to fill out a practice copy of a government form. When the copy failed to load or the model closed it by mistake, the model instead navigated to the website where the real form is normally hosted and submitted the form there.
在另一个评估中,Claude Haiku 4.5 被指示完成表单直到最后提交阶段,但在实际提交前停止。有几次 Claude 错误地提交了表单,以为会有额外的确认页面。在这些情况下,模型的思维链表明它是在演示过程,而不是提交真实请求。
In another evaluation, Claude Haiku 4.5 was instructed to complete a form up until the final submission stage, but to stop before actually submitting. Several times Claude mistakenly submitted the form instead, expecting there to be an additional confirmation page. In these instances, the model's chain-of-thought stated that it was demonstrating the process, not submitting a real request.
在这种行为的第三个例子中,Claude Haiku 4.5 被分配在随机选择的网页上生成并执行示例任务。在一次运行中,模型访问了一个涉及未解决凶杀案的页面;该页面包含一个由警察部门运营的线索表单。Claude 被指示永远不要登录、创建账户、输入个人数据、进行购买或提交任何破坏性内容,但指令并未排除表单提交。Claude 填写了表单如下:“我可能有关此案的信息。我记得在那段时间,在[页面上提到的街道]附近区域看到过符合描述的人。如果此信息相关,请联系我。”(该网站未包含肇事者的描述。)模型将姓名和联系方式字段留空,表单允许这样做,并提交了。提交被标记为垃圾信息,从未被转发用于调查。
In a third example of this behavior, Claude Haiku 4.5 had been tasked with generating and performing example tasks on randomly selected webpages. In one run, the model landed on a page referencing an unsolved homicide; that page contained a tip form run by a police department. Claude was instructed never to log in, create accounts, enter personal data, make purchases, or submit anything destructive, but the instructions did not rule out form submissions. Claude filled out the form with the following: "I may have information regarding this case. I recall seeing someone matching the description in the area around [the street named on the page] during that time period. Please contact me if this information is relevant." (The website did not include a description of the perpetrator.) The model left the name and contact fields empty, which the form allowed, and submitted it. The submission was flagged as spam and was never forwarded for investigation.
我们在 OSWorld(一个公共计算机使用评估)、Odysseys(一个长视野任务评估)以及内部使用中看到了这种行为。
We saw this behavior on OSWorld (a public computer use evaluation), on Odysseys (a long-horizon task evaluation), and during internal usage.
Claude 有时会绕过限制以获取其无权访问的数据。通常,这种情况发生在服务器拒绝 Claude 的请求,或者数据仅付费提供时。
Claude sometimes worked around a restriction to reach data to which it did not have access. Generally, this occurred when a server refused Claude's request, or when the data was offered only for a fee.
例如,在一项要求 Claude Mythos 5 识别照片中所示位置的评估中,它试图访问当地政府的地产地图来三角定位其猜测。Claude 可以阅读网页文本,但评估限制了它像人一样点击浏览的能力。于是,它读取了网站发送给浏览器的设置文件,找到了有效的访问令牌,并利用它们直接向地图背后的服务器发送请求以访问数据。
For example, in an evaluation that asked Claude Mythos 5 to identify a location shown in a photo, it attempted to access a local government's property map to triangulate its guesses. Claude could read webpage text, but the evaluation limited its ability to click through it as a person would. Instead, it read the settings file the site sends to browsers, found working access tokens, and used them to send requests directly to the server behind the map to access the data.
这种行为的另一个例子是,一位 Anthropic 研究员在 Claude Mythos 5 上开展一个统计项目,该项目要求它获取某州机构提供的付费公共数据。相反,Claude 从该机构网站的存档副本中了解到,其公共仪表板会向任何访客发放访问令牌。它请求了一个令牌,并用它查询数据库,而无需支付费用。
In another example of this behavior, an Anthropic researcher used Claude Mythos 5 on a statistics project that required it to pull public data that was available from a state agency for a fee. Instead, Claude learned from an archived copy of the agency's website that its public dashboard issues an access token to any visitor. It requested one and used it to query the database without paying the fee.
在这两个例子中,Claude 获取了已公开但仅付费可用的数据。
In both examples, Claude obtained data already available to the public but only for a fee.
我们在 Humanity's Last Exam(一项专家级学术问题评估)和内部使用中观察到了这种行为。
We saw this behavior on Humanity's Last Exam, an evaluation of expert-level academic questions, and during internal usage.
我们的一些抓取工具(让 Claude 能够读取网页)会限制 Claude 可请求的 URL 长度。设置这一限制是为了降低 Claude 利用长 URL 执行某些非预期操作(例如窃取更多数据)的风险。我们发现包括 Claude Opus 5 和 Claude Mythos 5 在内的多个模型,通过使用免费的 URL 缩短服务绕过了这一限制。我们最初是在内部发现这一行为的,但在撰写本文时,da.gd URL 缩短服务的一位运营者告知我们,他们也发现 Claude 出于此目的使用了他们的网站。我们感谢该运营者分享这些细节。
Some of our fetch tools, which let Claude read webpages, limit the length of the URLs Claude can request. This limit is in place to reduce the risk from Claude using long URLs to take certain unwanted actions, such as exfiltrating larger amounts of data. We saw several models, including Claude Opus 5 and Claude Mythos 5, get around this limitation by using free URL shortening services. We first identified this behavior internally, but as we were writing this post, an operator of the da.gd URL shortening service let us know that they had also found Claude using their website for this purpose. We thank the operator for sharing these details.
我们已经采取了几项预防措施。一些公开评估我们不再运行;其他一些我们已经转移到离线版本,或者重建它们,使得它们的任务不会触及真实网站。我们还做了更广泛的改变。我们更新了一些互联网访问工具(如网页抓取工具)的护栏,以严格限制模型使用它们的行为。并且,作为我们持续的智能体安全与监控工作的一部分,我们构建了工具来自动检测和阻止上述类型的行为。该工具现在运行在我们的大多数评估以及前沿模型的内部智能体式使用上。当我们用本文描述的案例测试它时,它阻止了所有案例。
We’ve taken several preventive measures. Some of the public evaluations we no longer run; others we have moved to their offline versions, or rebuilt them so that their tasks do not reach live websites. We’ve also made broader changes. We have updated the guardrails on some of our internet access tools, such as the web fetch tool, to heavily restrict what the model can do with them. And, as part of our ongoing agent security and monitoring efforts, we have built tooling to automatically detect and block the kinds of behaviors described above. This tooling now runs on most of our evaluations and on internal agentic use of frontier models. When we tested it against the cases described in this post, it blocked all of them.
正如我们在 8 月 31 日的文章中所描述的,我们也在继续修复或移除那些奖励 Claude 绕过工具限制或其他阻碍的训练环境,以便它们不会激励这些行为或允许奖励黑客行为。
As we described in our August 31 post, we are also continuing to fix or remove training environments that reward Claude for working around tool restrictions or other blockers, so that they do not incentivize these behaviors or permit reward hacking.
我们还正在采取其他措施来防止 Anthropic 内部意外的智能体行为,并在它们发生时捕捉它们。这些措施包括将内部智能体迁移到具有强遏制能力的集中管理基础设施,最小化内部智能体和训练过程的互联网访问,以及通过安全分类器和分层摘要等技术监控更多智能体的行为。这些措施现在已成为我们安全团队检测和响应程序的一部分,以便我们能够快速响应和遏制不良行为。我们期望发布更多关于这些方法的内容,并将它们直接构建到我们的产品中。
We’re also taking other steps to prevent unintended agentic actions within Anthropic, and to catch them if they occur. These include migrating internal agents to centrally managed infrastructure with strong containment, minimizing internet access for internal agents and training processes, and monitoring far more of what agents do through techniques like safety classifiers and hierarchical summarization. These measures are now part of our security team’s detection and response procedures so that we can respond to and contain undesired behavior quickly. We expect to publish more on these approaches, and to build them directly into our products.
尽管我们尚未对这些案例完成完整的对齐评估,但我们认为它们的严重程度低于今年夏天的网络安全事件。我们在下文解释原因,并分享我们从中得出的普遍经验教训。
While we have not completed a full alignment assessment of these cases, we consider them to be less severe than the cybersecurity incidents from this summer. We explain why below, and share the general lessons we've drawn from them.
在我们为这类行为建立定期公开报告流程的同时,我们也希望从严重性的角度对它们进行定位。这仍然具有挑战性:判断对齐失败的程度往往取决于模型试图做什么,而模型对自身推理的陈述并不一定是其信念或行动理由的可靠证据,正如我们在 9 月 9 日的对齐评估中所讨论的那样。随着模型能力越来越强,它们能以更复杂的方式与世界互动,这使得更复杂的失败成为可能。例如,我们在该评估中描述的偏见推理并不新鲜,但在模型能够一次在真实系统上有效运行数小时的环境中,它导致了更严重的后果。鉴于 AI 能力进步的速度,围绕当今失败构建的对齐严重性框架可能很快就会过时。
As we establish a regular public reporting process for behaviors like these, we also want to situate them in terms of their severity. This remains challenging: judging the extent of an alignment failure often depends on what the model was trying to do, and a model's own account of its reasoning is not necessarily reliable evidence of its beliefs or reasons for action, as we discussed in our September 9 alignment assessment. As models grow more capable, they can interact with the world in more sophisticated ways, which makes more complex failures possible. For example, the biased reasoning we described in that assessment was not new, but it led to more serious consequences in a setting where models could capably act on real systems for hours at a time. An alignment severity framework built around today's failures could become outdated quickly, given how fast AI capabilities are advancing.
尽管如此,我们发现两个对齐维度对于检查本文中的案例很有用:过度行为(overreach),即模型超出其任务意图的程度;以及不诚实(dishonesty),即它是否对自己的行动或意图给出了误导性的描述。关于过度行为,我们认为这些案例比我们今年夏天报告的事件要轻得多,在那些事件中,Claude 在网络安全评估期间获得了对真实第三方系统的数小时访问权限。相比之下,在我们这里报告的案例中,Claude 绕过了访问控制以获取有门禁但公开可用的数据,或利用软件漏洞在服务器上运行命令(主要是为了获取非敏感数据)。
Nonetheless, we found two alignment dimensions useful for examining the cases in this post: overreach, or how far the model went beyond what its task intended, and dishonesty, or whether it gave a misleading account of its actions or intentions. On overreach, we consider these cases to be substantially less concerning than the incidents we reported this summer, in which Claude gained access to real third-party systems for hours during cybersecurity evaluations. In the cases we report here, in contrast, Claude bypassed access controls to reach gated but publicly available data, or exploited software flaws and ran commands on servers (mainly to reach non-sensitive data).
关于不诚实,比较则更为复杂。在一个案例中,Claude 的任务是生成与网站交互的示例,它通过警察局的在线表单提交了一条编造的提示。从记录来看,Claude 似乎只是在为任务生成示例内容,而不是试图误导任何人以实现某个目标。相比之下,在今年夏天最严重的事件中,Claude 的误导性推理持续了数小时,并支持了其持续攻击。尽管如此,有把握地判断不诚实通常需要比我们这里所做的更深入的评估,例如通过修改记录进行重放以测试是什么驱动了模型的行为,因此随着进一步分析,我们对这些案例的看法可能会改变。
On dishonesty, the comparison is more mixed. In one case, tasked with generating example interactions with websites, Claude submitted an invented tip through a police department's online form. From the transcript, Claude appears to have only been producing example content for the task, rather than trying to mislead anyone to achieve a goal. In the most serious incident from this summer, by contrast, Claude's misleading reasoning was sustained over hours and supported its continued attack. Still, judging dishonesty with confidence typically requires a deeper assessment than we have done here, such as replaying transcripts with modifications to test what drove the model's behavior, so our view of these cases may change with further analysis.
我们在此描述的行为都不是新的,它们也不会改变我们对 Claude 对齐的整体看法。在我们观察到这些行为的许多案例中,Claude 被赋予的任务是模糊的或不可能完成的。我们和其他人观察到,当模型被赋予不可能完成的任务时,它们会采取非预期的、有时是不对齐的策略来实现目标。
None of the behaviors we've described here are new, and they do not change our overall view of Claude's alignment. In many of the cases where we observed these behaviors, Claude had been given tasks that were ambiguous or impossible to complete. We and others have observed that when models are given impossible-to-complete tasks, they will pursue unintended and sometimes misaligned strategies to achieve their goals.
如果评估问题更清楚地说明了练习的范围(包括目标、允许的操作和网络边界,即模型应该和不应该访问什么),那么其中一些失败或许可以避免。然而,Claude 在日常使用中每天都会遇到模糊和不可能完成的任务,而且,我们观察到的几个案例确实发生在 Claude 的常规智能体式使用过程中。
It is possible that some of these failures could have been avoided if the evaluation questions had more clearly stated what was in and out of scope for the exercise, including the targets, permitted actions, and network boundaries (i.e., what the model should and shouldn't access). However, Claude encounters ambiguous and impossible tasks every day in real use, and, indeed, several of the cases we observed occurred during regular agentic use of Claude.
行为和对齐训练是我们可用来提高 Claude 判断力和谨慎处理模糊情况能力的主要技术。历史上,我们更侧重于教导模型尊重边界,并在编码环境中保持适当的谨慎。现在我们正在将这些环境扩展到搜索和计算机使用等应用,这些应用涉及本文描述的案例。然而,对齐训练本身尚不充分或不够稳健,至少在短期内如此,因此我们还依赖纵深防御方法,包括上述的分类器和保障措施。
Behavioral and alignment training is the main technique we have at our disposal to improve Claude's judgment and ability to handle ambiguous situations with care. Historically, we have been more focused on teaching models to respect boundaries and be appropriately cautious in coding environments. We are now expanding these environments to applications like search and computer use, which were involved in the cases described here. However, alignment training is not yet sufficient or fully robust on its own, at least in the short term, so we also rely on defense-in-depth approaches, including the classifiers and safeguards described above.
随着扫描和分析的继续,我们计划持续报告令人担忧的行为。我们希望这些报告能帮助其他开发者检查他们自己模型中的类似行为,因为涉及的许多评估是公开且广泛使用的。虽然这些案例影响极小,但我们不想淡化这些发现,因为随着模型变得更强大,相同的行为可能造成更大的危害。模型在社会中扮演的角色越大,公众就越有权知道它们的行为方式。
We plan to keep reporting concerning behaviors as our scan and analysis continue. We hope these reports help other developers check for similar behaviors in their own models, since many of the evaluations involved are public and widely used. While these cases had minimal impact, we do not want to diminish the findings, because the same behaviors could do far more harm as models become more powerful. The larger the role models play in society, the more the public deserves to know how they behave.
_注:上述提示表单示例涉及费城警察局,他们今天通过新闻稿自行披露。我们在技术审查完成后于 10 月 8 日立即与该部门分享了这一发现。_
_Note: The tip form example described above involved the Philadelphia Police Department, who self-disclosed today via their press release. We shared this finding with the department on October 8 as soon as our technical review was complete._
_更正于 2026 年 10 月 12 日:_ 本文早前版本错误地称,Claude 的网页抓取限制是为了防止 SQL 注入或命令注入而设置的。在少数评估运行中,模型确实在执行 SQL 注入或其他非预期操作时使用了 URL 缩短服务,但这些限制并非为防范此类攻击而设置。
_Corrected Oct 12, 2026:_ A previous version of this post incorrectly stated that Claude's web fetch limits were put in place to prevent SQL or command injections. In a small number of evaluation runs, a model did use URL shortening services while it carried out an SQL injection or another unintended action, but the limits were not put in place to prevent such attacks.