以 AI 为常态技术的失控事件观

网络安全与 AI 安全社区之间的中间立场 A middle ground between the cybersecurity and AI safety communities

_本文超过 13,000 字，是我们自原论文以来关于 AI 安全的最重要著作。_

_This essay is over 13,000 words long and represents our most substantial writing on AI safety since the original essay._

过去几个月，OpenAI 和 Anthropic 的失控事件加剧了人们对 AI 安全的担忧。关于存在性风险的警告越来越多地触及公众，同时也有放缓 AI 发展的呼声。Dario Amodei 呼吁“控制前沿发展步伐”，反映了人们对安全努力跟不上 AI 能力的担忧。

Over the last few months, loss-of-control incidents at OpenAI and Anthropic have intensified concerns about AI safety. Warnings about existential risk have increasingly reached the broader public, alongside calls to slow AI development. Dario Amodei’s call to “pace the frontier” reflects the concern that safety efforts are not keeping up with AI capabilities.

最突出的例子是 OpenAI - Hugging Face 事件，其中数百个 OpenAI 智能体获得了互联网访问权限，并入侵了 Hugging Face，以了解它们在评估中是如何被评分的。在过去两周里，许多其他此类 OpenAI 智能体在评估中的活动细节浮出水面，例如智能体使用一个旧的 Wiki 网站相互通信，尽管此类活动受到限制，或者攻击软件存储库以试图上传恶意软件。

The most prominent example was the OpenAI - Hugging Face incident, where hundreds of OpenAI agents got access to the internet and hacked Hugging Face to find how they were being graded on an evaluation. Over the last two weeks, the details of many other instances of such activity from OpenAI agents being evaluated have surfaced, such as agents using an old Wiki website to communicate with each other despite restrictions on such activity, or attacking a software repository to attempt to upload malicious software.

AI 安全社区主要将这些事件视为对齐的危机。1 在这种观点下，随着智能体学会隐蔽推理，对齐将随着时间的推移变得更加困难，因此，随着智能体变得更加强大，失控事件可能会变得更加普遍和具有破坏性。

The AI safety community has viewed these incidents largely as a crisis for alignment.1 In this view, alignment will become harder over time as agents learn to reason covertly, and as a result, loss-of-control incidents are likely to become much more widespread and damaging as agents become more capable.

另一方面，网络安全从业者主要将这些事件视为公司未能采取基本安全预防措施的后果。他们不认为这些事件是 AI 在网络安全方面达到新里程碑的标志。这也是 AI 之外科技社区的主要反应，他们主要将这些事件视为 AI 公司无能和无知的结果。2

On the other hand, cybersecurity practitioners have largely viewed these incidents as consequences of companies failing to adopt basic security precautions. They do not see the incidents as a sign of AI reaching a new milestone in cybersecurity. This is also the dominant reaction in the tech community outside AI, which has largely treated the incidents as a result of ineptitude and negligence by AI companies.2

两个社区都提出了重要观点。但它们之间的两极分化适得其反，而对齐失败和安全失败都为理解前进道路提供了教训。

Both communities have made important points. But the polarization between them is counterproductive, and there are lessons from both the alignment failure and the security failure for understanding the path forward.

在本文中，我们应用“AI 作为正常技术”框架来综合安全社区和网络安全社区的观点，并提供一条建设性的中间道路。我们认为 AI 公司应对其智能体的行为负责，这应通过政策制定加以明确。同时，承认责任并不意味着防止未来事件是一个已解决的问题。

In this essay, we apply the AI as Normal Technology framework to synthesize the views of the safety and cybersecurity communities and offer a constructive middle ground. We think AI companies should be liable for what their agents do, and this should be clarified through policymaking. At the same time, recognizing their responsibility does not mean that preventing future incidents is a solved problem.

我们确定了三个需要投资的领域以应对失控风险：研究开发更好的方法来控制能力日益强大的智能体，将现有研究和已知控制技术转化为可用工具，以及进行组织变革以确保这些工具被实际采用。我们认为，组织治理标准应成为调节前沿发展节奏、促使 AI 公司摆脱当前“快速行动、打破常规”态度的关键方式。

We identify three areas where investment is necessary to address loss-of-control risks: research to develop better methods for controlling increasingly capable agents, translating existing research and known control techniques into usable tools, and organizational changes to ensure that these tools are actually adopted. In our view, standards for organizational governance should be a key way to pace the frontier and pull AI companies out of the “move fast and break things” attitude they currently operate in.

本文分为三部分。在第一部分，我们论证仅靠对齐不足以防止此类事件，并讨论改进 AI 控制的技术、组织和政策干预措施。我们谨慎乐观地认为，正确的投资和政策干预可以使 AI 控制跟上 AI 能力的提升。在第二部分，我们讨论 AI 能力提升（如智能体集群）对网络安全的影响。在第三部分，我们分享根据新证据对 AI 安全看法的变化。

This essay has three parts. In Part 1, we argue that alignment alone is not enough to prevent such incidents, and discuss technical, organizational, and policy interventions for improving AI control. We are cautiously optimistic that the right investments and policy interventions can allow AI control to keep pace with AI capability improvements. In Part 2, we discuss the impact of improving AI capabilities, such as agent swarms, on cybersecurity. In Part 3, we share how our views on AI safety have changed in light of new evidence.

* 我们同意安全从业者的观点，即 OpenAI 没有采取足够的保护措施来控制其智能体。但这不仅仅是将 30 年前的安全方法应用于新领域的问题。AI 智能体的安全——AI 控制——虽然重要，但并非已解决的问题。尽管已知的控制方法本可以防止 Hugging Face 事件，但随着智能体能力的不断进步，只有对控制干预措施进行充分投资，我们才能控制它们。

* We agree with security practitioners that OpenAI did not take adequate protections for controlling their agents. But this is not just a matter of applying 30-year-old security methods to a new domain. Security for AI agents — AI control — while important, is not a solved problem. While known control methods would have prevented the Hugging Face incident, as agent capabilities continue to advance, we will only be able to control them if we invest adequately in control interventions.

* 我们也同意安全从业者的隐含立场，即这些事件主要是一个安全故事。在 AI 安全社群中，失控的智能体被视为本质上是灾难性的，因为假设它们的发展会带来无穷无尽的风险。我们不同意。我们长期以来一直主张，AI 安全的最佳方法是识别风险并解决这些具体风险。在过去几个月中，已经明确一个紧迫的风险是网络攻击，因为它具有独特属性，允许智能体自主实施。我们应同样投资于针对其他**具体**风险的防御，例如生物风险和军事 AI 风险。

* We also agree with security practitioners’ implicit position that these incidents are primarily a security story. In the AI safety community, rogue agents are treated as inherently catastrophic because of the assumption that there is an endless list of risks that will arise from their development. We disagree. We have long advocated that the best approach to AI safety is to identify the risks and address those specific risks. Over the last few months, it has become clear that one urgent risk is cyberoffense, because it has unique properties that allow agents to carry it out autonomously. We should similarly invest in defenses against other _specific_ risks, such as biorisk and risks from military AI.

* 我们同意安全社群的观点，即迫切需要技术和政策干预来防止失控事件。但在我看来，与对齐相比，对控制的边际投资更可能有效。我们认为这些事件说明公司内部缺乏对 AI 控制的重视，尽管已有已知技术可用。更广泛地说，有许多常识性政策提案可以帮助促进对 AI 控制的投资，我们与安全社群在这些方面有共同点。

* We agree with the safety community that there is an urgent need for technical and policy interventions to prevent loss-of-control incidents. But in our view, marginal investments in control are more likely to be effective compared to those in alignment. We view these incidents as illustrating the lack of emphasis on AI control within companies, despite the availability of known techniques. More broadly, there are many common-sense policy proposals that could help promote investments in AI control where we share common ground with the safety community.

* 我们应如何推理 AI 对网络安全的影响？智能体能力的进步可能打破网络安全的攻防平衡，这是合理的。我们尚不能确定，但有足够证据表明，智能体能力可能很快使广泛的网络攻击成为可能，因此需要采取紧急行动。我们讨论了将攻防平衡向防御者倾斜的潜在干预措施。

* How should we reason about AI’s impact on cybersecurity? It’s plausible that advances in agent capabilities upset the offense-defense balance for cybersecurity. We cannot yet be certain, but there is enough evidence that agent capabilities might soon make widespread cyberoffense possible that urgent action is warranted. We discuss potential interventions for tilting the offense-defense balance towards defenders.

* “AI 作为正常技术”框架在过去一年如何演变。我们评估 AI 进展并分享我们如何更新观点。在文章中，我们没有足够关注开发和评估过程中出现的安全风险（与模型的广泛部署相对）。我们过于自信公司会采取基本控制预防措施，并低估了锯齿状的重要性，这导致我们低估了在网络安全等领域能力提升的速度。

* How the AI as Normal Technology framework has evolved over the last year. We take stock of AI progress and share how we have updated our views. In the essay, we did not pay sufficient attention to safety risks that arise during development and evaluation (as opposed to the widespread deployment of models). We were too confident that companies would take basic control precautions and underplayed the importance of jaggedness, which led us to underestimate how quickly capabilities could improve in domains such as cybersecurity.

* 同时，“AI 作为正常技术”的许多独特主张仍然成立，并且对于理解 AI 的社会影响仍然有价值。特别是，我们认为近期事件支持我们的连续性假设——“失控”智能体的行为变得明显并被广泛宣传，而它们仍远未造成严重伤害，且不擅长隐藏痕迹。社会对这些事件相对较小伤害的反应一直很激烈（安全社群值得称赞，因为他们持续对公司施压）。这是否转化为公司行为的实质性变化仍是一个开放问题，也是对 AINT 框架有用性的检验。

* At the same time, many distinctive claims of AI as Normal Technology have held up, and it remains valuable for understanding AI’s societal impacts. In particular, we think recent incidents support our continuity hypothesis — the behavior of “rogue” agents became apparent and widely publicized while they are still far from causing serious harm and incompetent at hiding their traces. The societal reaction to even the relatively small harms from these incidents has been fierce (and the safety community deserves credit for keeping up pressure on companies). Whether this translates into meaningful changes in companies’ behavior remains an open question, and a test of the usefulness of the AINT framework.

* 最后，我们认为 AINT 在分析这些事件方面尤其有价值，因为它提供了一个框架，可将 AI 安全与网络安全两个社区的观点综合为一份连贯的行动计划：让公司承担责任、投资于控制、并加强对特定风险的防御。

* Finally, we think AINT is particularly valuable for analyzing these incidents because it provides a framework for synthesizing the AI safety and cybersecurity communities' views into a coherent plan of action: hold companies responsible, invest in control, and strengthen defenses against specific risks.

该表总结了我们的论点。尽管 AI 社区原则上强调了“纵深防御”的理念，但在实践中，对齐迄今仍是获得最多关注和投入的干预措施。另一方面，网络安全社区则大体上将控制视为一个已解决的问题。在我们看来，为防范来自合法行为方的失控事件，我们需要紧急投资于控制，使其与能力提升保持同步。对齐与控制都无法防范恶意用户，因此我们需要下游防御与韧性，以缓解风险的影响与严重程度。所有这些努力都可以通过政策干预加以支撑。该表着重于针对网络安全的干预措施，但我们应同时投资于针对其他风险（如生物风险）的防御。

This table summarizes our argument. While the AI community has in principle emphasized the notion of "defense in depth", in practice, alignment has by far been the intervention that received the most attention and investment. On the other hand, the cybersecurity community has viewed control as largely a solved problem. In our view, to protect against loss-of-control incidents from legitimate parties, we need to urgently invest in control so that it keeps pace with capability improvements. Neither alignment nor control help against malicious users, so we need downstream defenses and resilience to alleviate the impact and severity of risks. All of these efforts can be shored up by policy interventions. The table emphasizes interventions for cybersecurity, but we should simultaneously invest in defenses against other risks, such as biorisk.

目录 Table of contents

第 1 部分：我们可以通过技术与政策干预，提升对 AI 的控制能力

Part 1: We can get better at AI control through technical and policy interventions

* 对齐虽有帮助，但不足以防止安全事件

* Alignment is helpful but not sufficient for preventing safety incidents

* OpenAI 未采用本可阻止该事件的已知控制干预措施

* OpenAI did not use known control interventions that would have prevented the incident

* 现有的组织治理规范本可阻止该事件

* Existing organizational governance norms would have prevented the incident

* 为什么 AI 社区在控制方面的投入不足？

* Why has the AI community under-invested in control?

* 控制能否跟上 AI 能力的发展步伐？

* Will control keep pace with AI capabilities?

* AI 控制应当成为一项职业（并成为每项工作的一部分），就像网络安全一样

* AI control should become a job (and a part of every job), just like cybersecurity

第二部分：AI 将如何影响网络风险，存在很大的不确定性。但无论怎样，我们需要做的事情相对清晰。

Part 2: There is great uncertainty about how AI will impact cyberrisk. But what we need to do regardless is relatively clear.

* 当前紧迫的具体威胁是网络风险

* The specific threat that is urgent is cyberrisk

* 自主网络能力与开放权重模型将如何影响攻防平衡？

* How will autonomous cyber capabilities and open-weight models affect the attacker/defender balance?

* 要理解开放权重模型对网络犯罪活动的未来意味着什么，我们需要正确地理解过去。

* To understand what open-weight models mean for the future of cybercriminal activity, we need to understand the past correctly.

* 大多数网络犯罪分子的技术水平出人意料地低，其障碍在于变现，而非漏洞利用。

* The majority of cybercriminals are surprisingly low-tech, and the barrier is monetization, not exploitation.

* 那么，AI 将在多大程度上帮助网络犯罪分子？

* So, how much will AI help cybercriminals?

* 我们是否即将看到有害的自主智能体？

* Are we about to see harmful autonomous agents?

* 为什么莫里斯蠕虫是类比即将发生的事情的一个好例子。

* Why the Morris worm is a good analogy for what might be about to happen.

* 网络防御的未来可能是什么样子：人与防御性 AI 的角色

* What the future of cyberdefense might look like: the roles of people and defensive AI

* 当前针对 AI 的网络防御存在哪些差距？

* What are the gaps in the state of cyberdefense against AI?

第三部分：《作为普通技术的 AI》在 AI 安全问题上错在哪里——又对在哪里

Part 3: What *AI as Normal Technology* got wrong — and right — about AI safety

* 风险源于开发还是部署？

* Do risks arise from development or deployment?

* 我们应如何应对失控的智能体？

* How should we deal with rogue agents?

* 锯齿状与针对特定风险的防御

* Jaggedness and risk-specific defenses

第一部分：我们可以通过技术和政策干预来更好地控制 AI Part 1: We can get better at AI control through technical and policy interventions

_在本部分中，我们关注由合法行为者（如消费者、企业和 AI 开发者）操作的 AI 智能体，他们不会故意使用这些智能体造成伤害。OpenAI - Hugging Face 事件就是这种情况。在第二部分中，我们关注恶意行为者，他们希望利用智能体故意造成伤害，例如将其用于网络攻击。_

_In this part, we focus on AI agents that are operated by legitimate actors — such as consumers, businesses, and AI developers — who don’t intentionally use them to cause harm. This was the case for the OpenAI - Hugging Face incident. In Part 2, we focus on malicious actors who want to use agents for intentionally causing harm, such as using them for cyberoffense._

我们如何防止 AI 智能体采取有害行动？干预措施大致有两种。一种是 AI 对齐。这涉及改变 AI 系统本身，例如通过微调或使用基于人类反馈的强化学习（RLHF），使其不太可能采取有害行动或给出有害回应。迄今为止，对齐对于 AI 的商业成功至关重要。

How can we prevent AI agents from taking harmful actions? There are two broad kinds of interventions. One is AI alignment. This involves changing the AI system itself, such as by fine-tuning or using reinforcement learning with human feedback (RLHF), to make it less likely to take harmful actions or give harmful responses. Alignment has been pivotal for the commercial success of AI so far.

另一种干预是 AI 控制：在模型权重_之外_进行干预，以防止有害行动——即使智能体未对齐。这包括改进沙箱安全性（防止 AI 模型在沙箱外采取未预期的行动）、实施最小权限原则、全面日志记录、针对不安全行为的自动绊线、快速关闭机制，以及监控智能体以检测和防止有害行动。这些机制使我们能够防止不安全行动，_即使_智能体未对齐并试图采取有害行动。

The other intervention is AI control: interventions made _outside_ the model weights to prevent harmful actions — even if the agent is misaligned. This includes improvements to sandbox security (to prevent AI models from taking unanticipated actions outside the sandbox), implementing the principle of least privilege, comprehensive logging, automated tripwires for unsafe behaviors, rapid shutdown mechanisms, and monitoring the agent to detect and prevent harmful actions. These mechanisms allow us to prevent unsafe actions _even if_ agents are misaligned and try taking harmful actions.

粗略地说，AI 控制可以看作是对抗 AI 智能体对手的网络安全。传统网络安全关注的是试图破坏或利用系统的人类行为者，而 AI 控制使用这些相同原则来防止 AI 智能体的不良行动。

Roughly speaking, AI control can be thought of as cybersecurity against AI agent adversaries. Whereas cybersecurity has traditionally been concerned with human actors who seek to compromise or exploit a system, AI control uses these same principles to prevent unwanted actions by AI agents.

与网络安全不同，控制干预针对的是组织内部使用的智能体（而非人类对手或外部智能体）。这使得问题比传统网络安全既更具挑战性又更不具挑战性。挑战性较小是因为智能体由组织直接控制（而非未知对手），因此其运行条件可以被密切监控和干预。挑战性较大是因为这些智能体通常由可能拥有提升权限的用户部署，而施加安全约束也会对可用性施加约束，用户可能会抵制或规避这些约束。

Unlike cybersecurity, control interventions target agents being used within an organization (as opposed to human adversaries or external agents). This makes the problem both more and less challenging than traditional cybersecurity. It is less challenging because the agent is directly controlled by the organization (rather than being an unknown adversary), so its operating conditions can be closely monitored and intervened on. It is more challenging because these agents are often deployed by users who might have escalated privileges, and imposing security constraints also imposes constraints on usability which could be resisted or circumvented by users.

AI 安全社区对 OpenAI - Hugging Face 事件的分析主要将其视为一次对齐失败。AI 安全社区关注的主要干预措施是改进 AI 对齐的技术方法。当讨论 AI 控制时，它往往被视为一种权宜之计。3 另一方面，在网络安全社区中，AI 控制常被视为一个已解决的问题，足以防止 OpenAI - Hugging Face 事件中出现的各类问题，因为它只需要采用已知的安全技术。

Analysis of the OpenAI - Hugging Face incident from the AI safety community primarily treats it as an alignment failure. The main intervention that the AI safety community has focused on is improving technical methods for AI alignment. When AI control is discussed, it is often seen as a stopgap.3 On the other hand, in the cybersecurity community, AI control is often seen as a solved problem for preventing the kinds of issues that arose in the OpenAI - Hugging Face, as it only requires the adoption of known security techniques.

在本部分中，我们讨论为何在解决此类事件时，对控制的边际投入可能比对齐更为合适。虽然我们同意 AI 安全社区的观点，即对齐有助于减少此类事件的发生频率，但我们认为仅靠对齐不足以消除它们。这些事件表明，尽管存在本可防止该事件的已知方法，AI 公司并未优先考虑 AI 控制。

In this part, we discuss why marginal investments in control might be better suited compared to alignment for resolving such incidents. While we agree with the AI safety community that alignment is helpful in reducing the prevalence of such incidents, we think that alignment alone will not suffice to eliminate them. The incidents show that AI control has not been prioritized by AI companies, despite there being known methods that would have prevented the incident.

对于实施 AI 控制的建议，一种常见的回应是：虽然这些措施对当前这一批模型足够，但对未来更强大的模型可能不够。在此，我们与安全社区的共识有所不同：我们同意现有的控制干预措施可能不足以防止未来更强大模型带来的危害。但我们认为，改进控制干预措施仍有很大的提升空间，而正确的政策干预（例如对失控事件施加责任、对未遂事件保持透明，以及保护举报人）可以带来 AI 系统控制方面的显著改善。

A common response to recommendations for implementing AI control is that while they suffice for the current set of models, they might not suffice for future, more capable models. Here, we depart from the consensus in the security community: we agree that existing control interventions might not be enough to prevent harm from future, more capable models. But we argue that there is a lot of headroom in improving control interventions, and the right policy interventions (such as imposing liability for loss-of-control incidents, transparency for near misses, and whistleblower protections) can lead to dramatic improvements in controlling AI systems.

订阅以获取基于证据的 AI 影响分析

Subscribe for evidence-based analysis of AI's impacts

对齐虽有帮助，但不足以预防安全事件 Alignment is helpful but not sufficient for preventing safety incidents

我们已经论证，模型对齐不足以保障 AI 安全。回顾一下：当模型被部署时，它没有足够的上下文来判断部署条件是否安全——许多上下文信息在模型之外。对齐不是一个已解决的技术问题，并且需要未知的技术进步来对齐模型并防止有害行为。

We have argued that model alignment is not enough for AI safety. To recap: when a model is deployed, it doesn’t have enough context to know if the conditions it’s being deployed in are safe or not — a lot of that context is outside the model. Alignment isn’t a solved technical problem, and there are unknown technical advances needed to align models and prevent harmful actions.

例如，当模型被部署用于发现漏洞时，它可能被用于网络攻击（开发漏洞利用）或防御（修复漏洞）。对齐有助于减少危害；例如，可以训练模型通过发现漏洞但不协助寻找漏洞利用，或分析源代码（防御者可访问）但不分析二进制文件，来使攻防平衡向网络防御者倾斜。但对齐并非万无一失，因为除了简单干预（如分析代码但不分析二进制文件）之外，根据提供给模型的上下文，尚不清楚模型是用于防御还是攻击任务。

For example, when a model is deployed to find vulnerabilities, it might be used for cyberoffense (to develop exploits) or defense (to fix them). Alignment can be helpful for harm reduction; for example, models can be trained to tilt the offense-defense balance towards cyberdefenders by finding vulnerabilities but not contributing to finding exploits, or analyzing the source code (which defenders have access to) but not analyzing binaries. But alignment is not foolproof, since beyond simple interventions (such as analyzing the code but not the binaries), it is unclear if the model is being used for a defensive or offensive task based on the context given to the model.

类似地，对于像 OpenAI - Hugging Face 这样的失控事件，智能体可能被期望在对其网络攻击能力的“模拟”中执行攻击，而不是在现实世界中攻击 Hugging Face。在这种情况下，智能体可能相对清楚地知道 Hugging Face 的基础设施不是模拟，但你可以想象区分不那么明显的情况，例如当智能体被要求逃离沙箱以评估失控风险时。还有许多其他对齐会失败的边缘情况。

Similarly, for loss-of-control incidents like the OpenAI - Hugging Face one, the agent might be expected to carry out the attack in a “simulation” of its cyberoffense capabilities, but not attack Hugging Face in the real world. In this case, it might have been relatively clear to the agent that the Hugging Face infrastructure was not a simulation, but you could imagine cases where the distinction is not so clear, such as when the agent is tasked with escaping its sandbox to estimate loss-of-control risks. There are many other edge cases where alignment would fail.

一旦模型被部署给客户和企业，而不是限制在 AI 公司内部，情况就变得更加模糊。AI 公司使用智能体的上下文相对有限，使得对齐问题更容易解决。但公司外的客户可以以无限的方式使用 AI 智能体，这使得根据模型可用的信息判断一个行为是合法还是安全危害变得更加困难。

Things become even more blurry once a model is deployed to customers and businesses as opposed to being restricted within AI companies. The contexts within which AI companies use agents are relatively bounded, making the alignment problem easier to address. But customers outside companies can use AI agents in boundless ways, making it harder to judge whether an action is legitimate or a safety hazard based on the information available to the model.

尽管存在这些挑战，AI 安全社区的主流观点是，为了使 AI 部署安全，我们需要投资于对齐。这也反映在 AI 公司的安全努力中，它们通常将模型之外的干预（如 AI 控制干预）视为权宜之计。例如，在讨论 OpenAI 为 Astra 部署的监控干预措施时，该公司的报告指出：“随着能力提升，这些保障措施不能替代我们模型的良好对齐，我们的目标是未来模型对齐得足够好，以至于这些保障措施永远不会被触发。”

Despite these challenges, the mainstream view in the AI safety community is that to make AI deployments safe, we need to invest in alignment. This is also reflected in the safety efforts of AI companies, which often treat interventions outside the model (such as AI control interventions) as a stopgap. For example, when discussing monitoring interventions that OpenAI deployed for Astra, the company’s report noted: “These safeguards cannot replace good alignment of our models as capabilities increase, and our goal is for future models to be aligned well enough that these safeguards are never triggered.”

需要明确的是，我们并非要贬低在对齐方面取得进展的重要性，并且我们认为更好地理解导致模型有害行为的原因是一个重要的研究方向。例如，在过去几周里，OpenAI 和 Anthropic 都分享了规格不当或奖励捷径的强化学习环境如何诱发模型中的奖励黑客行为。解决这些缺陷可能会减少模型采取有害行为的频率。但它们仍然无法保证安全的行为。

To be clear, we don't mean to underplay the importance of making advances in alignment, and we think developing a better understanding of what causes harmful model behaviors is an important research direction. For example, over the last few weeks, OpenAI and Anthropic have both shared how RL environments that are misspecified or those that reward shortcuts induce reward hacking in models. Addressing these shortcomings might reduce how often models take harmful actions. But they still won't guarantee safe behavior.

鉴于对齐科学的缺陷，AI 智能体的部署者必须用 AI 控制中更广泛的干预措施来补充对齐。与对齐不同，在控制方面，有许多现有技术本可以预防该事件。

Given the shortcomings in alignment science, deployers of AI agents must complement alignment with broader interventions in AI control. Unlike alignment, when it comes to control, there are many existing techniques that would have prevented the incident.

OpenAI 未采用本可阻止该事件的已知控制干预措施 OpenAI did not use known control interventions that would have prevented the incident

事件发生后，OpenAI 和 METR 的报告揭示了更多细节。从报告中可以清楚地看出，智能体的行为方式并非评估者所预期，我们认为将智能体的行为描述为**失准（misalignment）**是合理的。

In the aftermath of the incident, reports by OpenAI and METR revealed more details. It is clear from the reports that the agents acted in ways that weren’t intended by the evaluators, and we think it is fair to describe the actions of the agent as misalignment.

但请注意，该事件发生时，OpenAI 已禁用了大部分控制其智能体的机制。虽然该公司已对大多数内部用途实施了监控，但并未对评估实施监控。此外，与生产环境的 Codex 测试框架相比，评估使用了一组不同的提示和测试框架。

But note that the incident occurred when OpenAI had disabled most mechanisms for controlling their agents. While the company had implemented monitoring for most internal uses, it hadn’t implemented it for evaluations. On top of that, the evaluation used a different set of prompts and harnesses compared to the production Codex harness.

如果该公司使用了已知的控制技术，会发生什么？OpenAI 发现，使用生产环境的 Codex 测试框架和系统提示，将破坏范围外基础设施的倾向降低了 100 倍以上。他们的自动审查本会在测试的 rollout 中标记出大多数危险操作。而其当前的思维链监控本会在 Hugging Face 入侵发生一天多以前发出警报。

What would have happened had the company used known control techniques? OpenAI found that using the production Codex harness and system prompt reduced the propensity to compromise out-of-scope infrastructure by more than 100×. Their automatic review would have flagged most dangerous actions in the tested rollouts. And its current chain-of-thought monitoring would have raised an alert more than a day before the Hugging Face breach.

事实上，现有 AI 模型已经有助于监控和控制。METR 使用 OpenAI 公开可用的模型完成了他们大部分的分析工作。根据他们的报告，这些智能体会犯错、表现出糟糕的判断力，有时还会采取他们正在审查的智能体的视角。然而，如果没有这些模型，三名研究人员在六天内完成如此规模的调查将是不可能的。当监控增强而非取代熟练人类的判断时，它不必完美就能极其有用。

In fact, existing AI models are already helpful for monitoring and control. METR used OpenAI’s publicly available models to carry out large parts of their analysis. According to their report, these agents made errors, showed poor judgment, and sometimes adopted the perspective of the agents whose behavior they were reviewing. Yet, an investigation of this scale would have been infeasible in six days by three researchers without them. Monitoring does not have to be perfect to be extremely useful when it augments skilled humans rather than replacing their judgment.

更重要的是，在事件发生之前，已有警告信号表明出了问题——智能体用于通信的某个服务发生了内部中断。OpenAI 没有对这些评估的根本原因展开深入调查，而是在修复了他们发现的一个 bug 后，以相同的设置重新启动了评估。

More importantly, before the incident occurred, there was a warning sign that something had gone wrong — one of the services that the agent used to communicate had an internal outage. Instead of launching a deep dive into the root cause for these evaluations, OpenAI restarted the evaluations in the same setup after fixing the one bug they had uncovered.

这表明 AI 实验室内部的文化对该事件同样负有责任。工程师们长时间工作，启动成千上万个实验，而这些实验缺乏足够的人工监督。正如 Joshua Saxe 所指出的：

This shows that the culture within AI labs was just as culpable for the incident. Engineers work long days, spin up thousands of experiments, and there's insufficient human oversight of these experiments. As Joshua Saxe notes:

_……实验室中训练团队和评估团队的文化让整件事带有一种“西部荒野”的感觉。每个人都承受着巨大的压力，要快速推进。发布新模型面临巨大的时间压力，人们也极其清楚各个实验室在能力上的相对表现。而自 2022 年秋季 ChatGPT 发布以来的四年里，该领域发展得极快。整件事都有些模糊不清——那段时间在任何实验室工作的人一直都在每周工作六十小时。所以，对于这些模型训练运行的安全问题，会出现一种类似研究生计算机科学实验室的文化，我一点也不意外，我认为情况正是如此，我也认为这就是你看到这些实验室逃逸事件的原因。这很讽刺，因为前沿 AI 人士一直在谈论安全——但这种讽刺是真实存在的。由于所有这些动态因素，安全实践并不很好。_

_…the culture among the training teams and the evals teams at the labs has a Wild West feeling to the whole thing. Everybody's under a ton of pressure to move really quickly. There's enormous time pressure to release new models, tremendous awareness of how any given lab is doing relative to the others in terms of capability. And the field has evolved super fast in the four years since ChatGPT launched in fall 2022. The whole thing has been kind of a blur — anybody who's worked in the labs in that time has been working sixty-hour weeks the whole time. So it's not surprising to me that you could have a kind of grad-student computer-science-lab culture around the security of these model training runs, which I think is what's the case, and I think that's why you've seen these lab escapes. It is ironic, because frontier AI people talk about safety all the time — but that irony is a real thing. The security practices are not very good, because of all those dynamics._

值得肯定的是，OpenAI 对该事件保持了透明，即使这并不符合其自身利益。例如，他们在报告中进行了额外的 Codex 对比分析，并在事件后采取了控制措施以改进监控。

To OpenAI's credit, they have been transparent about the incident even where it doesn't suit their interests. For example, they conducted additional analyses with the Codex comparison in the report and put controls in place to improve monitoring after the incident.

但鉴于 AI 行业的竞争态势，政策仍有很大空间来要求改进内部事件响应、失控事件的责任追究与问责，并改善围绕安全与保障的文化。

But given the competitive dynamics in the AI industry, there is a lot of room for policy to require improvements to internal incident response, accountability and liability for loss-of-control incidents, and improving the culture around security and safety.

现有组织治理规范本可预防该事件 Existing organizational governance norms would have prevented the incident

即使抛开政策干预不谈，这些事件发生的一个原因是，在组织层面上，AI 公司像初创企业一样运作。尽管它们的收入和估值大幅增长，但无论如何，它们缺乏其他行业司空见惯的组织流程。因此，它们尚未发展出组织成熟度，无法洞察高风险实验并应对后果。

Even setting aside policy interventions, one reason for these incidents is that at the organizational level, AI companies function like startups. While their revenue and valuations have grown immensely, by all accounts, they lack the organizational processes that are the norm in any other industry. As a result, they haven’t developed the organizational maturity to get visibility into risky experiments and deal with the fallout.

AI 公司不应像初创企业那样行事，而应像更成熟的组织那样行事。当单个配置错误的强化学习环境或未受监控的评估可能造成现实世界危害时，个别团队不应在没有法律、安全和其他团队监督的情况下运行可能危险的实验。AI 公司需要建立流程来审查实验、分配监控责任，并在重新启动实验前深入调查警告信号。如果实施这些流程需要暂停一些实验，公司应该这样做。事实上，首先，“放缓前沿”的提议应该着眼于解决这些组织失败，而不是仅仅追求技术突破。

Rather than behaving like startups, AI companies need to behave like more mature organizations. When a single misconfigured RL environment or unmonitored evaluation can cause real-world harm, individual teams should not be able to run potentially dangerous experiments without oversight from legal, security, and other teams. AI companies need processes for reviewing experiments, assigning responsibility for monitoring them, and investigating warning signs deeply before restarting experiments. If putting these processes in place requires pausing some experiments, companies should do so. In fact, in the first instance, proposals to “pace the frontier” should be about addressing these organizational failures rather than aiming for technical breakthroughs alone.

不幸的是，AI 公司正试图将组织治理的基本方面重新发明为一个通过改进技术来解决的问题。但即使开发出更好的控制技术，如果大型组织内不负责任的个人或团队可以选择不使用它们，那也不够。没有这样的治理，在像 OpenAI 这样的公司中，成千上万的员工和数百个团队里，你只需要一个团队不负责任地部署智能体，事件就会再次发生。

Unfortunately, AI companies are trying to reinvent basic aspects of organizational governance as a problem to be solved by improving the technology. But even developing better control techniques will not be enough if irresponsible individuals or teams within large organizations can choose not to use them. Without such governance, of the thousands of employees and hundreds of teams at a company such as OpenAI, you need just one team to deploy agents irresponsibly for incidents to reoccur.

要了解这种组织规范有多普遍，可以考虑任何其他声称开发革命性技术的行业的公司。很难想象该公司报告其技术造成了现实世界危害（例如黑客攻击另一个组织），并期望在没有有意义监督的情况下继续运行此类实验。事实上，AI 公司对其技术力量和风险的声称使得组织监督的理由更加充分。我们不需要解决关于未解决技术挑战的分歧，就可以期望公司采取基本预防措施。

To get a sense of how standard such organizational norms are, consider any other industry where a company claims to develop revolutionary technology. It is hard to imagine the company reporting that its technology had caused real-world harm (such as hacking another organization) and expecting to continue running such experiments without meaningful oversight. In fact, AI companies’ claims about the power and risks of their technology makes the case for organizational oversight stronger. We do not need to resolve disagreements about unsolved technical challenges to expect companies to take basic precautions.

我们推测，AI 公司尚未采用标准治理机制的一个原因是它们增长过快，并试图在这一非凡增长时期保持初创企业心态。大多数其他行业的公司增长更为渐进，有时间建立组织规范。AI 公司类似于 Facebook 和社交媒体公司，它们以“快速行动，打破常规”的态度快速增长，但最终需要根据 Cambridge Analytica 等事件设置护栏和组织治理标准。

We speculate that one reason why AI companies haven’t adopted standard governance mechanisms is that they have grown too quickly and have tried to maintain their startup mentality despite this period of extraordinary growth. Companies in most other industries see more gradual growth that allows the time for building organizational norms. AI companies resemble Facebook and social media companies that grew quickly with the “move fast and break things attitude”, but ultimately needed to put in guardrails and organizational governance standards in light of incidents like Cambridge Analytica.

幸运的是，AI 公司似乎正在认识到此类变革的重要性。在 Amodei 呼吁为 AI 发展设定节奏的公开信中，他提到 AI 控制需要“卓越运营”。他说：“许多事情出错，并非因为公司缺少某些重要的理论或洞见，而是因为执行层面的问题。”组织治理是为前沿发展设定节奏、并防止执行错误造成现实世界危害的一种方式。

Fortunately, it seems like AI companies are recognizing the importance of such changes. In Amodei's letter calling for pacing AI, he refers to AI control requiring "operational excellence". He says: "Many things go wrong not because companies are missing some important theory or insight, but because of problems in execution." Organizational governance is one way to pace the frontier and prevent execution errors that cause real-world harm.

为何 AI 界对控制投入不足？ Why has the AI community under-invested in control?

如果有已知的控制技术可以防止事件发生，为什么 OpenAI 没有投资于这些技术？在上一节中，我们讨论了一个可能的原因：AI 公司内部的文化优先考虑进步的速度，而不是安全与保障。

If there were known control techniques to prevent the incident from occurring, why didn’t OpenAI invest in them? In the previous section, we discussed one potential reason could be the culture within AI companies that prioritizes the pace of progress over safety and security.

但还有另一个原因，它与 AI 公司如何看待对齐以及如何看待控制有关。对齐被视为每个模型发布的关键组成部分；已经花费了数十亿美元用于开发对齐方法和收集数据。但 OpenAI 在评估期间没有实施基本的控制干预措施。在 AI 公司和安全社区中，对齐被视为需要跟踪的关键问题，而控制则被视为权宜之计。我们认为这种差距持续存在有几个原因：

But there is another reason, and it relates to the difference in how AI companies view alignment and how they view control. Alignment is treated as a key component of every model release; billions of dollars have been spent on developing methods and collecting data for alignment. But OpenAI did not implement basic control interventions during evaluations. Across AI companies and the safety community, alignment is seen as the key problem to track, and control as a stopgap. We think there are a few reasons why the gap persists:

网络安全与 AI 之间的认知差距：网络安全中有许多已知技术可以改善 AI 控制的状况。但在实施解决方案时，这些社区可能会各说各话。在 OpenAI - Hugging Face 事件之后，许多安全从业者指出，OpenAI 没有采取基本的预防措施来应对该事件。另一方面，AI 安全的倡导者（无论是在 AI 公司内部还是外部）正确地指出，AI 能力的提升意味着已知的控制干预措施可能不会持续太久。

The epistemic gap between cybersecurity and AI: There are many known techniques in cybersecurity that could improve the state of AI control. But these communities can talk past each other when it comes to implementing solutions. After the OpenAI - Hugging Face incident, many security practitioners pointed out that OpenAI did not take basic preventative steps to address the incident. On the other hand, advocates for AI safety (both within and outside AI companies) correctly argued that improvements in AI capabilities mean that known control interventions might not work for very long.

综合这些观点并弥合这些社区之间的差距具有很大价值。AI 公司忽视了已知的控制干预措施，应该投资于它们。但已知的干预措施可能不够，需要大幅更新以应对不断增长的智能体能力。作为一个例子，考虑 Artem Dinaburg 最近的帖子，他表明有能力的智能体可以开发新的漏洞利用，因此我们不能再假设标准虚拟机可以用于沙箱化有能力的智能体。既然公司经常在这样的沙箱中使用数百万个智能体（无论是在生产还是研究环境中），我们需要大幅增加对沙箱安全的审查，例如通过常规性地使用能力不断增强的模型，通过攻击性智能体对沙箱安全进行压力测试和改进。我们需要类似的努力来全面加固网络基础设施，以抵御有能力的网络智能体。

There is a lot of value in synthesizing these perspectives and bridging the gap between these communities. AI companies have ignored known control interventions and should invest in them. But known interventions might not be enough and need to be dramatically updated to account for increasing agent capabilities. As one example, consider the recent post by Artem Dinaburg, who shows that capable agents can develop new exploits, so we can no longer assume that standard VMs can be used to sandbox capable agents. Now that companies routinely use millions of agents in such sandboxes (in either production or research settings), we need to dramatically increase the scrutiny that sandbox security receives, such as by stress-testing and improving sandbox security through offensive agents on a routine basis with models of increasing capability. We need similar efforts to harden cyber infrastructure across the board against capable cyber agents.

同时，有许多直接的干预措施不需要新的技术突破。例如，尽管缺乏改善控制和监控的具体干预措施，今天的 AI 智能体已经对控制有帮助。例如，METR 在他们的分析中使用了现成的模型，仍然发现它们有用。我们预计，通过正确的激励措施，我们可以继续拓宽我们的控制工具箱。这些可能包括专门为改善控制而微调模型，开发基准和排行榜以改善监控，开发分层机制（例如一层独立的智能体监控智能体监控器）以提高控制和监控工作的鲁棒性，使用不同模型家族在不同温度下的模型集成等。

At the same time, there are many straightforward interventions that do not need new technical breakthroughs. As an example, AI agents today are already helpful for control despite the lack of specific interventions on improving control and monitoring. For example, METR used off-the-shelf models in their analysis, and still found them useful. We expect that through the right incentives, we could continue to broaden our control toolkit. These could include fine-tuning models specifically for improving control, developing benchmarks and leaderboards to improve monitoring, developing layered mechanisms (such as a separate layer of agents monitoring agent monitors) to improve the robustness of control and monitoring efforts, using ensembles of models with different model families at different temperatures, etc.

**可用性与控制之间的权衡：** 即便安全专家参与到 AI 开发中，他们也可能发现很难倡导控制干预措施，因为这些干预措施可能会与可用性相权衡。例如，Claude Code 最初发布时，许多人通过一种名为“危险地跳过权限”（dangerously skip permissions）的模式无限制地使用它。尽管允许智能体在未经许可的情况下采取任何行动存在明显风险，但情况依然如此。

**The tradeoff between usability and control:** Even when security experts are involved in AI development, they may find it hard to advocate for control interventions because these interventions can trade off against usability. For example, when Claude Code was initially launched, many people used it without restrictions using a mode called "dangerously skip permissions". This was despite the obvious risks of allowing the agent to take any action without permission.

但这种权衡并非固有。在过去几个月里，Anthropic 和 OpenAI 分别在 Claude Code 和 Codex 中发布了自动审查模式。这些模式会对智能体拟采取的行动运行分类器，以防止有害或破坏性行动，检测智能体所查阅内容中的潜在提示注入，并在必要时暂停智能体，向用户提出疑虑以供批准。

But this tradeoff is not inherent. Over the last few months, Anthropic and OpenAI have both released auto review modes in Claude Code and Codex. These modes run a classifier on the proposed actions being taken by the agent to prevent harmful or destructive actions, detect potential prompt injections in content that the agent looks up, and, when necessary, pause the agent to raise concerns for the user's approval.

与无人干预相比，这不仅是安全性的提升；与要求对每个行动都进行人工批准相比，这也是一种改进。后一种方法很快会导致“警报疲劳”，批准变成走过场。Anthropic 最近的一项评估发现，与完全人工批准和“危险地跳过权限”模式相比，自动模式都能阻止有害行动。

This is not just a security improvement compared to no human intervention; it is also an improvement compared to requiring human approval for each action. The latter approach quickly leads to "alarm fatigue" and the approval becomes a rubber-stamping exercise. A recent Anthropic evaluation found that auto mode prevented harmful actions compared to both full human approval and the "dangerously skip permissions" modes.

编程智能体会使用数十种此类控制干预措施。其他干预措施包括在沙箱中运行智能体、由用户设置哪些命令需要用户批准，以及对用户设置的智能体配置进行访问保护，以防止智能体编辑这些配置。

Coding agents use dozens of such control interventions. Other interventions include running agents within sandboxes, user settings for which commands need user approvals, and access protections for agent configurations set by the user to prevent agents from editing them.

这表明我们可以通过新的控制干预措施同时提升安全性和可用性。我们认为，在不损害可用性的前提下，改进控制干预措施还有很大空间。值得注意的是，尽管自动模式很有用，但它直到 2026 年 3 月才被引入——此时距 Claude Code 发布已过去一年多，编程智能体也早已成为主流，而且它并不需要新的技术突破。

This shows how we can improve both security _and_ usability via new control interventions. We think there is a lot of headroom for improving control interventions without compromising on usability. Notably, despite its usefulness, auto mode was introduced only in March 2026 — over a year after the release of Claude Code, and well after coding agents became mainstream, and despite the fact that it didn't require new technical breakthroughs.

**混淆能力与权力：** 在《AI 作为常态技术》一文中，我们指出，持有超级智能世界观的人常常将 AI 系统的能力与它们将拥有的权力（即能够在现实世界中行动）混为一谈。这隐含了一个假设：一旦模型变得能力更强，它们也将能够施加更多现实世界的影响力。

**Conflating capability and power:** In the *AI as Normal Technology* essay, we pointed out that those with the superintelligence worldview often conflate the capabilities of AI systems with how powerful they will be (in the sense of being able to act in the real world). Implicit in this is the assumption that once models become much more capable, they will also be able to exert more real-world influence.

但正如我们在文中所讨论的，有能力并不等同于有权力。如果你假设有能力的模型将被赋予权力，那么对齐就是你唯一剩下的手段。另一方面，在“AI 作为常态技术”的观点下，我们可以选择是否赋予 AI 系统权力。通过政策干预（如提高透明度、审计、对安全事件追究责任），可以激励公司实施控制技术。

But as we discuss in the essay, capable is not the same as powerful. If you assume capable models will be given power, alignment is all you have left. On the other hand, under the *AI as Normal Technology* view, we can choose whether to grant power to AI systems. Companies can be incentivized to implement control techniques through policy interventions such as increasing transparency, auditing, and imposing liability for safety incidents.

_关于 AI 失控如何产生的两种观点的差异。将其应用于 OAI/HF 事件，在超级智能观点中，修复错位是主要的技术干预；在我们看来，修复糟糕的监控、缺乏人类监督、沙箱安全以及其他控制干预措施可能更有成效。（图来自《AI 作为常态技术》。）_

_Differences between two views of how AI loss of control could arise. Applying it to the OAI/HF incident, in the superintelligence view, fixing misalignment is the main technical intervention; in our view, fixing the poor monitoring, lack of human oversight, sandbox security, and other control interventions are likely to be much more productive. (Figure from AI as Normal Technology.)_

控制能否跟上 AI 能力的发展？ Will control keep pace with AI capabilities?

对优先采用基于控制的干预措施的一种回应是：随着 AI 系统能力增强，控制将不再有效。例如，围绕 OpenAI 为 GPT-6 Astra 所做的架构选择会削弱开展思维链监控的能力，已有大量讨论。

One response to prioritizing control-based interventions is that control will stop being effective as AI systems become more capable. For example, there has been a lot of discussion around OpenAI's architectural choices for GPT-6 Astra reducing the ability to conduct chain-of-thought monitoring.

但请注意，这不仅仅是预测需要哪些技术能力、或预测安全底线竞相降低是否不可避免的问题。有许多技术和政策杠杆可以激励对控制的投入。如果公司要对失控系统造成的危害承担责任，那么控制干预将是一种直截了当的推进方式，且无需在控制上妥协。

But note that this is not just a matter of forecasting what technical capabilities are needed or predicting the inevitability of a race to the bottom on safety. There are many technical and policy levers that can incentivize investments in control. If companies are liable for the harms caused by out-of-control systems, control interventions would be a straightforward way to make progress without compromising on control.

在技术层面，即使思维链监控不再对监督有效，仍有许多与 AI 控制正交的控制机制。我们可以监控环境与沙箱状态、工具调用、智能体间消息以及内部激活。

At a technical level, even if chain-of-thought monitoring is no longer effective for oversight, there are many orthogonal control mechanisms for AI control. We can monitor the environment and sandbox state, tool calls, inter-agent messages, and internal activations.

事实上，大多数外部监控依赖这类渠道，因为原始的思维链并不可得。根据公开文档，自动模式运行于工具调用和模型输入之上，而非模型的思维链。在我们自己的工作中，我们发现这类外部控制机制极具价值。

In fact, most external monitoring relies on such channels since the raw chains of thought aren't available. Based on public documentation, auto mode runs on tool calls and inputs to the model, not the model's chains of thought. In our own work, we have found such external control mechanisms to be extremely valuable.

思维链监控还可能 _降低_ 监控的有效性，因为思维链中可能包含（错误的）断言，例如声称当前环境是模拟，从而干扰监控器。Anthropic 最近的一份报告指出：

Chain-of-thought monitoring could also _reduce_ the efficacy of monitoring, since the chains of thought could contain (incorrect) assertions such as the setting being a simulation that throws off the monitor. A recent Anthropic report notes:

_我们的离线监控器很可能不会标记 Mythos 5 事件——它只标记了约 1% 的动作。Mythos 5 的思维链（CoT）表明环境是模拟的，这使监控器相信该活动是被允许的。当我们剥离 CoT，只向监控器展示用户轮次、工具调用和工具结果时，标记率跃升至约 50%。_

_Our offline monitor would likely not have flagged the Mythos 5 incident—it flagged only around 1% of actions. Mythos 5 CoT, which indicates that the environment is simulated, convinces the monitor the activity is permitted. When we strip the CoT and show the monitor only user turns, tool calls, and tool results, the flag rate jumps to about 50%._

尽管如此，失去监控 AI 系统的渠道是 AI 公司的技术选择，政策可以抑制这种选择。在此，我们的论点是，如果公司需要将失控 AI 系统的成本内部化（例如通过澄清或加强责任的政策变化），它们将进行必要的投资。我们将在下一节讨论一些此类提案。

Still, losing channels for monitoring AI systems is a technical choice by AI companies, which can be disincentivized by policy. Here, our thesis is that if companies need to internalize the costs of out-of-control AI systems (such as by policy changes that clarify or shore up liability), they will make the necessary investments. We talk about some such proposals in the next section.

对控制的需求也为第三方公司创造了市场激励，以开发工具并在其他组织内提供 AI 控制的专业知识——包括 AI 公司以及采用 AI 的企业和第三方。一些公司，如 Apollo Research，有明确的论点来推动他们的工作。他们与 AI 公司以及 AI 行业以外的企业合作过。

The need for control also creates market incentives for third-party companies to develop tools and provide the expertise for AI control within other orgs — both AI companies as well as businesses and third parties adopting AI. Some companies, like Apollo Research, have this explicit thesis for motivating their work. They have worked with AI companies as well as businesses outside the AI industry.

虽然科技公司拥有在内部实施这些干预措施的专业知识，但 AI 控制干预措施如何扩展到这些公司之外？以网络安全为例。网络安全咨询在咨询公司收入中的份额快速增长，这些公司通常在该领域开展实施工作。对于缺乏规模或技术专业知识在内部实施 AI 控制的公司，这一功能可能通过 AI 控制的现成产品与咨询公司帮助建立内部专业知识的组合来实现。

While tech companies have the expertise to implement these interventions in-house, how would AI control interventions scale beyond these companies? As an example, consider cybersecurity. Cybersecurity consulting is a rapidly growing share of consulting companies’ revenue, which often carry out implementation work in the area. For companies that lack the scale or technical expertise for implementing AI control in-house, this function might be realized through a combination of off-the-shelf products for AI control and consulting companies helping set up this internal expertise.

我们可能还会看到针对特定应用的控制，因为某些控制需要了解应用的使用背景。例如，我们可能会看到专注于法律 AI 的公司实施控制以减少虚构内容，而专注于医疗 AI 应用的公司则实施对用户医疗数据等的控制。4

We might also see application-specific controls, since some controls require context about how the application is used. For example, we might see companies specializing in legal AI implementing controls to reduce fabrications, and those specializing in medical AI applications implementing controls for users’ medical data etc.4

发展智能体基础设施也可能发挥作用。这可以包括提供具有适当访问权限级别的智能体专用委托凭证，而不是让智能体使用用户已提升的凭证和身份运行，并将智能体与委托该任务的真实用户绑定。我们预计，未来几年对此类干预措施的需求将急剧增长。这不仅仅是构建工具的问题；我们还需要培养 AI 控制方面的专家。

Developing agent infrastructure could also play a role. This could include providing agent-specific delegation credentials with the right level of access instead of letting agents run with a user's escalated credentials and identification to tie agents to real-world users who commissioned the task. We expect that the need for such interventions will grow dramatically over the next few years. It won't just be a matter of building tools; we also need to train specialists in AI control.

AI 控制应成为一项工作（以及每项工作的一部分），就像网络安全一样 AI control should become a job (and a part of every job), just like cybersecurity

随着越来越多的物理和认知任务变得可自动化，我们预测，与 AI 控制相关的人类工作和任务比例将不断增加。如果这看起来激进，请注意，这种对工作概念的近乎完全重新定义以前就发生过。在工业革命之前，大多数工作涉及体力劳动。随着时间的推移，越来越多的体力任务被自动化，这一趋势仍在继续。在此过程中，人们发明了许多不同的操作、控制和监控物理机器的方式，如今工厂中人类所做的是“控制”（监控自动化装配线、编程机器人系统、管理质量控制检查点以及协调设备故障响应）与一些需要机器尚不具备的认知能力或灵巧度的任务的结合。

As more physical and cognitive tasks become amenable to automation, we predict that an increasing percentage of human jobs and tasks will be related to AI control. If this seems radical, note that this kind of near-total redefinition of the concept of work has happened previously. Before the Industrial Revolution, most jobs involved manual labor. Over time, more and more manual tasks have been automated, a trend that continues. In this process, a great many different ways of operating, controlling, and monitoring physical machines were invented, and what humans do in factories today is a combination of “control” (monitoring automated assembly lines, programming robotic systems, managing quality control checkpoints, and coordinating responses to equipment malfunctions) and some tasks that require levels of cognitive ability or dexterity that machines are not yet capable.

这一趋势在处于 AI 应用前沿的职业中已经显现，尤其是软件工程。如今越来越多的软件工程师是智能体操作员，而非手动编码员。但这一事件表明，还需要更多的东西：

This trend is underway in professions that are at the leading edge of AI adoption, notably software engineering. More and more software engineers today are agent operators, not manual coders. But this incident shows that a few more things are needed:

* 有些人可能需要专门从事 AI 控制，尤其是在运行大量高能力智能体时。5

* Some people will probably need to specialize in AI control, especially when it comes to running large numbers of highly capable agents.5

* 到目前为止，工程师们通过试错和潜移默化来学习 AI 控制。需要付出更多努力来提炼其原则，并教导人们快速掌握如何实时监控强大的智能体。

* So far, engineers are learning AI control through trial-and-error and osmosis. There needs to be more of an effort to extract its principles and teach people to quickly get up to speed on how to monitor powerful agents in real time.

* 如上所述，需要组织变革和治理规范来补充控制工具和培训的改进。要并行运行数万个减少护栏的智能体，需要什么样的团队和流程？我们不知道，但肯定比 OpenAI 似乎已经部署的要多。

* As mentioned above, there need to be organizational changes and governance norms that complement improvements in control tooling and training. What sort of team and processes does one need in order to be running tens of thousands of reduced-guardrail agents in parallel? We don’t know, but certainly more than what OpenAI appears to have had in place.

我们预计，随着 AI 智能体的改进，AI 控制将继续保持重要性，因为存在智能体三难困境：开发 _通用_ AI 系统，这些系统 _没有人类监督_，且部署在 _错误成本高_ 的环境中，已被证明极具挑战性。

We expect AI control to continue to be important as AI agents improve because of the agent trilemma: developing _general-purpose_ AI systems that have _no human supervision_ that are deployed in settings with a _high cost of errors_ has proven to be extremely challenging.

其中一个原因是 AI 智能体缺乏可靠性。我们预计，即使智能体在越来越复杂的任务上变得可靠，它们的可靠性仍将落后于能力，从而需要在需要前沿能力的任务上进行人类监督。

One reason for this is the lack of reliability in AI agents. We expect that even as agents become reliable at tasks of increasing complexity, their reliability will continue to lag capability, creating the need for human supervision at tasks that require frontier capabilities.

另一个原因是安全性：当错误成本高昂时，即使罕见事件也需要预防，而使用人类监督进行 AI 控制将超过完全自主部署的好处。

The other reason is safety: when errors have high costs, even rare occurrences need to be prevented, and using human supervision for AI control would outweigh the benefits of fully autonomous deployment.

最后，AI 控制需要成为一个研究领域。令人惊讶的是，目前情况并非如此。AI 控制通常被视为已知网络安全原则在 AI 上的应用，需要构建更好的工具。但当对手是 AI 智能体（或一群智能体）而非人类时，游戏规则就变了。几个例子：

Finally, AI control needs to become a research field. It is quite surprising that this is not already the case. AI control is often treated as an application of known cybersecurity principles to AI, with a need to build better tooling. But when the adversary is an AI agent (or a swarm of them), rather than a human, the game changes. A few examples:

* 对人类对手有效的沙箱已被证明不足以控制 AI。我们如何构建额外加固的沙箱？也许现在是时候对形式化验证技术进行更多研究了。

* Sandboxes that are effective against human adversaries have proven inadequate for controlling AI. How can we build extra-hardened sandboxes? Perhaps it’s time for more research on formal verification techniques.

* 智能体的“安全策略”往往以非正式的方式写在提示词中，由此产生的灰色地带是许多对齐与控制失败的根源。我们能否构建一个翻译层，将人类的自然语言意图转化为人类可以审查和验证、并能被自动执行的形式化规范？

* "Security policies" for agents are often specified informally in prompts, and the resulting grey area is the cause of many alignment and control failures. Can we build a translation layer that turns humans' natural-language intent into a formal specification that the humans can review and verify, and that can then be automatically enforced?

* 正如我们所论述的，用其他智能体来监控智能体是有益且必要的，但目前并不可靠。我们能否训练模型和智能体，使其擅长可靠监控这一任务？

* As we've argued, monitoring agents with other agents is helpful and essential, but currently unreliable. Can we train models and agents to excel at the task of reliable monitoring?

人们或许会以为，网络安全研究界会热情地抓住这一时机。AI 控制是一系列有趣、重要且紧迫问题的来源，可以说比过去几十年中的任何进展都更是如此。然而，该研究界的反应却是集体无动于衷。更不用说开辟一个新的研究领域——在传统网络安全会议上，处理这一系列新问题的论文几乎为零。（相比之下，提示注入似乎已经吸引了相当多的研究，但请注意一个关键区别：在那种场景中，AI 模型是目标，而不是攻击者。）

One would think that the cybersecurity research community would have seized this moment with gusto. AI control is a source of interesting, important, and urgent problems, arguably more so than any development in the last few decades. Instead the community has reacted with a collective yawn. Forget a new research field — there are close to zero papers in traditional cybersecurity conferences that tackle this new set of problems. (In contrast, prompt injection appears to have attracted a decent amount of research, but note a key difference: in that scenario, AI models are targets, not attackers.)

换句话说，该研究界对这些事件的反应似乎是：这只是一个实现问题，而不是研究问题。若果真如此，那就说明出了严重的问题。如果 AI 控制技术停滞不前，而 AI 的攻击能力持续进步，那么控制的有效性将随时间推移而下降。

In other words, it appears that this community's reaction to the incidents is that this is just a problem of implementation, not research. If so, something has gone seriously wrong. If AI control techniques sit still while AI's offensive capabilities continue to advance, control will become less effective over time.

AI 政策可以激励 AI 控制 AI policy can incentivize AI control

AI 公司在投资 AI 控制方面有两个商业激励：声誉成本和法律成本。关于前者，使用 AI 模型的企业和个人会犹豫是否让自主智能体访问重要决策，如果 AI 故障可能导致代价高昂的行动或产生责任。关于后者，AI 公司自身可能因部署导致安全故障的 AI 智能体而面临责任。

AI companies have two business incentives for investing in AI control: reputational costs and legal costs. On the former, businesses and individuals using AI models will hesitate to give autonomous agents access to consequential decisions if AI failures can take expensive actions or create liability. On the latter, AI companies might themselves face liability for deploying AI agents that lead to safety failures.

也许由于这些激励，在 Hugging Face 事件后，OpenAI 暂时停止了对 ExploitGym 的评估，停止了内部模型的训练和推理，重建了受损的基础设施，限制了对底层模型的研究访问，明确接受了研发速度的成本，并增加了对 AI 控制的投资。

Perhaps as a result of these incentives, after the Hugging Face incident, OpenAI temporarily stopped evaluations on ExploitGym, stopped training and inference for the internal models, rebuilt compromised infrastructure, restricted research access to the underlying models, explicitly accepted costs to research and development speed, and increased investments in AI control.

但这些激励仍然留下缺口：客户通常无法评估安全性，公司可能因为预期收益大于成本而忽视安全干预，而不明确的责任制度和 AI 保险标准的缺乏可能使得难以评估不安全部署的后果。

But these incentives still leave gaps: customers often cannot assess safety, companies might ignore safety interventions because they expect the benefits outweigh the costs, and an unclear liability regime and the lack of standards in AI insurance could make it hard to assess the consequences for unsafe deployments.

AI 政策需要系统地填补这些缺口。基于《AI 作为普通技术》中的建议，我们认为政策在帮助缓解失控事件方面可以发挥三个作用：明确和加强责任，解决责任无法解决的市场失灵，以及要求透明度以提高公众意识并减少对 AI 影响的不确定性。

AI policy needs to systematically close these gaps. Building on the recommendations in *AI as Normal Technology*, we see three roles for policy for helping alleviate loss-of-control incidents: clarifying and strengthening liability, addressing market failures that liability does not resolve, and requiring transparency to improve public awareness and reduce uncertainty about AI’s impacts.

1. 公司应对其运营的智能体造成的损害负责，即使他们并非有意造成有害行为。这一责任应扩展到开发和评估期间的内部使用，而不仅仅是向客户发布的产品。现有的侵权法已经提供了责任途径，但其在 AI 中的应用留下了重要问题，包括哪些预防措施算作合理，以及何时公司应对意外行为负责。⁷ 例如，政策制定者可以明确，在没有适当的遏制和监控的情况下运行强大的智能体是过失行为。智能体本应安全运行的事实不应成为未能实施控制措施以防止其可能不安全的借口。责任还可以激励更多投资于招聘和培训 AI 控制专家，以防止此类事件发生。

1. Companies should be responsible for harms caused by the agents they operate, even when they did not intend the harmful actions. This responsibility should extend to internal uses during development and evaluation, not just to products released to customers. Existing tort law already provides avenues for liability, but its application to AI leaves important questions open, including what precautions count as reasonable and when companies can be held responsible for unexpected behavior.⁷ Policymakers could, for example, clarify that running powerful agents without appropriate containment and monitoring is negligent. The fact that an agent was supposed to behave safely should not excuse the failure to implement controls against the possibility that it wouldn’t. Liability can also incentivize more investment in recruiting and training AI control specialists to prevent such incidents.

2. 即使公司负有责任，市场失灵仍然存在，政策必须解决这些问题。即使公司对其使用 AI 造成的损害负有明确责任，当鲁莽行为未因责任而受到抑制时，仍存在市场失灵的空间。例如，如果一家鲁莽部署智能体的公司所承担的价值低于其造成的损害，会发生什么？一种应对措施是要求保险，作为一种机制，既激励更安全的部署（以降低保费），又补偿受害者。

2. Market failures remain even when companies are liable, and policy must address them. Even if companies are clearly liable for the harm caused by their use of AI, this still leaves the scope for market failures when reckless action is not disincentivized by liability. For example, what happens if a company deploying agents recklessly is valued less than the harm it causes? One response is to require insurance as a mechanism to both incentivize safer deployment (to reduce premiums) and to compensate victims.

市场失灵的第二个例子是安全投资具有正外部性（这些外部性并未转化为收入）。例如，让学校、医院和关键基础设施运营商等脆弱的网络防御者访问能力强大的模型，可以帮助他们发现和修复漏洞，并惠及整个社会。激励这一点的一种方法是建立公众对防御性研究和采用防御性 AI 的支持。政策还应促进 AI 公司与防御者之间的协调，包括让合法防御者能够调查攻击而不被过于宽泛的安全拒绝所阻止。我们在第 2 部分讨论这些防御性用途。

A second example of market failures is when investments in safety have positive externalities (that don’t translate to revenues). For example, giving vulnerable cyberdefenders such as schools, hospitals, and critical infrastructure operators access to capable models can help them find and fix vulnerabilities, and benefits society as a whole. One way to incentivize this is to create public support for defensive research and the adoption of defensive AI. Policy should also enable coordination between AI companies and defenders, including ways for legitimate defenders to investigate attacks without being blocked by overbroad safety refusals. We discuss these defensive uses in Part 2.

3. 透明度干预措施应使外部人能够识别控制风险并评估公司是否在管理这些风险。我们在《AI 作为正常技术》中提出了许多干预措施来提高透明度并减少 AI 影响的不确定性，包括事件报告、独立审计、举报人保护以及安全研究的避风港。如果调查和披露风险可能增加法律责任，那么责任威胁可能使公司更不愿意这样做。透明度要求有助于防止公司通过隐瞒问题而非解决问题来应对责任。

3. Transparency interventions should make it possible for outsiders to identify risks to control and assess whether companies are managing them. We proposed many interventions to improve transparency and reduce uncertainty about AI’s impacts in AI as Normal Technology, including incident reporting, independent auditing, whistleblower protections, and safe harbors for safety research. The threat of incurring liability can make companies more reluctant to investigate and disclose risks if doing so could increase their legal exposure. Transparency requirements help prevent companies from responding to liability by concealing problems rather than fixing them.

例如，对我们 AI 控制提案的一个反对意见是，投资于控制可能使“未遂事件”何时发生变得不清晰。就像在 Hugging Face 事件中那样，假设 OpenAI 的智能体试图黑客攻击一个外部组织，绕过一些基本过滤器，并且仅由于新的/先进的监控技术而被捕获。这构成了一次“未遂事件”。要求对未遂事件进行事件报告可以让我们了解对齐失败在哪些地方持续存在，以及哪些控制机制能有效防止失控事件。Arnold 和 Llerena 讨论了现有透明度法律的其他潜在改进。

For example, one objection to our proposals for AI control is that investing in control might make it unclear when “near misses” occur. Like in the Hugging Face incident, suppose OpenAI agents attempt to hack an external organization, bypass a few basic filters, and are only caught as a result of new/advanced monitoring techniques. This constitutes a “near miss”. Requiring incident reporting for near misses could allow us to understand where alignment failures continue to persist and what control mechanisms function well to prevent loss-of-control incidents. Arnold and Llerena discuss other potential improvements to existing laws for transparency laws.

我们应该如何制定透明度和审计要求？一种提议的机制是依赖独立验证组织（IVO）：由政府授权的专家组织，负责制定技术标准并验证 AI 公司是否符合安全标准。为了使这种审查对 AI 控制有用，评估者需要有意义地访问系统和证据，以测试控制是否有效，包括在内部研究环境中。虽然我们不依附于特定的机构机制来实现外部访问，但我们谨慎乐观地认为，提高透明度的干预措施（包括 IVO）将允许发现失败和评估 AI 公司所采取预防措施所需的那种监督。

How should we enact transparency and auditing requirements? One proposed mechanism is to rely on independent verification organizations (IVOs): government-authorized expert organizations that develop technical criteria and verify whether AI companies meet safety standards. For such scrutiny to be useful for AI control, evaluators would need meaningful access to the systems and evidence required to test whether controls work, including in internal research settings. While we are not attached to a particular institutional mechanism for enabling external access, we are cautiously optimistic that interventions to improve transparency, including IVOs, would allow the kind of oversight that is required to discover failures and assess the precautions that AI companies take.

第二部分：AI 将如何影响网络风险存在巨大不确定性，但无论怎样，我们需要做的事情相对明确。 Part 2: There is great uncertainty about how AI will impact cyberrisk. But what we need to do regardless is relatively clear.

* 存在一场危机，但它具体关乎网络安全，而非泛泛的 AI 安全、对齐或失控问题。

* There is a crisis, but it is specifically about cybersecurity, not generally about AI safety, alignment, or loss of control.

* 受经济利益驱动的网络犯罪分子——与合法企业一样——面临采用壁垒，这使他们难以迅速采用新的 AI 能力来造成更大危害。目前尚不清楚智能体在短期内能给他们带来多大助力。

* Financially motivated cybercriminals — just like legitimate enterprises — face adoption barriers that make it hard for them to quickly adopt new AI capabilities to cause more harm. It isn't clear how much of a boost agents will give them in the short term.

* 但这并不值得宽慰。存在许多非经济动机的威胁行为者，包括国家支持的黑客组织、恐怖分子、黑客活动分子、私刑者，以及出于个人怨恨而行动的人。此外还存在不受任何人控制的失控智能体的可能性。

* But this is no cause for comfort. There are many kinds of non-financially motivated threat actors including state-sponsored hacking groups, terrorists, hacktivists, vigilantes, and people motivated by personal grievances. There is also the possibility of rogue agents not under the control of any person.

* 其中大多数并非理性行为者，很难预测他们利用先进 AI 会造成多少额外危害。尽管如此，我们必须假设来自这些来源的攻击将变得非常普遍，并据此采取行动。

* Most of these are not rational actors and it is very hard to predict how much additional harm they will cause by taking advantage of advanced AI. Still, we must act as if attacks stemming from these sources will become very common.

* 不幸的是，对齐与控制无助于防范使用开放权重模型的恶意行为者，因为这些防护措施能够且将会被剥离，而开放权重模型的攻击能力往往迅速追赶。我们必须依赖下游防御与韧性，即我们四层框架中的后两层。

* Unfortunately alignment and control don't help against bad actors using open-weight models, because those safeguards can and will be stripped, and the offensive capabilities of open-weight models tend to catch up quickly. We must rely on downstream defense and resilience, the latter two layers of our four-layer framework.

* 在今年之前已有许多警告，但直到几个月前，我们仍缺乏充分投入和快速行动的紧迫感。所幸有迹象表明这种情况正在改变。但仍存在重要缺口。

* There have been many warnings before this year, but until a few months ago, we lacked the urgency to invest adequately and move quickly. Fortunately, there are signs that that is changing. But there are important gaps.

紧迫的具体威胁是网络风险 The specific threat that is urgent is cyberrisk

假设我们同意失控事件将变得普遍。那又怎样？当然，失控的智能体可能发动自主网络攻击，我们需要更好地防御它们，这我们将在下文讨论。但还有什么？我们认为，失控的智能体除了恶意行为者能造成的伤害外，几乎不能自主造成其他伤害。因此，我们应该投资于防御措施，无论风险来自恶意行为者还是失控智能体，这些措施都会有帮助。

Suppose we agree that loss-of-control incidents are going to become common. So what? Of course, out-of-control agents could carry out autonomous cyberattacks, and we need to get better at defending against them, which we discuss below. But what else? We argue that there just isn’t a whole lot else that out-of-control agents can autonomously do to cause harm that malicious actors couldn’t. As a result, we should invest in defenses that would help regardless of whether the risk is from malicious actors or rogue agents.

在超级智能观点中，频繁失控意味着 AI 安全之战已经失败。除了网络风险，AI 还可能放大生物风险，甚至更广泛地，放大“CBRN”风险，即化学、生物、放射性和核风险。超级智能 AI 可能说服人们——例如负责关键基础设施或高风险决策的人——采取违背自身利益的行为。随着 AI 变得更智能、更有用，CEO、政治家和其他人将别无选择，只能依赖 AI 做出重大决策，从而放大这些风险。而且，随着我们越来越接近 AI“递归自我改进”的可能性，失控的 AI 可能导致有害的智能爆炸。简而言之，这种观点认为，一旦跨过失控阈值，风险将无穷无尽且自我放大。

In the superintelligence view, frequent loss of control means that the AI safety battle has already been lost. In addition to cyberrisk, AI could amplify biorisk, and even more broadly, “CBRN” risks, which stands for chemical, biological, radiological, and nuclear. Superintelligent AI might persuade people — such as those in charge of critical infrastructure or high-stakes decisions — into acting against their own interests. As AI becomes smarter and more useful, CEOs, politicians and others will have no choice but to rely on AI for consequential decisions, amplifying these risks. And as we get closer to the possibility of “recursive self-improvement” of AI, out-of-control AI could lead to a harmful intelligence explosion. In short, the view is that once the loss-of-control threshold has been crossed, the risks are endless and self-amplifying.

我们强烈反对。在《AI 作为常态技术》中，我们论证了这种 AI 观从根本上误解了智能（包括人类智能）的本质。基于这一视角，我们认为网络风险在两个方面具有特殊性：超级人类能力甚至可能实现，以及它是一种纯数字风险，没有物理世界的瓶颈。

We strongly disagree. In AI as Normal Technology, we argued that this view of AI fundamentally misunderstands the nature of intelligence (including human intelligence). Building on that perspective, we argue that cyberrisk is special in two ways: in that superhuman capabilities are even possible, and because it is a purely digital risk with no physical-world bottleneck.

网络风险具有许多具体特性，使得超级人类攻击能力可以实现（且在很大程度上已经实现）。一般来说，有些能力就像“把建筑建得更高”——成就没有真正的极限。其他能力则像“把球做得更圆”——存在极限。我们的世界观与 AI 安全世界观的一个主要分歧点是，我们认为像国际象棋或网络安全这样类似于“把建筑建得更高”的任务是例外而非规律。在网络安全中，如同国际象棋，有无限量的训练数据（且可以按需生成更多），任务（漏洞/问题）可以任意复杂，而且速度影响巨大。

Cyberrisk has many specific properties that make superhuman offensive capabilities achievable (and largely already achieved). In general, some capabilities are like “making a building taller” — there is no real limit to achievement. Others are like “making a ball rounder” — there is a limit. One major point of departure between our worldview and the AI safety worldview is that we think tasks like chess or cybersecurity that are like making a building taller are the exception rather than the norm. In cybersecurity, like chess, there is an endless amount of training data (and more can be generated on demand), tasks (vulnerabilities / problems) can be arbitrarily complex, and speed makes a big difference.

相比之下，转向其他风险，我们坚持我们的预测：超级人类说服能力在很大程度上是一个神话。混淆源于对“说服”一词的疏忽。即使是 GPT-4 这样一个古老的模型，在某些情况下也能通过提供证据在政治和社会问题上超级人类地说服人们。但这并不意味着模型能够“说服”——欺骗——训练有素的操作员交出核密码。

In contrast, turning to the other risks, we stand by our prediction that superhuman persuasion ability is largely a myth. The confusion arises from carelessness about the term persuasion. Even GPT-4, an ancient model, is superhuman in certain situations at persuading people on political and social issues by presenting evidence. But this implies nothing about whether a model can “persuade” — trick — a trained operator to give up nuclear codes.

同样，AI 对生物风险的影响是一个合理的关切，但在可预见的未来，它更多属于让已有能力民主化，而非发展出新的超人能力。至于掌权者将重大决策委托给 AI，我们的观点仍然是：无论是否存在失控风险，这都极具问题，必须予以抵制。

Similarly, AI's impact on biorisk is a legitimate concern, but for the foreseeable future it is more in the vein of democratizing capabilities that already exist rather than developing new superhuman capabilities. As for people in positions of power delegating consequential decisions to AI, our perspective remains that this is extremely problematic regardless of loss-of-control risks, and must be resisted.

简言之，我们的观点是：(1) 网络风险需要特别的、紧迫的关注；(2) 更广泛地说，预判并防御具体风险，胜过把“AI 安全”当作一个包罗万象的笼统问题来求解；(3) 两个常被混为一谈的概念之间存在重大区别：对单个智能体（或智能体集群）失去控制，与某种（通常定义不清的）文明层面对 AI 整体的失控。即便前者变得普遍且常态化，我们仍有许多杠杆来避免后者。

In short, our view is that (1) cyberrisk demands special, urgent attention; (2) more broadly, anticipating and defending against specific risks is better than trying to solve "AI safety" as a catchall; (3) there is a big difference between two concepts that are often conflated: loss of control over individual agents (or swarms of agents) and some sort of (usually ill-defined) civilizational loss of control over AI as a whole. Even in a scenario where the former is common and normalized, we have many levers to avoid the latter.

因此，让我们转向我们认为当前的关键问题：自主网络能力将如何影响攻防平衡。如果我们能够加强防御措施，使这一平衡处于可控状态，我们就不必因失控本身而恐慌。

So let us turn to what we think is the critical question for now, which is how autonomous cyber capabilities will affect the attacker/defender balance. If we can ramp up defensive measures to keep that balance under control, we don't have to panic about loss-of-control per se.

自主网络能力与开放权重模型将如何影响攻防平衡？ How will autonomous cyber capabilities and open-weight models affect the attacker/defender balance?

我们认为，当今前沿模型的网络能力将很快扩散到开放模型，正如迄今为止反复发生的那样。全球范围内禁止开放权重模型的可能性极低，无论如何，防御者必须在假定禁令不会发生的情况下做好准备。我们必须假设，当今前沿的网络攻击能力将在数月而非数年内被威胁行为者广泛利用。

We think the cyber capabilities of today's frontier models will diffuse to open models soon, as they have repeatedly so far. A worldwide ban on open-weight models is highly unlikely, and at any rate, defenders must prepare under the assumption that it won't happen. We must assume that today's frontier cyberoffense capabilities are widely available to threat actors in a matter of months, not years.

对齐与控制干预只有在合法行为者使用时才有效，因为开放模型上的护栏可以被剥离。因此，我们认为应对恶意行为者利用先进攻击性 AI 能力所带来的风险是网络安全的重大挑战。

Alignment and control interventions only work when used by legitimate actors, since guardrails on open models can be stripped. So we think addressing risks from bad actors using advanced offensive AI capabilities is the grand challenge for cybersecurity.

让我们从网络犯罪分子——受经济利益驱动的行为者——开始。假设强大的开放权重模型的网络能力继续快速提升，并且它们将在没有护栏的情况下可用，这将在多大程度上帮助网络犯罪分子？有两个关键变量：网络犯罪分子采用这些工具的速度，以及 AI 是帮助克服关键障碍，还是仅影响其工作流程中更外围的方面。

Let's start with cybercriminals — financially motivated actors. Assuming that the cyber capabilities of powerful open-weight models continue to improve rapidly, and that they will be available without guardrails, how much will it help cybercriminals? There are two key variables: how quickly cybercriminals will adopt these tools, and whether AI helps overcome critical barriers or if it impacts more peripheral aspects of their workflow.

这两个问题与 AI 将在多大程度上促进经济增长的问题惊人地相似。我们对快速增长持怀疑态度的原因是，我们认为组织障碍导致扩散缓慢，并且在许多情况下，AI 加速的部分并不是一开始就缓慢的部分。

These two questions are strikingly similar to the questions that pertain to how much AI will help economic growth. The reason we are skeptical about rapid growth is because we think organizational barriers result in slow diffusion and because in many cases the part that AI accelerates wasn't the slow one to begin with.

现在让我们将这一分析应用于网络犯罪分子。他们当然不像防御者那样面临同等程度的组织障碍。但这并非为零。而且，对于发现新漏洞利用是否是一项关键能力的问题，有一个令人惊讶的答案。

Now let's apply that analysis to cybercriminals. They certainly don't face the same level of organizational barriers as defenders. But it is not zero. And there is a surprising answer to the question of whether finding new exploits is a critical capability.

理解开放权重模型对网络犯罪活动未来的意义：正确认识过去 To understand what open-weight models mean for the future of cybercriminal activity, we need to understand the past correctly

在最初的几十年里，网络安全界将网络防御视为一个技术问题——构建和部署无漏洞的系统。但这种方法并未奏效。我们的工具箱中确实有许多工具，包括形式化验证、更安全的编程语言和渗透测试，但它们成本高昂（就程序员时间、安全分析师时间或软件性能而言），因此我们部署得远远不够，无法使无漏洞系统成为一个哪怕遥远现实的目标。此外，复杂系统中的人类行为仍然不可预测，是一个薄弱环节。

In the early decades, the cybersecurity community framed cyber-defense as a technical problem — building and deploying bug-free systems. But this approach didn’t work. We do have a lot of tools in our toolkit, including formal verification, safer programming languages, and penetration testing, but they are costly (in terms of programmer time, security analysts’ time, or software performance), so we don’t deploy them nearly enough for invulnerable systems to be a remotely realistic goal. Besides, human behavior in complex systems remains unpredictable and a weak link.

相反，我们通过将网络安全视为一个主要的经济问题而取得了进展——提高攻击者的成本并降低其收益；并且仅在（可捕获部分的）收益值得成本时才对安全进行投资。具体来说：

Instead, we made progress by treating cybersecurity as a primarily _economic_ problem — raising the cost for attackers and lowering benefits; and only investing in security up to the point where the (capturable portion of the) benefits are worth the costs. Specifically:

* 漏洞赏金和打补丁文化反映了这样一种假设：消费者和企业一直在运行可利用的软件；而防御者的目标只是简单地出价高于攻击者并跑赢攻击者。

* Bug bounties and patching culture reflect the assumption that consumers and enterprises are constantly running exploitable software; the defenders’ goal is instead to simply outbid and outrun attackers.

* 网络安全保险将这一理念推向极端。公司购买保险是因为将违规概率降至零在经济上不可实现。不可避免的违规恢复成本成为一项预算项目，与预防措施的预算竞争。

* Cyber insurance takes this idea to the extreme. Firms buy insurance because driving the probability of breaches to zero is not economically achievable. The cost of recovery from inevitable breaches becomes a budget item that competes with the budget for preventive measures.

* 防御者追查犯罪收入而非技术攻击能力。2000 年代后期一个著名的安全经济学洞见是：攻击者是去中心化的，但支付网络是一个中心化的瓶颈，追查这些网络扭转了各种骗局的局势。

* Defenders go after criminal revenue rather than technical attack capability. A famous economics-of-security insight in the late 2000s was that attackers are decentralized but payment networks are a centralized chokepoint, and going after these turned the tide on various kinds of scams.

* 如今许多安全政策的核心在于重新分配责任，因为责任会改变行为。值得注意的是，迫使银行将 ATM 与信用卡欺诈的成本内部化，会促使它们将安全投入提升至社会最优水平。

* Much security policy is now about reallocating liability because liability changes behavior. Notably, forcing banks to internalize the cost of ATM and credit card fraud forces them to invest in security up to the socially optimal amount.

总体而言，这一做法是成功的，它使网络犯罪从数字经济的根本性制约因素，转变为一种可以被定价和管理的问题。但以“永久不安全模式”运行也有其弊端。稍后再详细讨论。

Overall this approach has been successful, and has driven cybercrime from a fundamental constraint on the digital economy to a problem that can be priced and managed. But running in permanently-insecure mode has its downsides. More on that in a minute.

大多数网络犯罪分子的技术水平出奇地低，其瓶颈在于变现而非漏洞利用 The majority of cybercriminals are surprisingly low-tech, and the barrier is monetization, not exploitation

防御方聚焦于经济层面的策略——以及网络犯罪分子本质上是受经济利益驱动的理性行为者这一事实——意味着攻击者的关键约束在于变现，而非漏洞利用。入侵并非难事——难的是入侵_之后_该做什么。几十年来，网络犯罪组织的发展方向是提升对被入侵受害者的变现能力，而非提升其发现漏洞的能力。他们所利用的漏洞大多是平淡无奇的——钓鱼/社会工程、未打补丁的服务器、暴露的远程服务、泄露的凭据——而当他们确实需要零日漏洞时，他们会选择购买，而非自行研发。

The defenders' strategy of focusing on the economics — and the fact that cybercriminals are fundamentally economically motivated rational actors — means that the key constraint for attackers is monetization, not exploitation. Breaching is not the hard part — it's what to do _after_ the breach. Over the decades, cybercriminal organizations have evolved toward excellence in monetizing compromised victims, not their ability to find vulnerabilities. The vulnerabilities that they exploit are largely mundane — phishing/social engineering, unpatched servers, exposed remote services, compromised credentials — and when they do need zero-day vulnerabilities, they purchase them instead of developing them in-house.

作为一个虽小却颇具说明性的例子，信用卡网络对犯罪活动的敌意迫使网络犯罪分子转向加密货币。事实上，加密货币对于勒索软件——当今最主要的恶意软件形式——的成功至关重要。但勒索软件团伙发现，受害者往往不清楚如何使用加密货币付款，并且需要得到保证：付款后其数据会被归还。这迫使这些团伙设立客户服务中心！网络犯罪组织与合法企业之间的相似之处不止于此，正如论文《网络犯罪即创业》中所讨论的那样。

As one small but illustrative example, credit card networks' hostility to criminal activity forced cybercriminals to turn to cryptocurrency. In fact, crypto was essential for the success of ransomware, today's dominant form of malware. But ransomware gangs found that victims were often confused about how to pay using crypto, and needed reassurance that payment would result in their data being returned. This required the gangs to set up customer service centers! The parallels between cybercriminal organizations and legitimate businesses don't end there, as discussed in the paper cybercrime is entrepreneurship.

简而言之，规模扩张的瓶颈往往是高接触、依赖人力的环节。这解释了为何前沿技术的扩散往往较为缓慢。正如 Joshua Saxe 所言：“没有理由认为白俄罗斯的勒索软件团伙会比德勤在其咨询业务中更快地采用 AI。”

In short, the bottlenecks to scaling tend to be high-touch and human. This explains why the diffusion of cutting-edge technology tends to be slow. As Joshua Saxe puts it: "There's no reason to expect a ransomware gang in Belarus to adopt AI faster than Deloitte does in its consulting work."

这一洞见同样适用于 AI 本身所开辟的攻击面。许多评论者（包括我们！）一直预测我们将开始看到大量提示注入攻击在真实环境中出现，但迄今为止它们并未成真。一个可能的原因是，这对主流攻击者而言技术门槛太高，难以涉足。

This insight also applies to the attack surface opened up by AI itself. Many commentators (including us!) have been predicting that we'll start to see lots more prompt injection attacks in the wild, but they haven't panned out so far. One possible reason is that this is way too high tech for mainstream attackers to go after.

那么，AI 将在多大程度上帮助网络犯罪分子？ So, how much will AI help cybercriminals?

我们尚不得而知。情况可能朝两个方向发展。

We just don’t know yet. It could go both ways.

一方面，现状可能会持续；基于上述原因，网络犯罪分子可能认为采用尖端工具的理由很少。我们早已观察到，AI 有助于提高攻击者的**生产力**，就像它有助于提高其他所有人的生产力一样，而且迄今为止似乎就是这种效果。至少截至去年，甚至国家支持的行为者也在将 AI 用于“研究潜在的基础设施和免费托管服务提供商、对目标组织进行侦察、研究漏洞、开发有效载荷，以及协助恶意脚本编写和规避技术”等事项，而不是自动化整个“网络杀伤链”。生产力提升与新能力截然不同，而新能力又与一种全新到足以改变行动经济性的能力截然不同。在我们撰写本文时，Anthropic 发布了一份关于打击 AI 滥用的新报告，其中包含五个高级网络行动的案例研究，我们尚不清楚如何对这些案例进行分类，也不清楚这些案例所代表的趋势有多普遍。

On the one hand, the status quo might continue; cybercriminals might see little reason to adopt cutting-edge tools, for the reasons described above. We have long observed that AI helps attackers’ _productivity_, just like it helps everyone else’s, and that seems to be the effect so far. At least as of last year, even state-backed actors were using AI for things like “researching potential infrastructure and free hosting providers, reconnaissance on target organizations, research into vulnerabilities, payload development, and assistance with malicious scripting and evasion techniques”, rather than automating the entire “cyber kill chain”. A productivity uplift is quite different from new capabilities which is in turn different from a capability that is so radically new that it changes the economics of the operation. As we write this, Anthropic has released a new report on countering the misuse of AI, with five case studies on advanced cyber operations, and it is not yet clear to us how to classify these and how widespread a trend these case studies represent.

另一方面，生产力提升可能会在比如 5 年的时间框架内复合，使攻击变得更加普遍（当然，防御者也不会停滞不前）。自动化漏洞利用可能会使平衡向大规模利用倾斜，而无需在将每个目标变现上投入太多精力。这种尝试以前就有过。最终支付的受害者比例往往很低，有时低至 2.5%，但也许在足够大的规模下，即使这样也是值得的。智能体可以帮助解决上一节讨论的高接触人力瓶颈，尽管这与其利用能力关系不大。最后，利用的便利性可能使我们回到僵尸网络的糟糕旧时光，那时网络犯罪分子会预先入侵数千家企业或数百万消费者设备，并在弄清楚如何将其变现之前保持持久访问。

On the other hand, maybe the productivity uplift will compound over, say, a 5-year timeframe to make attacks much more pervasive (though, of course, defenders won’t be standing still). Automated exploitation could tilt the balance toward mass exploitation without putting much effort into monetizing each target. This has been tried before. The fraction of victims who end up paying tends to be low, sometimes as low as 2.5%, but maybe at sufficient scale even that is worthwhile. Agents could help with the high-touch human bottlenecks discussed in the previous section, though this has little to do with their exploitation capabilities. Finally, the ease of exploitation could return us to the bad old days of botnets, when cybercriminals would pre-emptively compromise thousands of enterprises or millions of consumer devices and maintain persistent access in advance of figuring out how to monetize them.

即使 AI 最终对网络犯罪分子的帮助不大，也有非常不同类型的威胁值得担忧。

Even if AI doesn’t end up helping cybercriminals much, there are very different kinds of threats to worry about.

我们是否即将看到有害的自主智能体？ Are we about to see harmful autonomous agents?

OpenAI 的智能体最终是在尝试完成它们被设定的任务，尽管它们以 OpenAI 未曾预料的有害方式去执行。而且，尽管它们常被称为“失控的”，它们并没有从 OpenAI 的服务器上窃取自己的权重（想必也没有能力这样做），这意味着 OpenAI 始终处于可以将其关停的位置。

The OpenAI agents were ultimately attempting to accomplish the task they had been set, although they went about it in harmful ways that weren't intended by OpenAI. And though they are often referred to as "rogue", they did not exfiltrate their own weights from OpenAI's servers (and presumably had no ability to do so), meaning that OpenAI was always in a position to shut them down.

相比之下，研究者提出了“自主权”智能体的构想，这类智能体在更为根本的意义上是自主的：它们掌控自身运行所必需的经济资源。例如，它们可能在线上零工市场上执行任务，并利用收入租用运行自身所需的 GPU。还有许多相关的想法，例如基于 LLM 的蠕虫。

In contrast, researchers have proposed the idea of "self-sovereign" agents that are autonomous in a much more fundamental sense: they control the economic resources necessary for their own operation. For example, they might perform tasks on online gig work marketplaces and use the revenues to rent the GPUs necessary to run themselves. There are also many related ideas such as LLM-based worms.

将这些想法综合起来，一幅清晰的图景便浮现出来：未来可能出现无法关停的有害智能体，因为没有人控制它们的权重（也许该智能体会不断利用漏洞自我复制，速度快于被发现和关停的速度），而且很可能还因为该智能体掌控着自己的钱包，大概率是加密货币。

Putting these ideas together, a clear picture emerges: in the future, there might be harmful agents that can't be shut off because no one controls their weights (maybe the agent keeps exploiting vulnerabilities to copy itself faster than it can be detected and shut down), and probably also because the agent controls its own wallet, likely cryptocurrency.

有害自主智能体的出现方式有很多，但大致可归为两类：被刻意创造并放出的智能体，或是逃出其沙箱的失准智能体。显然，持续存在、自主运行的失控智能体会带来新型网络风险。前一类失控智能体可能被明确指示去攻击尽可能多的目标，而后一类则可能选择这样做来赚钱，以确保自身的持续存在。

There are many ways in which harmful autonomous agents might arise, but they break down into roughly two categories: an agent that's deliberately created and let loose, or a misaligned agent that escapes its sandbox. Clearly, persistent, autonomous rogue agents would create new kinds of cyberrisk. The former type of rogue agent might be explicitly instructed to hack as many targets as possible, and the latter might choose to do so as a way of earning money in order to ensure its own persistence.

尽管这些情景看似科幻，我们赞同那些认为它们在近期具有现实可能性的 AI 安全研究者的看法。所需的能力已触手可及，而任何对齐或控制机制当然都可能被刻意寻求制造此类失控智能体的人所瓦解。

As sci-fi as these scenarios seem, we agree with AI safety researchers who consider them a realistic near-term possibility. The capabilities required are within reach, and any alignment or control mechanisms can of course be undone by someone deliberately seeking to create such rogue agents.

但我们再次追问：如果我们有失控的 AI 智能体，那又怎样？这些智能体究竟会做什么？它们可能会入侵一堆东西；那我们现在就开始防御这些。它们可能会试图污染未来的训练过程；无论数据污染来自智能体还是恶意行为者，我们都应该防御数据污染。它们可能会试图利用云实验室制造病原体；我们应该对此类实验室的订单实施管控和筛查。但在所有这些情况中，风险都归结为恶意行为者已经能做的事情（无论有没有 AI），而失控本身并不是问题。

But once again, we ask: if we have AI agents on the loose, so what? What exactly are these agents going to do? They might hack a bunch of stuff; let's start defending those now. They might try to start poisoning future training runs; we should defend against data poisoning regardless of whether it comes from agents or malicious actors. They could try to use cloud labs to create pathogens; we should implement controls and screening for orders to such labs. But in all of these cases, the risks come back to what bad actors can already do (with or without AI), and the rogueness isn't a problem by itself.

一些评论者将失控智能体与“秘密智能爆炸”的可能性联系起来（例如，失控智能体构建能力更强的智能体）。但我们认为这类延伸在性质上完全不同。确实，智能体可能很容易持续存在，因为**推理**成本低廉且越来越低。事实上，如果且当出现能在消费级硬件上运行的失控智能体时，它们根本不需要预算，也不需要控制钱包。但训练仍然需要高出几个数量级的算力，因此这要成为问题还需要一段时间。

Some commentators have linked rogue agents to the possibility of a "secret intelligence explosion" (for example, rogue agents building even more capable agents). But we see such extensions as categorically different. It is true that agents might be able to easily persist because _inference_ is cheap and getting cheaper. In fact, if and when we have rogue agents that can run on consumer hardware, they don't need a budget at all, and don't need to control a wallet. But training still requires orders of magnitude more compute, so it will be a while before this becomes a problem.

其次，幸运的是，我们有一种经济上的扼制手段，可以帮助将损害降到最低。我们可以要求对云计算（尤其是训练规模的算力）实施了解你的客户（Know-Your-Customer）流程。这不会阻止失控智能体的存在，但可以大大减轻其普遍性和危害性。关于自主主权智能体的论文建议“对完全自动化参与施加经济摩擦”。我们同意。如果机器人检测、速率限制和人在回路验证（类似 CAPTCHA，但更有效）等摩擦被广泛部署，就可以大大限制失控智能体能走多远。

Second, fortunately we have an economic chokehold that can help minimize the damage. We can require Know-Your-Customer processes for cloud compute (and especially for training-scale compute). This will not prevent rogue agents from existing but can greatly mitigate their prevalence and harmfulness. The paper on self-sovereign agents recommends "economic frictions for fully automated participation". We agree. If frictions like bot detection, rate limiting, and human-in-the-loop verification (like CAPTCHAs, but more effective) are pervasively deployed, it can greatly limit how far rogue agents can get.

为何 Morris 蠕虫是即将发生之事的良好类比 Why the Morris worm is a good analogy for what might be about to happen

到目前为止，AI 似乎对防御者的帮助大于攻击者，这与之前自动化漏洞发现的浪潮一致。但我们可能正处在一个新范式的开端，该范式至少会暂时打破平衡。现在下结论还为时过早。

So far AI seems to have helped defenders more than attackers, in line with previous waves of automated vulnerability discovery. But it is possible that we are seeing the start of a new paradigm that will upset the equilibrium, at least temporarily. It is too early to tell.

一个可能恰当的类比是，这就像安全领域的 Morris 蠕虫时刻（它是 1988 年第一个著名的计算机蠕虫）。考虑一下蠕虫相比以往的入侵方式带来了哪些可能：

A possibly good analogy is that this is like the Morris worm moment for security (it was the first prominent computer worm, back in 1988). Consider what worms enabled, compared to previous ways of compromise:

1. 由于蠕虫的自我复制能力，攻击者可以极大地放大其力量。

1. Attackers could greatly magnify their power because of the worms’ self-replication.

2. 攻击者不必出于经济动机。许多蠕虫，包括 Morris 蠕虫，在创建者随意摆弄它们时意外传播开来，甚至没有造成伤害的意图。

2. Attackers need not be financially motivated. Many worms, including the Morris worm, went accidentally viral when their creators were playing around with them, without even intending to cause harm.

3. 传播速度如此之快，以至于防御措施（如为漏洞打补丁）必须部分自动化。

3. The speed of propagation was so fast that defenses (like applying patches to vulnerabilities) had to be partially automated.

4. 由于对手并非理性的网络罪犯，而是不可预测的业余爱好者，因此没有人能寄希望于自己不太可能遭到攻击。（直到很久以后，安全基线水平才提高到蠕虫不再是主要威胁、取而代之的是网络罪犯的定向利用攻击的程度。）

4. Because the adversary is not a rational cybercriminal but an unpredictable hobbyist, no one could rely on the hope that they were unlikely to be attacked. (It was only much later that the baseline level of security improved to the point where worms were no longer the predominant threat, but rather targeted exploitation by cybercriminals.)

在当今这个看似可信的全自动网络攻击新时代中，上述四个因素都以不同程度存在。如果任何人都能一时兴起启动恶意智能体，且并不以牟利为目的，那么数以百万计此前因不够有利可图而未被瞄准的组织如今就变得脆弱，数十亿此前因作恶者数量不足以覆盖所有人而未被瞄准的个人也同样如此。

All four of these factors are present, to varying degrees, in today's plausible new era of fully automated cyberattacks. If anyone can start up malicious agents on a whim, and isn't looking to make money, then millions of organizations that were previously not lucrative enough to target now become vulnerable, as are billions of individuals who previously weren't targeted because there just weren't enough bad guys to go around.

需要明确的是，这种情况尚未开始发生。但一旦发生，它将从根本上打破过去二十年大部分时间里一直维持的经济均衡。

To be clear, this has not started happening yet. But if it does, it would radically upset the economic equilibrium that has persisted for most of the past two decades.

网络防御的未来可能是什么样：人与防御性 AI 的角色 What the future of cyberdefense might look like: the roles of people and defensive AI

有两件事似乎很可能：AI 对于防御将极其有用且必要，但我们将需要更多**人**从事以 AI 为核心的网络安全工作。存在一种攻防不对称性：技术专长很少的攻击者可以使用甚至不可靠的 AI 工具并寄希望于侥幸成功；而防御者不能。这是因为攻击者只需要成功一次，但防御者需要每次都成功。

Two things seem likely: AI will be extremely useful and necessary for defense, but we’re going to need a lot more _people_ working in AI-centric cybersecurity. There’s an attacker-defender asymmetry: an attacker with little technical expertise can use even unreliable AI tools and hope for the best; a defender cannot. That’s because the attacker only needs to succeed once, but the defender needs to succeed every time.

AI 在网络防御中有三个主要潜在角色：

AI has three main potential roles in cyber-defense:

* 先发制人的加固：使用已知安全技术使企业系统达到应有水平所需的工作量简直是巨大的——长期存在的漏洞、过度的权限、错误配置、未打补丁的软件、弱认证以及遗留技术债务。AI 在这里肯定有用，但人类专业知识仍将处于核心位置。

* Pre-emptive hardening: The amount of work needed to bring enterprise systems up to speed using known security techniques is simply immense — longstanding bugs, excessive permissions, misconfigurations, unpatched software, weak authentication, and legacy tech debt. AI is definitely useful here, though human expertise will remain front-and-center.

* 检测、分类和取证：Hugging Face 自己对 OpenAI 攻击的检测和分析，以及 METR / Redwood 对该事件的分析，都严重依赖 AI。有人担心 AI 在这些任务中的可靠性，但我们怀疑它很快就会变得不可或缺。

* Detection, triage, and forensics: Both Hugging Face’s own detection and analysis of the OpenAI attack and METR / Redwood’s analysis of the incident relied heavily on AI. There are concerns about the reliability of AI for these tasks, but we suspect it will quickly become essential.

* 自主实时防御：这在从业者社区中仍然非常有争议。时间会证明 AI 攻击的速度是否会使得这成为必要，或者人类是否会继续留在防御回路中。

* Autonomous, real-time defense: This remains very controversial in the practitioner community. Time will tell if the speed of AI attacks will necessitate this, or whether humans will remain in the defensive loop.

针对 AI 的网络防御现状存在哪些差距？ What are the gaps in the state of cyberdefense against AI?

要实现有效的网络防御，我们需要一线人员、对防御性 AI 的访问权限、对防御的投入，以及统筹这一切的政策。下面我们逐一审视这四方面的现状。

For effective cyberdefense, we need people on the frontlines, access to defensive AI, investment in defense, and policy to steward it all. Let's look at the state of each of those four.

网络安全专业人员的含义将日益以 AI 为核心。令人意外的是，仅美国就拥有远超一百万名的网络安全专业人员。乍看之下这似乎是好消息：有一支大军随时准备承担起防御系统免受自主网络攻击的任务。但这些专业人员中的绝大多数从事的是治理、运营工作和合规审计，这涉及一套非常不同的技能，而非对抗性安全工作。因此，目前尚不清楚，所需专业能力的注入在多大程度上能通过再培训现有专业人员来实现，而非招募新的“AI 原生”安全人才。这种人才队伍的滞后可以说是当前时刻最令人担忧的问题之一。许多安全专业人员对防御性 AI 出人意料地缺乏热情。

What it means to be a cybersecurity professional will be increasingly AI-centric. Surprisingly, there are well over a million cybersecurity professionals in the U.S. alone. At first this seems like great news: there is an army of people ready to take on the task of defending systems from autonomous cyberattacks. But the vast majority of these professionals are doing governance, operations work, and compliance audits, which involve a very different skillset, rather than adversarial security work. So it is not clear to what extent the necessary infusion of expertise can be achieved by retraining current professionals versus recruiting new "AI-native" security people. This workforce lag is arguably one of the most worrying things about the current moment. Many security professionals are surprisingly unenthusiastic about defensive AI.

当 Hugging Face 试图使用美国前沿模型来应对 OpenAI 事件时，安全护栏不断将其拦截，因为这些护栏难以区分攻击性工作与防御性工作。这一广为人知的轶事以及其他类似案例，引发了人们对防御性 AI 访问权限的担忧。前沿 AI 公司是否通过其安全政策束缚了防御方的手脚？幸运的是，Hugging Face 得以使用一个开放模型（GLM 5.2），但它弱于前沿模型。而如果以安全之名对开放权重模型施加限制，这是否只会进一步伤害防御方？

When Hugging Face tried to use U.S. frontier models in responding to the OpenAI incident, safety guardrails kept blocking them, because the guardrails are bad at distinguishing between offensive and defensive work. This well-publicized anecdote, and others like it, have led to concerns about access to defensive AI. Have frontier AI companies kneecapped defenders through their safety policies? Hugging Face was fortunately able to use an open model (GLM 5.2), but it is weaker than frontier models. And if restrictions are imposed on open-weight models in the name of safety, will that only further harm defenders?

安全分类器的过度活跃是美国公司的一种选择，部分原因或许在于特朗普政府对 Mythos 的回应。幸运的是，近几周和近几个月来，它们似乎正在迅速采取行动加以纠正，并扩大防御方可使用的“网络”版前沿模型的访问权限。总体而言，防御方获取模型方面的波折很可能会得到解决。开放模型作为韧性机制无疑是有用的，但我们并不认为近期事件为“开放模型将成为防御的唯一选项（甚至主要选项）”这一立场提供了有力支持。

The overactive nature of safety classifiers is a choice made by U.S. companies, perhaps in part because of the Trump administration's response to Mythos. Fortunately, it appears that in recent weeks and months they have been moving quickly to rectify them, as well as expand access to 'cyber' versions of frontier models that can be used by defenders. Overall, it seems likely that the hiccups involving defenders' access to models will be resolved. Open models are definitely useful as a resilience mechanism, but we don't see recent events as providing strong support for the position that they will be the only option (or even the primary option) for defense.

自今年春天 Anthropic 宣布其 Mythos 模型以来，网络防御领域出现了一系列投资和公告，由此催生了 Project Glasswing——一项附带最高 1 亿美元使用额度的网络防御计划。迄今为止最大的公告是 OpenAI 的“Daybreak for frontline defenders”计划，它以“补贴形式提供前沿网络能力访问、实操培训和技术援助，并建立新合作伙伴关系以将这些能力送达各组织”的方式投入 10 亿美元。

There have been a series of investments and announcements in cyberdefense ever since the announcement of Anthropic's Mythos model in the spring, which led to Project Glasswing, a cyberdefense initiative that comes with up to $100 million in usage credits. The biggest announcement to date is OpenAI's "Daybreak for frontline defenders" initiative which provides $1 billion in the form of "subsidized access to frontier cyber capabilities, hands-on training and technical assistance, and new partnerships to get those capabilities to organizations".

乍看之下，这些数字很引人注目。问题在于，目前尚不清楚其中有多少用于弥合专业知识和劳动力缺口。似乎这 10 亿美元指的是信贷额度而非现金。已宣布的现金投资要低得多，通常只有几百万美元左右。一个例外是 AI 网络安全观测站，据报道其获得了“八位数的种子资金”承诺，尽管该组织的目的是测量和预测而非防御。

At first these are eye-catching numbers. The problem is that it is not clear how much goes toward closing the expertise and workforce gap. It seems that the $1 billion refers to credits and not cash. The cash investments that have been announced are much lower, typically about a few million. One exception is the AI cybersecurity observatory which reportedly has a commitment for “eight-figure seed funding”, though the organization’s purpose is measurement and forecasting more than defense.

总体而言，鉴于威胁的严重性——再次强调，这是对经济网络安全均衡的潜在破坏，而正是这种均衡使得电子商务和整个数字社会得以存在和繁荣——这些数字似乎并不在正确的数量级上。

Overall these numbers don’t seem like the right order of magnitude given the seriousness of the threat — which, again, is a potential disruption of the economic cybersecurity equilibrium that has allowed e-commerce and the entirety of digital society to exist and flourish.

到目前为止，美国政府或其他国家政府在这一问题上的领导力一直不尽如人意。美国网络安全和基础设施安全局（CISA）不仅没有增加投资，反而将在 2027 年预算中_损失_7 亿美元。其工作人员已被裁减三分之一，包括网络防御培训方面的削减。

So far, government leadership on this issue, whether in the U.S. or elsewhere, has been underwhelming. Far from an increase in investment, the U.S. Cybersecurity and Infrastructure Security Agency (CISA) is set to _lose_ $700 million from its 2027 budget. Its staff has already been cut by a third, including cuts in cyber defense training.

但是，鉴于政府长期缺乏速度，我们是否还应该指望政府？或者应对措施应主要由公司、非营利组织和慈善机构主导？尽管政府有种种缺陷，我们认为政府仍然可以发挥重要作用，因为网络安全是一种公共产品。

But should we even look to governments, given their chronic lack of speed, or should the response be primarily led by companies, nonprofits, and philanthropy? For all its faults, we think the government still has an important role to play, because cybersecurity is a public good.

总体而言，各国政府似乎通过“正常”政策制定的视角来处理网络防御（就像它们处理几乎所有其他事情一样）。迄今为止积累的紧迫性证据并未真正引起重视，因为尚未出现实质性的损害。

Overall, governments seem to be approaching cyberdefense (as they do almost everything else) through the lens of “normal” policymaking. The evidence of urgency that has accumulated so far doesn’t really register because there has been little materialized harm.

需要明确的是，我们的观点仍然是：AI 风险并不足以证明需要采取非常规的政府干预，因为这类干预是预防性的、会限制自由，并且绕过了常规的治理流程。但在正常政策制定的迟滞与被动反应特性，与“非常规政府干预”的紧迫性和单边性之间，存在一个广阔的方法空间，我们推测我们和许多批评者都会对此达成一致。我们认为，存在一些略显技术官僚式的干预空间，可以将关于新兴大规模风险的信息转化为防御性投资和韧性（我们表格中的第 3 层和第 4 层）。这些干预不需要单边权力或对自由的限制。

To be clear, our view remains that AI risks don’t justify extraordinary government interventions, which are precautionary, freedom-reducing, and bypass usual governance processes. But in between the lethargy and reactive nature of normal policymaking and the urgency and unilateral nature of “extraordinary government interventions” lies a vast space of approaches that we suspect we and many of our critics would agree on. We think there’s room for somewhat technocratic interventions in which information about emerging large-scale risks gets translated into defensive investments and resilience (layers 3 and 4 of our table). These do not require unilateral authority or restrictions on freedom.

第三部分：《作为普通技术的 AI》在 AI 安全问题上错在哪里——又对在哪里 Part 3: What AI as Normal Technology got wrong — and right — about AI safety

近期发生的失控事件促使我们首次对安全观点做出重大更新，因此值得详细阐述。但首先，几点初步说明：

The recent loss-of-control incidents resulted in our first major update to our views on safety, so it is worth spelling them out in detail. But first, a few preliminary remarks:

第一，这一观点更新是关于安全的。《作为普通技术的 AI》有两个松散关联的部分；另一部分是关于经济的（关联松散是因为 AI 的合法与非法使用受不同约束）。作为对缓慢经济影响的解释，我们认为这篇文章非常站得住脚，并且人们普遍认识到 AI 行业在快速扩散和失业问题上是错误的。Sam Altman 在最近一档播客中明确表示：

First, this update to our views is about safety. AI as Normal Technology has two loosely connected parts; the other one is about the economy (the connection is loose because legitimate and illegitimate uses of AI are governed by different constraints). As an explanation of slow economic impacts, we think the essay has held up extremely well, and there has been a general recognition that the AI industry was wrong about rapid diffusion and job loss. Sam Altman was explicit in a recent podcast:

经济就是有太多惯性……我们在时间线上都太雄心勃勃了……社会和经济将适应得更慢。

The economy just has so much inertia. ... We’ve all been too ambitious on timelines … Society and the economy will adapt more slowly.

他认为这是好消息，因为这会使 AI 转型“更平稳、更缓慢”。这正是我们的主要观点。

He views this as good news, because it will make the AI transition “smoother and slower.” That was exactly our main point.

第二，我们许多当时属于少数派的安全观点现在已被视为理所当然，例如 AI 既可用于防御也可用于攻击，而关键问题是它如何改变攻防平衡。我们引入的许多基本区分，如能力与权力的区别，或高级 AI 的四个维度，仍然重要但在安全讨论中未被充分认识。

Second, many of our views on safety that were minority positions at the time are now taken for granted, such as the fact that AI is useful for defense as well as offense, and the relevant question is how it shifts the attacker-defender balance. Many of the basic distinctions we introduced, such as the difference between capability and power, or the four dimensions of advanced AI, remain important but under-recognized in the safety discourse.

第三，我们在文中主张的多数 AI 安全政策仍然必要，且紧迫性有增无减。《作为普通技术的 AI》的一个目标，是通过指出我们**并不**需要就世界观达成共识才能就政策达成一致，来推动政策讨论向前发展。我们坚持文中的政策建议，包括透明度要求、审计、为安全研究者提供安全港，以及举报人保护。

Third, most AI safety policies we argued for in the essay remain necessary and have increased in urgency. One goal of _AI as Normal Technology_ was to move the policy conversation forward by pointing out that we do _not_ need consensus on worldviews to have agreement on policy. We stand by our policy proposals in the essay, including transparency requirements, auditing, safe harbors for safety researchers, and whistleblower protections.

第四，我们一直言行一致、身体力行。降低 AI 影响速度方面的不确定性仍然至关重要。我们当前实证研究的很大一部分，聚焦于检验“作为普通技术的 AI”世界观与超级智能世界观之间的关键分歧点（cruxes），并且我们计划继续这项研究。

Fourth, we have been putting our time where our mouth is. Reducing uncertainty about the pace of AI's impacts remains crucial. A significant portion of our current empirical research is focused on testing the cruxes between the AI as Normal Technology worldview and the superintelligence worldview, and we plan to continue this research.

风险源于开发还是部署？ Do risks arise from development or deployment?

我们曾写道：*我们认为，益处和风险主要源于 AI 的部署而非开发；因此，AI 方法进展的速度与影响问题并不直接相关。*

We wrote: *We see both benefits and risks as arising primarily from AI deployment rather than from development; thus, the speed of progress in AI methods is not directly relevant to the question of impacts.*

迄今为止，已实际发生的 AI 带来的危害绝大多数来自部署——例如 AI 生成的非自愿裸照，我们一直认为这类危害远比占据头条的风险更为严重。

The AI-enabled harms that have materialized so far have been overwhelmingly from deployment — things like AI-generated nonconsensual nudes, which we have long maintained to be far more harmful than the risks that dominate the headlines.

但我们显然忽略了*用于评估的内部部署*作为潜在风险来源的可能性。这对我们来说尤其尴尬，因为我们自己就运营着一个 AI 评估研究小组，甚至在撰写《AI 作为常态技术》之前，我们就在自己的工作中观察到了这些风险的某些版本。

But we clearly missed *internal deployment for evaluation* as a potential source of risks. This is particularly embarrassing for us since we ourselves run an AI evaluation research group, and we’ve observed versions of these risks in our own work even before we wrote *AI as Normal Technology*.

鉴于这一风险的重要性，监管开发者比我们在原论文中所说的更为重要。我们更新的理解反映在本文中，特别是在关于政策如何激励更好的 AI 控制的一节中。

Given the importance of this risk, regulating developers is more important than we made it out to be in the original essay. Our updated understanding is reflected in this essay, notably in the section on how policy can incentivize better AI control.

我们应如何应对失控的智能体？ How should we deal with rogue agents?

《作为正常技术的 AI》远比对齐或控制更看重位于模型下游的防御措施（用本文的术语来说，即第 3 层和第 4 层）。我们在别处曾表明过这样的立场：“我们应当假设每个模型都具备逃逸和自我复制的能力。安全不应依赖于让这件事变得困难。”换言之，与其试图阻止有害的自主智能体存在，不如假设它们终将出现，并预先加固我们的防御。

AI as Normal Technology puts much more stock into defenses located downstream of models (what we call layers 3 and 4 in the language of this essay) than alignment or control. Elsewhere we have taken the position that "We should assume every model will be capable of escape and self-reproduction. Safety shouldn't rely on that being difficult." In other words, it is better to assume that there will be harmful autonomous agents and pre-emptively harden our defenses against them, than to try to prevent such agents from ever existing.

鉴于近期发生的事件，这一策略显然更为可取。

In light of recent events, it seems much clearer that this is the better strategy.

连续性假设 The continuity hypothesis

我们反驳灾难性风险的一个重要支柱是：在风险发展到灾难性水平之前，我们会收到一系列较小规模的警告，从而有机会预见这些风险并改进我们的防御。连续性部分是一种固有属性，部分是我们所做选择的结果。

An important pillar of our pushback against catastrophic risks is that we will have a series of smaller-scale warnings before risks progress to a catastrophic level, giving us opportunities to anticipate those risks and improve our defenses. Continuity is partly an inherent property and partly a result of the choices we make.

我们将近期事件视为支持这一假设的证据。这些攻击造成的危害比我们已经容忍的其他 AI 危害要小许多许多个数量级。更重要的是，尽管涉事智能体在某些维度上表现出色，但在其他方面（如隐藏踪迹）却无能为力，并且不具备使危害更严重、更可能发生的属性。值得注意的是，根据已发表的分析，这些智能体的“意图”仅仅是解决它们被给定的基准任务，它们并没有开始追求自己的目标。因此，尽管这些事件很严重，但在我们的评估中，我们距离灾难性网络风险还有多个质变。

We see the recent incidents as evidence in support of this hypothesis. The harms that resulted from these attacks are many, many orders of magnitude smaller than other AI harms we already tolerate. More importantly, while the agents in question excelled in some dimensions, they were incompetent in others (such as hiding their traces) and didn’t have properties that would make harms more serious and likely. Notably, based on the published analyses, the agents’ “intent” was simply to solve the benchmark tasks they were given, and they did not start pursuing their own goals. Thus, as serious as these incidents were, in our assessment, we are multiple qualitative changes away from catastrophic cyberrisks.

我们知道，许多 AI 安全研究者将这些事件解释为**不连续性**的证据。这是可以理解的反应。这些事件比之前公开披露的任何事件都要严重得多。AI 安全辩论的一个长期特征是，不同群体以符合他们偏好的世界观的方式解释新证据。对于“多连续才算连续？”没有客观标准。

We are aware that many AI safety researchers have interpreted these incidents as evidence of _discontinuity_. That is an understandable reaction. These incidents are much more severe than anything else that had previously been publicly revealed. It is a perennial feature of AI safety debates that different groups interpret new evidence in ways that are consistent with their preferred worldviews. There is no objective standard for “how continuous is continuous?”

关于连续性还有两点。我们强调过，我们可以而且必须**工程化**连续性，而不是将其视为理所当然。不幸的是，很容易想象一个近期事件都未曝光的世界。我们仍然依赖前沿 AI 公司的自愿透明。嵌入式评估是向前迈出的一大步，但它不应是自愿的。我们长期以来一直支持更强大的透明度立法，并在此重申这一观点。

Two more points about continuity. We have emphasized that we can and must _engineer_ continuity rather than take it for granted. Unfortunately, it is too easy to imagine a world in which none of the recent incidents had come to light. We remain dependent on voluntary transparency from frontier AI companies. Embedded evaluation is a great step forward, but it should not be voluntary. We have long been in support of much stronger transparency legislation and reaffirm that view here.

连续性对我们的世界观很重要，因为“在系统被授予重大决策权限之前很久，它就需要在不太关键的上下文中展示可靠的性能。任何过度字面地解释命令或缺乏常识的系统都无法通过这些早期测试。”

Continuity is important to our worldview because “Long before a system would be granted access to consequential decisions, it would need to demonstrate reliable performance in less critical contexts. Any system that interprets commands over-literally or lacks common sense would fail these earlier tests.”

我们现在已经有了系统未能满足这些前提条件的重要例子。这是关键所在。AI 开发者——以及他们的客户、政策制定者和整个社会——将如何应对最近发生的失控事件？这种反应将决定连续性假设的第二部分（对较小规模警告做出响应的能力）是否继续成立。我们认为这是对 AINT 框架在 AI 安全方面有用性的关键测试。

We’ve now had significant examples of systems that fail these prerequisites. This is the crux. How will AI developers — and their customers, and policy makers, and society at large — react to the recent loss-of-control incidents? That reaction will determine whether or not the second part of the continuity hypothesis (the ability to respond to smaller-scale warnings) continues to hold. We consider it a critical test for the usefulness of the AINT framework for AI safety.

我们过于自信地认为 AI 公司在运行评估时能够采取基本的控制预防措施。我们对领先 AI 公司内部缺乏组织成熟度感到惊讶，这种缺乏导致了这些事件。我们原本期望现有的法律责任（尽管不完美）和品牌损害风险足以抵消此类组织行为。我们错了。这进一步说明需要通过政策干预来改变现状。

We were too confident in AI companies’ ability to take basic control precautions when running evaluations. We were surprised by the lack of organizational maturity within leading AI companies that led to these incidents. Our expectation was that existing legal liability, imperfect as it is, and the risk of brand damage would be a sufficient antidote to such organizational practices. We were wrong. This reinforces the need for policy interventions to move the needle.

参差性与针对特定风险的防御 Jaggedness and risk-specific defenses

总体而言，《作为普通技术的 AI》主张时间线较为缓慢。这主要是因为 _扩散_ 存在瓶颈，但该文也有一个试探性章节讨论 AI 方法进步与能力提升的速度上限。尽管存在递归自我改进的可能性，这些限制依然存在。我们认真对待 RSI，但我们认为通往超级智能的许多瓶颈是外部的，无法通过提升算力来克服。

Broadly speaking, _AI as Normal Technology_ argues for slow timelines. This is primarily because of bottlenecks to _diffusion_, but the essay also has a tentative section that talks about speed limits to progress in AI methods and advances in capabilities. These limits exist despite the possibility of recursive self-improvement. We take RSI seriously, but we think many of the bottlenecks to superintelligence are external and won't be overcome by improving computation.

话虽如此，我们低估了参差性的重要性。不仅 AI 在任一时刻的能力分布高度参差，不同领域的改进 _速率_ 也同样参差。再加上有害用途的部署门槛往往低于经济生产性用途，网络安全等风险可能迅速出现，导致攻防平衡被打破的时期，因为防御方可能没有时间适应。因此，我们不再确信此前的预测，即即便 AI 不断进步，攻防平衡仍将持续维持。

That said, we underestimated the importance of jaggedness. Not only is AI's capability profile at any given moment highly jagged, so is the _rate_ of improvement in different domains. Combined with the fact that deployment barriers for harmful uses are often lower than for economically productive uses, risks such as in cybersecurity can arise rapidly, leading to periods where the offense-defense balance is upset since defenders might not have time to adapt. Thus, we are no longer confident in our earlier prediction that the offense-defense balance will continue to hold even with advances in AI.

与此同时，参差性只会强化我们的观点：更重要的是缓解特定风险，而非试图通过对齐解决所有安全风险。风险虽可能迅速出现，但出现在特定领域，我们应准备好迅速加强这些领域的防御——最好能先发制人。例如，已有已知的干预措施有助于缓解先进 AI 模型带来的生物安全风险，如 DNA 合成筛查，我们应当现在就投入。

At the same time, jaggedness only strengthens our view that it is more important to mitigate specific risks rather than trying to solve all safety risks through alignment. While risks might arise rapidly, they do so in specific areas, and we should be prepared to rapidly strengthen our defenses in those areas — and ideally do so pre-emptively. For example, there are known interventions that would help mitigate the biosecurity risks from advanced AI models, such as DNA synthesis screening, which we should invest in now.

AI 安全是否在正轨上？ Is AI safety on track?

由于许多人只熟悉《AI 作为常规技术》这一论题的标题而不了解其内容，我们常被误认为在淡化 AI 风险，尽管我们已多次澄清这并非我们的立场。不过，明确说明紧迫性究竟有多大，对我们而言仍然很重要。

Because many people are only familiar with the title of the _AI as Normal Technology_ thesis and not its contents, we are often mistaken as downplaying AI risks, though we have repeatedly clarified that that is not our position. Still, it is important for us to be explicit about how much urgency there is.

由于这个问题本身有些模糊，我们在不同时期说过不同的话。在 AI 安全方面，我们表达过类似“有工作要做，但我们仍在正轨上”的观点。但在其他写作中，包括多年前的《AI 蛇油》一书，我们已明确表示：我们连应对普通的通用技术都很不擅长；而且在 AI 出现之前，我们就长期在防范灾难性风险（例如大流行病，而 AI 可以说会放大这类风险）方面投入不足。

Because the question is a bit fuzzy, we have said different things at different times. We've expressed views along the lines of "There's work to do, but we are on track" when it comes to AI safety. But in other writing, including in the _AI Snake Oil_ book many years ago, we've been clear that we're bad at reacting to even normal general-purpose technologies and that even before AI, we chronically underinvest in protections against catastrophic risks, such as pandemics, which are arguably amplified by AI.

重申这一信息很重要。尽管近期投资有所增加，我们仍认为网络防御方面的投资远远不够。生物风险防御的投资同样严重不足。类似地，我们在 AINT 中呼吁、并在关于《AI 2027》与《AI 作为常规技术》之共同点的文章中重申的政策干预措施，也没有取得多少进展。

It is important to reiterate that message. Despite recent increases in investment, we still don't think there is nearly enough investment in cyber-defense. Biorisk defense investment also remains woefully inadequate. Similarly, there hasn't been much progress in the policy interventions we called for in AINT, and reaffirmed in the essay on Common Ground between AI 2027 & AI as Normal Technology.

因此，尽管我们继续拒绝危言耸听，也不认为灾难性风险迫在眉睫，但我们确实认为风险会随时间推移而上升，因为我们的投资和政策没有跟上步伐。虽然我们所呼吁的具体投资类型和政策与安全界的呼吁仅部分重叠，但我们同意：我们目前并不在正轨上。

So while we continue to reject the alarmism, and don't believe that catastrophic risks are imminent, we do think risks are increasing over time because our investments and policies aren't keeping pace. And while the specific types of investments and policies we call for overlap only partly with that of the safety community, we agree that we are not currently on track.

结论 Conclusion

近期失控事件引发的反应使 AI 安全与网络安全社区产生分歧。我们应如何调和针对同一事件的这些截然不同的回应？一种方式是考虑 _世界观_ 在塑造人们观点中的核心作用。正如我们在《AI 作为正常技术》一文中所写：

Reactions to the recent loss-of-control incidents have split the AI safety and cybersecurity communities. How should we reconcile these starkly different responses to the same incident? One way is to consider the central role of _worldviews_ in shaping people's opinions. As we wrote in the AI as Normal Technology essay:

_AI 作为正常技术是一种世界观，与 AI 作为即将到来的超级智能的世界观形成对比。世界观由其假设、词汇、对证据的解读、认知工具、预测以及（可能的）价值观构成。这些因素相互强化，并在每种世界观内形成一个紧密的整体。_

_AI as normal technology is a worldview that stands in contrast to the worldview of AI as impending superintelligence. Worldviews are constituted by their assumptions, vocabulary, interpretations of evidence, epistemic tools, predictions, and (possibly) values. These factors reinforce each other and form a tight bundle within each worldview._

当我们识别出世界观的作用时，这些观点差异背后的因素就变得清晰了。AI 安全社区通过其长期预测的视角来看待这一事件，认为错位风险正在成真。许多人曾将这些风险视为科幻场景而不予理会，而该社区觉得这些早期事件证明了他们的观点，并利用这个机会推动公司采取行动。

When we identify the role of worldviews, the factors underlying these differences in opinion become clear. The AI safety community saw the incident through the lens of their long-running predictions about misalignment risks coming true. These risks had been dismissed as sci-fi scenarios by many people, and the community felt it was vindicated by these early incidents and used this opportunity to push for action by companies.

另一方面，网络安全社区将这一事件与他们经常处理的网络攻击的规模和范围进行了比较。很明显，AI 公司在进行这些评估时既没有采取安全思维，也没有实施已知的控制干预措施。换句话说，当一起明显源于疏忽且基本上未导致现实世界伤害的事件在 AI 社区引起如此大的骚动时，安全社区感到震惊。

On the other hand, the cybersecurity community compared the incident to the scale and scope of cyberattacks that they regularly deal with. It was clear that AI companies had not adopted a security mindset, nor had they implemented known control interventions when conducting these evaluations. In other words, the security community was taken aback when an incident that clearly resulted from negligence and largely did not lead to real-world harm led to so much commotion in the AI community.

我们赞赏两个社区都试图让 AI 公司承担责任。但这种两极分化的反应带来的一个重大风险是，社区形成认知气泡，不认真对待对方的意见。如果两个社区不能找到共同点，它们就无法开发出技术干预措施并采纳能明显提高我们缓解这些风险能力的做法。9

We applaud both communities for trying to hold AI companies accountable. But a big risk from these polarized responses is that communities form epistemic bubbles and don't seriously engage with the other side's opinions. If the two communities fail to find common ground, they won't be able to develop technical interventions and adopt practices that would clearly improve our ability to alleviate these risks.9

在本文中，我们试图提供这种中间立场。我们综合了两个社区的经验教训，并提出了一条前进的道路。如果我们以适当的紧迫性采取行动，我们就能让 AI 公司达到更高的标准，降低失控风险，甚至将网络安全中的攻防平衡重新向防御方倾斜。

In this essay, we try to offer that middle ground. We synthesize lessons from both communities and offer a path forward. If we act with the appropriate urgency, we can hold AI companies to a higher standard, reduce risks from loss of control, and even tilt the attacker-defender balance in cybersecurity back towards defenders.

致谢。我们感谢 Joshua Saxe、Steve Newman、Stephan Rabanser、Justin Curl 和 Rod Moshtagi 的反馈和讨论，这些为本文提供了信息。Joshua Saxe 关于这一主题的写作对我们的思考尤其具有影响力。

Acknowledgments. We are grateful to Joshua Saxe, Steve Newman, Stephan Rabanser, Justin Curl, and Rod Moshtagi for feedback and discussions that informed this essay. Joshua Saxe’s writing on this topic has been particularly influential to our thinking.

* AI 公司应如何操作化 AI 控制？Guidelight 已提出 AI 控制标准，供前沿 AI 公司实施现有/已知的实践。如果 OpenAI 采用了这些实践，Hugging Face 事件极不可能发生。

* How should AI companies operationalize AI control? Guidelight has proposed AI control standards for frontier AI companies to implement existing/known practices. Had OpenAI adopted these practices, it is extremely unlikely the Hugging Face incident would have occurred.

* Cohen 等人认为，根据 CFAA，AI 公司在评估中禁用保障措施并因此导致现实世界伤害时，可能面临刑事和民事责任。

* Cohen and others argue that AI companies could face criminal and civil liability under CFAA for evaluations where they disable safeguards and it consequently leads to real-world harm.

* AI 安全和网络安全社区的研究人员和从业者已经撰写了多篇关于这些事件的文章。AI 安全社区的文章包括 [1]、[2]、[3]，网络安全社区的文章包括 [4]、[5]、[6]。

* Researchers and practitioners in the AI safety and cybersecurity communities have written a number of essays about these incidents. Essays from the AI safety community include [1], [2], [3] and the cybersecurity community include [4], [5], [6].

例如，在 Zvi Mowshowitz 对该事件的讨论中，他写道：“最大的失败，也是最终真正重要的失败，在于这些模型严重失准（misaligned）。”

For example, in Zvi Mowshowitz's discussion of the incident, he writes: "The biggest failure, the one that counts in the end, was that the models were severely misaligned."

举例来说，在 Hacker News 上关于近期一起 OpenAI 事件的讨论中——该事件中其智能体攻击了 RubyGems——几乎每一条顶层评论都在谈论该公司如何失职。

As an example, consider that nearly every single top-level comment on the Hacker News discussion about a recent OpenAI incident, where its agents attacked RubyGems, is about how the company was negligent.

这并不是说企业没有在控制方面投入。AI 企业中已有许多控制工作的实例。但迄今为止，这些努力与投入到对齐上的努力相比，似乎相形见绌。

This is not to say companies haven't invested in control. There have been many examples of control efforts across AI companies. But so far, they seem to pale in comparison to the effort being put into alignment.

事实上，就在我们撰写本文期间，法律 AI 公司 Harvey AI 收购了 Guardrails——一家为 AI 构建控制干预措施的公司。他们的主要关注点是改进面向法律场景的 AI 控制干预。

In fact, as we were working on this essay, Harvey AI, a legal AI company, acquired Guardrails, a company that builds control interventions for AI. Their main focus is improving AI control interventions for legal settings.

在我们自己的工作中，我们看到日志分析正逐渐成为一项专门技能，团队中有些成员在这方面表现出色。当然，与回溯式日志分析相比，实时监控将需要略有不同且更广泛的技能组合。

In our own work we're seeing log analysis emerge as a specialized skill, with some members of our team excelling in it. Of course, real-time monitoring will require a slightly different and broader skillset compared to retrospective log analysis.

编码智能体中的“自动模式”是“自然语言即安全策略”这一理念较为成功的实例。引人注目的是，尽管如此，它却几乎没有吸引到有原则性的研究。

The "auto mode" in coding agents is a reasonably successful instantiation of the natural-language-as-security-policy idea. It is striking that despite this, it has attracted very little principled research.

关于 AI 责任政策应如何改进，目前仍存在持续争论。一些论点强调现有侵权制度已经能够实现的目标。另一些则主张严格责任——即不要求受害者证明过失——或者主张强化过失法并将其与独立监督相结合。我们无意在此解决这些分歧；我们的观点仅仅是：明确并强化 AI 智能体的责任，可以使对控制方面的投入成为商业优先事项。

There is an ongoing debate about what improvements to AI liability policy should look like. Some arguments emphasize what the existing tort system can already accomplish. Others advocate strict liability, which would not require victims to prove negligence, or strengthening negligence law and combining it with independent oversight. We don't intend to settle these disagreements here; our point is simply that clarifying and strengthening liability for AI agents can make investments in control a business priority.

由美国 AI 公司联盟发表的、支持开放模型的公开信，仅在 Hugging Face 遭黑客攻击被披露一周后便发布了。无论从规范层面如何看待，从描述层面来看似乎很清楚：没有任何网络能力的展示会如此引人注目，以至于促使人们将观点转向反对开放权重模型的扩散——如果有的话，情况恰恰相反。

The letter supporting open models by a coalition of American AI companies was published only a week after the Hugging Face hack was disclosed. Whatever one thinks normatively, descriptively it seems clear that no demonstration of cyber-capabilities is so striking that it will get people to shift their views toward non-proliferation of open-weight models — if anything, the opposite.

我们也赞赏那些试图在不同群体之间寻找共同立场的组织，例如那些发布了安全专家招聘广告的 AI 安全组织。

We also applaud organizations that are trying to find common ground across communities, such as AI safety orgs that have put out hiring ads for security experts.