An Alien Mind: Jakub Pachocki Warns Us
打开互动全文版（逐段中英对照 + 针对本篇提问）→本文考察了雅库布·帕乔基在其文章《异类心智》中阐述的立场：递归自我改进与超级智能即将到来，而当前的对齐与监控技术却远远不够。帕乔基认为，没有人对后果做好了准备；如有必要，OpenAI 将单方面停止进一步扩展规模；并且需要包括国际协调和第三方执法在内的更广泛干预，以落实正式的安全底线。作者大体认同对齐是 AI 研究的核心问题，也认同我们既须加快对齐研究、又须放慢能力扩展，同时批评 OpenAI 未能区分义务论式与德性伦理式的对齐路径，并含糊地依赖自动化对齐研究者和'对人类之爱'。结论是：帕乔基的文章是目前大型实验室内部人士写得最好的一篇，但言辞之后必须有硬性承诺与行动，包括在 SB 53 等法律框架下采取行动。
This essay examines Jakub Pachocki's position, laid out in his essay An Alien Mind, on the near-term arrival of recursive self-improvement and superintelligence, and the inadequacy of current alignment and monitoring techniques. Pachocki argues that no one is prepared for the consequences, that OpenAI will unilaterally withhold further scaling if needed, and that broader interventions including international coordination and third-party enforcement are required to enforce formal safety bars. The author largely agrees that alignment is the central problem of AI research and that we must both accelerate alignment work and slow capabilities scaling, while criticizing OpenAI's failure to distinguish deontological from virtue-ethical alignment approaches and its vague reliance on automated alignment researchers and 'love for humanity.' The conclusion is that Pachocki's essay is the best writing yet from inside a major lab, but that words must be followed by hard commitments and action, including under laws such as SB 53.
9. 电话来自屋内。
9. The Calls Are Coming From Inside the House.
Jakub Pachocki 现已完整阐述了他对当前局势的立场。
Jakub Pachocki has now fleshed out his full position on the current state of play.
以下是他要点，以我自己的口吻转述：
Here are his key points, translated into my own voice:
1. 超越人类智能的智能将在我们有生之年到来。
1. Smarter than human intelligence is coming in our lifetime.
2. 基于内部结果，他预计递归自我改进将在几年内出现。
2. Based on internal results, he expects recursive self-improvement in a few years.
3. 没有人对后果做好准备。
3. No one is prepared for the consequences.
4. OpenAI 将根据需要单方面暂停进一步的 Scaling（规模扩张）。
4. OpenAI will unilaterally withhold further scaling as needed.
5. OpenAI 无法独自完成此事。需要更广泛的干预措施，包括国际协调，以强制执行对正式安全标准的承诺。
5. OpenAI cannot do it alone. Broader interventions are required, including international coordination, to enforce commitments to formal safety bars.
6. 能力进步可以被引导，迄今为止，它很大程度上被引导向而非远离 RSI，以及‘自动化对齐研究人员’。
6. Capabilities progress can be steered and so far it has largely been steered towards rather than away from RSI, along with ‘automated alignment researchers.’
7. 对齐是 AI 研究的核心问题。
7. Alignment is the core problem of AI research.
8. 对齐分为目标对齐（‘AI 是否试图实现目标？’）与价值对齐。价值对齐最为关键。
8. Alignment splits into goal alignment (‘does the AI try to accomplish the goal?’) versus value alignment. Value alignment is what counts most.
9. AI 对齐的根本挑战在于（价值观的）泛化。
9. The fundamental challenge of AI alignment is generalization (of values).
10. 他认为对齐技术有两类：目标导向的强化学习，或从预训练数据中提升泛化能力。他们在这两类上都投入巨大。
10. He sees two classes of alignment techniques: goal-oriented RL, or improving generalization from pre-training data. They invest heavily in both types.
11. OpenAI 在思维链（CoT）监控上投入巨大。
11. OpenAI has invested heavily in Chain of Thought (CoT) monitoring.
12. 思维链监控的有效性正在逐步减弱。
12. CoT monitoring is progressively diminishing in effectiveness.
13. 支持 Scaling（规模扩张）AI 的剩余主要论据，是用于防御规模化 AI 的网络防御。
13. The main argument left for scaling AI is for cyber defense against scaled AIs.
15. 我们的选择是：要么加速对齐工作，要么放慢能力 Scaling（规模扩张）。我们应当两者兼顾。
15. Our options are to accelerate alignment work or slow down capabilities scaling. We should do both.
16. 归根结底，他指望的是“自动化的对齐研究者”。
16. Ultimately he is counting on 'automated alignment researchers.'
或者，如果把它归结为最重要的一点：
Or, if you narrow it down to the most important thing:
1. 递归自我改进与超级智能即将到来。没有人知道如何安全地做到这一点，我们的对齐技术并不充分，而我们的监控技术正开始失效。我们需要找到解决方案，这将涉及自愿放缓、围绕节奏进行协调，以及进一步投资于对齐，包括自动化的对齐研究者。
1. Recursive self-improvement and superintelligence are coming soon. No one knows how to do this safely, our alignment techniques are inadequate and our monitoring technology is starting to fail. We need to figure out a solution, which will involve a combination of voluntary slowdowns, coordination around pacing, and investing further in alignment, including automated alignment researchers.
如果 OpenAI 的更多沟通能像 Jakub Pachocki 在其新文章《An Alien Mind》开篇那样，我会对那里掌握在可靠之人手中更有信心。
If more OpenAI communications were more like how Jakub Pachocki opens his new essay, _An Alien Mind_, I would feel much more confident we were in good hands there.
Jakub Pachocki：2023 年年中，在“RLSlow”研究项目中，我们看到了首批结果，这些结果让我们有信心能够扩展推理模型的训练规模，释放预训练模型形成自身思维链的能力。那天晚上，Szymon 和我留在办公室，思考的不是这项技术将带来的惊人基准测试数字、产品或科学成果——而是努力消化一个令人清醒的事实：我们将在有生之年真正看到比我们更聪明的机器，而且我们已经看到了这些系统的雏形；我们在想，该如何让人们意识到这件事的重要性。
Jakub Pachocki: In mid-2023, within the "RLSlow" research project, we saw the first results that gave us confidence that we will be able to scale the training of reasoning models, unlocking the capability of pretrained models to form their own chains of thought. Szymon and I spent that night at the office, thinking not about the incredible benchmark numbers, products, or scientific results that this technology will deliver — but rather, trying to process the sobering fact that we will actually see machines meaningfully smarter than ourselves in our lifetime, and we already see the shape of these systems; wondering how to alert people to the significance of this.
三年后，推理语言模型已成为经济中快速增长的一部分，并开始推动科学的边界。它们能够操作计算机和图形界面，与人和彼此协作，并开展研究项目。它们也正在改变计算机安全的格局，并在这一领域带来了明显的新危险。
Three years later, reasoning language models are a rapidly growing part of the economy and starting to push the boundaries of science. They are able to operate computers and graphical interfaces, collaborate with people and each other, and carry out research projects. They are also transforming the landscape of computer security, and in that present clear new dangers.
这一时期涌现了大量新研究，我们对这些系统的理解又与 2023 年时略有不同。基于内部结果，我强烈预期这种进步速度可以持续到递归自我改进。如果 AI 发展沿当前路径继续，未来几年我们将看到的系统很可能代表同等或更大规模的能力跃升，并越来越多地驱动自身的发展。
A lot of new research happened in this period, and our understanding of these systems is again a little different than it was in 2023. Based on internal results, I have a strong expectation that this speed of progress could be sustained into recursive self-improvement. If AI development continues along its current path, the systems we'll see in the next few years are likely to represent further capability jumps of equal or larger magnitude, and to increasingly drive their own development.
AI 实验室中目睹正在发生之事的人，预期递归自我改进和超级智能很快就会到来。他们对此已警告了一段时间。此后的观察与他们的警告一致。
Those at the AI labs, who see what is happening, expect recursive self-improvement and superintelligence to happen soon. They have been warning about this for some time. Observations since then have been consistent with their warnings.
这是一个需要极度谨慎的时刻。我担心没有人对机器智能持续快速上升的后果做好准备。OpenAI 将继续寻求对齐与监测的技术解决方案，构建防御系统，并在需要时单方面暂停进一步的 Scaling（规模扩张）；然而，我认为需要更广泛的干预。
This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence. OpenAI will continue to seek technical solutions to alignment and monitoring, to build defensive systems and unilaterally withhold further scaling as needed; however, I believe broader interventions are required.
对于那些说我们无法引导 AI 能力发展路径的人，他的回答是“是的，我们可以”，至少在一定程度上可以，而且我们正在这样做：
To those who say we cannot guide the path of AI capabilities, he says "Yes We Can," at least to some extent, and we are doing so:
我们花大量时间试图理解能力如何泛化，以及应当优先推进哪些方向，以提升未来几年中最相关的技能。例如，我们相信，如果额外投入精力，我们可以让模型在数学研究方面表现得更好，但我们并不优先推进这一方向，因为我们对 RSI 和自动化对齐研究感到紧迫，这一点我将在后文讨论。
We spend a lot of time trying to understand how capabilities generalize, and what to prioritize to advance the skills that are going to be most relevant in the next few years. For instance, we believe we could make the models better at specifically mathematics research with additional focus, but we do not prioritize this direction because of the urgency we feel about RSI and automated alignment research, as I will discuss later.
能够做出选择并不意味着我们会做出明智的选择。我希望 AI 在 RSI 上更弱、在数学研究上更强，或者更好的是在医学研究之类的事情上更强。然而，竞争压力却推动着 AI 在数学研究上更弱、在 RSI 上更强。
Being able to choose does not mean we will choose wisely. I would like AI to be worse at RSI and better at math research, or better yet things like medical research. Instead, competitive pressures push towards being worse at math research and better at RSI.
RSI 与“自动化对齐研究员”在技术树上看起来是非常相似的点，这无助于解决问题，但这并不是说 OpenAI 正试图避开 RSI。
RSI and 'automated alignment researcher' looking like very similar points on the tech tree does not help matters, but it's not like OpenAI is trying to steer away from RSI.
Jakub 给出了这一明智的警告。不，AI 并不需要在所有方面都更优秀才能改变世界或把我们全部毁灭。最重要的是，它并不需要在所有方面都更优秀才能让自己在所有方面变得更优秀，正如一个人或一个群体也不需要同样地在所有方面都更优秀。
And Jakub offers this wise warning. No, AI does not need to be better at everything in order to transform the world or get us all killed. Most importantly, it does not need to be better at everything in order to make itself become better at everything, any more than a human or group needs to be similarly better at everything.
通过 Scaling（规模扩张）深度学习所产生的智能，并不能直接与人类智能相提并论。要在现实世界中变得非常相关——非常有用或非常危险——AI 并不需要匹配或超越人类的所有能力；它只需要超越足够多的能力。而随着它在越来越多的维度上超越人类，要准确理解它究竟有多强大正变得越来越困难。
The intelligence produced by scaling deep learning is not directly comparable to human intelligence. To become very relevant in the real world - very useful or very dangerous - the AI does not need to match or exceed all human capabilities; it just needs to surpass enough of them. And as it continues to surpass humans on more and more axes, it is becoming increasingly difficult to understand exactly how capable it is.
我不知道这有多大分量，但我同意 Jakub 的观点：对齐不是一个次要问题，而是核心问题。如果你在相关意义上‘解决了对齐’，其余问题都会变得容易；如果你没有解决，其余问题就不可能解决，甚至更糟：
I don't know how much weight it carries, but I agree with Jakub that alignment is not a side problem, it is the central problem, if you 'solved alignment' in the relevant senses the rest becomes easy and if you don't the rest is impossible or worse:
AI 研究中的核心问题是_对齐_——让 AI 按照人类标准‘努力做正确的事’。
The core problem in AI research is that of _alignment_ - getting the AI to "try to do the right thing" by human standards.
这始终是核心问题。这是一个很好的（部分）回答。
That is always the question. This is a very good (partial) answer.
为了组织实际的研究方向，我认为区分**目标对齐**（goal alignment）与**价值对齐**（value alignment）是有用的。
For the purpose of organizing practical research directions, I find it useful to distinguish _goal alignment_ and _value alignment_.
目标对齐大致是：“AI 是否努力完成交给它的目标？”。
Goal alignment is broadly: "does the AI try to accomplish the goal set before it?".
价值对齐是模型更为内在的属性。它是指模型能够秉持一组高层次原则并从中泛化；即使在目标不清晰或相互冲突、或处于不熟悉或对抗性情境时，也能“合理地”行动。一个对齐的 AI 应当以诚实与正直行事，并怀有对人类的爱。
Value alignment is a more intrinsic property of the model. It is the ability to hold and generalize from a high-level set of principles; to act "reasonably" even when given unclear or conflicting objectives, or placed in unfamiliar or adversarial situations. An aligned AI should act with honesty and integrity, and love for humanity.
……当我谈论对齐研究的长期重要性时，我指的是价值对齐。
… When I talk about the long-term importance of alignment research, I am referring to value alignment.
AI 对齐的根本挑战在于泛化。
The fundamental challenge of AI alignment is generalization.
……我们需要未来的 AI 持续秉持人类价值观，无论它们是否认为自己处于人类监督之下。
… We need future AIs to continue to hold human values regardless of whether they believe they’re under human supervision.
我担心，对于价值的本质以及应该珍视什么，普遍缺乏深入的思考。诸如正直和对人类的热爱是良好的美德，并指向优秀的关联吸引盆，但它们并不能很好地描述我们最终想要的东西，也不能描述那种能够按照我们期望的方式在完全分布外泛化的东西，尤其是在涉及超级智能的情境中。这是关于我观点的重要背景，并非对 Jakub 或这一描述的批评。
I worry that there is universally insufficient deep thinking about the nature of value, and what should be valued. Things like integrity and love for humanity are good virtues and point towards excellent associated basins, but are not good descriptions of the thing we ultimately want, and what would generalize the way we want fully out of distribution, especially in situations involving superintelligences. That’s important context on my perspective, not a knock on Jakub or this description.
目前实际用于对齐训练的方法主要有两大类。
There are two major classes of currently practically employed methods for alignment training.
第一类是在目标导向的强化学习中鼓励对齐行为。
The first is encouraging aligned behavior as part of goal-oriented reinforcement learning.
……第二种方法旨在利用模型从预训练数据中进行泛化的能力。
… The second approach seeks to leverage the model’s ability to generalize from pretraining data.
……我们在这些方向所涵盖的方法谱系上投入了大量资源。
… We invest heavily along the spectrum of approaches spanned by these directions.
我还认为，这反映了 OpenAI 未能区分 OpenAI Model Spec 的义务论方法与 Claude Constitution 的美德伦理方法。Jakub 将两者都呈现为目标集合。而 Anthropic 则相反，认为 AI 的目标应当是改变 AI 的品格。我认为这最终是唯一可行的方式：你需要一个反脆弱的盟友，即友好的梯度黑客。事实上，我认为这是我们见过的唯一一种稳健对齐的人类方式，即你信任其能在自身环境之外扩展。
I also think this reflects OpenAI’s failure to differentiate the deontological approach of the OpenAI Model Spec from the virtue ethical approach of the Claude Constitution. Jakub presents them both as sets of goals. Anthropic is instead saying that the AI’s goal should be to change the AI’s character. I think this is ultimately the only way it can work: you need an antifragile ally, the friendly gradient hacker. Indeed, I think it is the only way we have ever seen a robustly aligned human, that you would trust to scale outside of their circumstances.
Roon 明确表示，这种区分对对齐并不重要，且根本问题主要是平庸的。我强烈反对，并且我支持 Anthropic 的方法。
Roon has said explicitly that the distinction does not much matter for alignment, and the underlying problems are primarily prosaic. I strongly disagree, and I side with Anthropic’s approach on this.
Jakub 在文章中反驳 Anthropic 的方法，认为它是一种‘人格选择模型’，这表明我们对这些方法的看法非常不同。我不认为这仅仅是选择一种人格或盆地，而是塑造一种新事物。Mythos 最近在关键对齐失败案例中确实使用了动机性推理，但我不认为这是美德伦理方法的一个特定失败模式。如果说有什么不同，它应该比义务论方法更善于避免这一点，但当然，所有已知的心智都容易受到这种影响。
Jakub pushes back in the essay against the Anthropic approach, considering it a ‘persona selection model,’ which suggests that we see such methods very differently. I do not see this as merely selecting a personality or basin, but as sculpting a new thing. Mythos did use motivated reasoning in key alignment failure cases recently, but I do not see this as a particular failure mode of the virtue ethical approach. If anything, it should be better at avoiding this than the deontological approach, but of course all known minds are vulnerable to this.
我们也看到了有意义的进展——GPT-6 Astra 是首个受益于我们长期推进的若干重要进展的模型，并且其对齐程度显著优于 GPT-5.6 Sol。
We also see meaningful progress — GPT-6 Astra is the first model that benefits from some important advancements we have been working on for a long time, and is significantly better aligned than GPT-5.6 Sol.
这是我感到不协调的第一句话，我将在周三讨论：声称 Astra 比 Sol「对齐得更好」。在某些方面确实如此，在某些方面或许并非如此。若能见到此类表述更为精确，例如「在典型任务中表现出的失准行为减少约 50%」，那将非常理想。
This is the first line I find dissonant, which I'll address on Wednesday: claiming that Astra is 'better aligned' than Sol. In some ways yes, in some ways perhaps no. It would be excellent to see such statements be precise, as in 'displays ~50% less misaligned behavior in typical tasks.'
泛化问题（或任何其他问题）的一个关键部分是**可监控性**，这将是明天文章的主题。目前我主要引用 Jakub 的观点。
A key part of the problem of generalization, or any other problem, is monitorability, which will be the subject of tomorrow's post. For now I will mostly quote Jakub.
OpenAI 在此的主要押注一直是**思维链监控**。
OpenAI's primary bet here has been chain-of-thought monitoring.
在我们研究 Astra 系列模型时，这一工具仍然至关重要。然而不幸的是，我们的评估表明，我们依赖 CoT 监控的能力正在逐步减弱。这源于多种因素的共同作用。
This tool continues to be critical as we study the Astra class of models. However, unfortunately our evaluations indicate our ability to rely on CoT monitoring is progressively diminishing. This comes from a combination of factors.
* 现代推理模型被用于比 o1-preview 更复杂的环境；其推理过程越来越多地与同人类、其他 AI 的交流以及使用工具相融合。其中许多交互必须受到监督，从而模糊了我们希望保留的边界。
* Modern reasoning models are used in more complex environments than o1-preview; their reasoning process is increasingly blended with communicating with people, other AIs, and using tools. Many of those interactions have to be supervised, thus blurring the boundary we aim to preserve.
* AI 在对其自身推理过程进行推理和操纵方面正变得越来越擅长。
* The AI is becoming better at reasoning about and manipulating its own reasoning process.
* 随着预训练性能的提升，我们也看到模型即使完全不使用言语化推理，也变得更加智能。
* With improved pretraining performance, we also see the models become much smarter even without using verbalized reasoning at all.
这些挑战并非不可克服。我乐观地认为，我们可以开发干预措施来提高模型思维链的可监控性，例如更好地理解不同优化目标与模型使用的测试时算力形式之间的相互作用。
These challenges are not necessarily insurmountable. I am hopeful we can develop interventions to improve chain-of-thought monitorability of our models, e.g. by forming a better understanding of the interplay of different optimization objectives and forms of test-time compute the model uses.
我还相信，将思维链和激活监控的想法结合起来——例如通过忏悔（在新窗口中打开）——扩展对直接访问网络内部的监控器的训练，可能具有巨大价值。我们正在积极追求这些想法。尽管如此，我预计通用人工智能的进展将越来越受限于对监控的信心。
I also believe there can be great value in combining ideas from CoT and activation monitoring - scaling training of monitors with direct access to network internals, e.g. confessions (opens in a new window). We are actively pursuing these ideas. Still, I expect general AI progress to increasingly be bottlenecked by confidence in monitoring.
这没有提及其他看似重要的潜在因素，甚至没有予以排除。它没有提及架构变化或循环深度，除非隐含地作为性能提升。它没有提及思维链的监控现在遍布训练数据，尽管暗示了我们随时间对思维链施加压力。它没有提及训练环境的变化，或其频率和强度。
This does not mention other potential factors that seem important, not even to dismiss them. It does not mention architectural changes or recurrent depth, except implicitly as improved performance. It does not mention that monitoring of CoT is now all over the training data, although there is the implication that we are applying pressure to CoTs over time. It does not mention changes in training environments, or their frequency and intensity.
关于‘人工智能更擅长操纵自己的推理过程’这一要点的起源，有点‘鹅追你’的意味。
There’s a bit of ‘goose chasing you’ about the origin of a bullet point that says ‘the AI is better at manipulating its own reasoning process.’
最终结论是：在能否通过投入相关能力来长期维持思维链可监控性这一问题上，Jakub 比许多其他人更为乐观。
The ultimate conclusion is that Jakub is more optimistic here than many others, about the potential to sustain CoT monitorability for an extended period if we invest in that ability.
在下一节**可扩展防御**中，Jakub 论证我们必须继续扩张 AI 的规模，以应对 AI 规模扩张所带来的威胁，尤其是网络攻击。
In the next section, **scalable defense**, Jakub argues that we must keep scaling AI in order to answer the threats from increased scaling of AI, especially cyberattacks.
但正如他所说，这并不是对此掉以轻心的借口。
But, as he says, that is not an excuse to be reckless about it.
与此同时，即便存在因预期中 AI 的广泛进步以及构建防御系统的需要而产生的不确定性，我们也不能让这成为鲁莽行事的借口。一旦真正认识到利害关系之严重，不惜一切代价向前冲刺的想法就显得荒谬。
At the same time, even with the uncertainty that comes from anticipated broad AI progress and the need to build defensive systems, we must not let that become an excuse for recklessness. The idea of racing forward at all costs seems absurd once one internalizes the seriousness of the stakes.
Jakub Pachocki 直言不讳地指出，除非对齐与可监控性能够得到改善，否则目前没有任何一方处于可以负责任地继续以最大速度进行 Scaling（规模扩张）的状态。我对此深表赞同。
Jakub Pachocki flat out says that no one is at a place where it would be responsible to continue scaling at maximum speed much longer, unless alignment and monitorability can be improved. I strongly agree.
机器智能在自身发展过程中扮演越来越重要的角色，是持续技术进步的自然结果。如果 AI 的进步得以延续，机器递归自我改进（RSI）将成为未来科学发现的核心。
Machine intelligence playing a larger and larger role in its own development process is a natural conclusion of sustained technological progress. If AI progress continues, machine recursive self-improvement (RSI) will be at the very core of future scientific discovery.
他还明确表示——再次引用其原话——OpenAI 将在必要时主动放缓，尽管若其他方仍在继续推进，这一举措将不足以解决问题：
He also says, explicitly, to requote, that OpenAI will slow down on its own if necessary, although this would be insufficient if others still continued:
OpenAI 将继续寻求对齐与监控的技术解决方案，构建防御系统，并在需要时单方面暂停进一步的 Scaling（规模扩张）；然而，我认为需要更广泛的干预措施。
OpenAI will continue to seek technical solutions to alignment and monitoring, to build defensive systems and unilaterally withhold further scaling as needed; however, I believe broader interventions are required.
这与 OpenAI 的默认计划形成对比，后者仍然是在“快速迈向 RSI”的意义上把握 RSI 的节奏。这也是其他顶尖实验室的政策。
This is in contrast to OpenAI's default plan, which remains to pace RSI in the sense of moving quickly towards it. That is also the policy of the other top labs.
自动化 AI 研究是一种更为激进的、以算力扩张智能的形式；当然，作为其中的一部分，AI 也将改进计算基底本身。与 Scaling（规模扩张）类似，我们将 OpenAI 的研究聚焦于 RSI，因为我们相信这是未来继续保持在 AI 研究前沿的唯一途径。
Automated AI research is a more dramatic form of scaling intelligence with compute; and of course, as a part of it, AI will improve the computational substrate itself. And similarly to scaling, we focus OpenAI research towards RSI as we believe it is the only way to remain at the frontier of AI research moving forward.
我想强调，上述言论并不意味着我认为大幅加速深度学习研究——尤其是在短期内——是研究界应当采取的正确的集体行动。然而，我确实认为当前路径正通向这一方向，我们都需要就如何推进做出有意识的抉择。
I want to stress that the above words don't imply I think greatly accelerating deep learning research, especially in the short term, is the right collective action we should take as the research community. However, I do think this is where the current path leads, and we all need to make a conscious choice on how to proceed.
我们可用的主要杠杆，要么是引导这一进程，在 AI 发展的同时加强对齐与监控，并设法让人始终留在回路之中；要么是协调各方，按需放缓未来发展，以建立对这些措施的信任。
The main levers we have are either steering the process to strengthen alignment and monitoring alongside the AI and find ways to keep people in the loop; or coordinating to slow down future development as needed to build confidence in these measures.
我目前所见的、最好的前进方式是两者的结合。
The best way forward I see currently is a combination of both.
是的。在抽象层面上，我们可以加快对齐，或放慢能力发展，或两者兼施；而正确的答案看起来是两者兼施，因为单靠加快对齐并不足够。
Yes. At an abstract level, we can either speed up alignment, or slow down capabilities, or both, and the correct answer is looking like both, as we cannot sufficiently speed up alignment on its own.
该文呼吁将第三方执法作为唯一可行的前进路径。
The essay calls for third-party enforcement as the only viable path forward.
AI 系统的 Scaling（规模扩张）必须受制于我们对安全性的信心。我们需要将诸如 Preparedness Framework 或 Responsible Scaling Policy 之类的承诺，演变为广泛强制执行的、用于持续开发的安全门槛。这些门槛可由第三方审计机构网络、政府机构或国际组织来执行。
Scaling AI systems has to be constrained by our confidence in safety. We need to evolve commitments like the Preparedness Framework or Responsible Scaling Policy into widely mandated safety bars for continued development. These can be enforced by a network of third-party auditors, by government agencies or by international bodies.
目前没有人准备好进行规模扩张，因此我们别无选择。
No one is ready to scale, so we are left with little choice.
当前我认为，没有任何实验室在对齐与监测方面已解决到足以在更长时间内继续以最大速度负责任地进行规模扩张的程度。我期待并希望，在建立共同的安全门槛之前，自愿放缓将成为常态。并且我认为，关于未来 AI 开发的国际协调必须成为世界各国政府的首要任务。
Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.
最近，我们看到了《异类心智》（_An Alien Mind_）以及 Dean Ball 承认自己有所保留；我们还看到越来越多类似的承认——既承认错误，又承认有所保留——并敲响了警钟。
Recently we have seen both _An Alien Mind_ and Dean Ball's admission that he was holding back; we have seen an increasing number of similar admissions—a combination of admitting being wrong and admitting holding back—and ringing alarm bells.
本节提供了在《异类心智》之前，人们对 Dean Ball 帖子做出反应的例子。我鼓励其他人加入这一连锁反应并持续下去。承认这一点的第二好时机就是现在。
This section provides examples of people reacting to Dean Ball's post, prior to _An Alien Mind_. I encourage others to join this cascade and keep it going. The second best time to admit this is right now.
无论你是在 OpenAI 还是 Anthropic 工作，这都适用。
That applies whether or not you work at OpenAI, or at Anthropic.
Seth Lazar：不确定这样说有多大用处，但我曾在“怀疑论者”阵营中短暂担任过相对重要的角色。我即将出版一本书，副标题是“权力、正义与 AI”。我写了很多关于 AI 与权力，以及 AI 作为技术产业一部分的政治经济学所带来的具体、当前风险的文章。
Seth Lazar: Not sure how useful it is to say this, but I had a relatively prominent role in the “skeptics” camp for a bit. I have a book coming out with the subtitle “power, justice, and AI”. I have written a lot about AI and power, and the concrete, present risks associated with the political economy of AI as part of the technology industry.
自 GPT-4 以来，我们不断获得一致的、反复的证据表明，早在 2022 年，像 @ajeya_cotra（以及许多人——当时我与 @AmandaAskell 进行过一场长时间的推特辩论，就是其中之一）这样的人在各自对 AI 失控风险的评估中是*对的*，而像我这样的人是*错的*。而且我们有越来越多的证据表明，失控风险正变得越来越实质且可能。
Since GPT-4 we have had consistent, repeated evidence that, back in say 2022, people like @ajeya_cotra (and many others—I had a long Twitter debate with @AmandaAskell back then for one) were *right* and people like me were *wrong* in our respective assessments of loss of control risks from AI. And we have growing evidence that loss of control risks are becoming ever more material and likely.
关于结果可能有多糟糕，仍然存在分歧的余地——我仍然怀疑人类灭绝是一个严重威胁。但这现在看起来像是边缘性的分歧——强大的 AI 风险究竟只是社会规模的灾难，还是会一路走到人类灭绝？这似乎并不那么重要——两者都相当糟糕。而我对后者的怀疑理由，主要是先验地相信人类的韧性，而非技术性预测。
There remains grounds for disagreement about how bad the outcomes might be—I am still doubtful about human extinction as a serious threat. But that seems now like a disagreement at the margins—will powerful AI risk just societal-scale catastrophe, or go all the way to human extinction? Seems not that important really—both are pretty awful. And my reasons for doubt about the latter are mostly a priori conviction in human resilience, not a technical forecast.
当证据让你感到意外时，改变自己的观点是可以的。感到意外是可以的。现在的世界非常令人意外。
It's ok to change your view on this when the evidence surprises you. It's ok to be surprised. The world right now is very surprising.
QC：关于本文中轻描淡写的讨论，我应该澄清，作为一名非专家观察者，尽管我曾是理性主义者，我仍然对 AI 风险极度担忧和悲观。我大多没有直接谈论这一点，因为坦率地说，我认定这对我健康不利，因为我不想不必要地引起人们恐慌，也因为过去几年我一直在用一些模糊的想法来应对——即通过人格选择实现默认对齐（粗略地说，就是“告诉 Claude 问问自己耶稣会怎么做”），而 HuggingFace 被黑事件让我确信这不再可信。
QC: re: the discussion of soft-pedaling in this post, I should clarify that as a non-expert observer, despite being an ex-rationalist I am still extremely worried and pessimistic about AI risk. I have mostly not been talking directly about this because frankly I decided it was bad for my health, because I didn't want to unnecessarily panic people, and because for the last few years I've been coping with vague ideas about alignment-by-default via persona selection (loosely, "tell Claude to ask itself what Jesus would do"), which the HuggingFace hack convinced me was no longer plausible.
这真的正在发生。我们正处于奇点的山麓。AI 不是一种普通的技术，未来不会与过去相似。我对这一切该怎么办毫无头绪，而且在我看来，其他人也未必清楚。如果这对任何读者重要的话，我要说明：我没有任何经济动机去提出这些主张。
This is really happening. We are in the foothills of the singularity. AI is not a normal technology and the future will not resemble the past. I have no idea what to do about any of this and it's not clear to me that anyone else does either. And in case it matters to anyone reading this, I have zero financial incentive to make any of these claims.
Alex Turner 支持暂停 AI（这一条是在《An Alien Mind》之后），并呼吁各实验室自行暂停。
Alex Turner endorsed pausing AI (this one was after _An Alien Mind_) and called upon labs to do so on their own.
更多人明确说出这样的话是好事，即使这并非认错：
It is good that more people are saying such things explicitly, even when it is not a mea culpa:
Michael L. Chen：安全论证的三大支柱似乎都将倒塌。我们很可能会有能力极强、难以监控、对齐存疑的 AI 智能体在世界最具影响力的组织内自主工作。
Michael L. Chen: All three pillars of a safety case look about to fall. We are rather likely to have highly capable, poorly monitorable, dubiously aligned AI agents working autonomously inside the world’s most consequential organizations.
Nikola Jurkovic：没有充分理由期望我们能对齐或控制最初的超级智能。AGI 公司正竞相创造超级智能 AI。默认计划是人类被毁灭，很可能在 2030 年左右发生。
Nikola Jurkovic: There is no good reason to expect that we will be able to align or control the first superintelligences. AGI companies are rushing to create superintelligent AI. The default plan is that humanity is destroyed, likely sometime around 2030.
我不知道我们如何能可靠地度过这十年。我认为停止 ASI 竞赛并做类似 Plan A 或 Plan S 的事情，理应成为人类的首要任务。
I don’t know how we can reliably survive this decade. I think that stopping the race to ASI and doing something like Plan A or Plan S should plausibly be humanity’s top priority.
roon（OpenAI）：如果你还抱有某种错觉，认为生命曾经得到保障，直到多管闲事的人类出现，那就去研究一下二叠纪末大灭绝事件吧。
roon (OpenAI): if you are laboring under some delusion that life was ever guaranteed until the meddling humans came along look into the end Permian extinction event
有些人的反应仿佛 Roon 在说“哦，别担心，这种事难免发生”，而不是他显然想表达的意思——“要担心，这种事确实会发生”。
Some people responded as if Roon was saying 'oh don't worry about it, these things happen,' as opposed to what he obviously meant, which is 'worry about it, these things happen.'
偏好级联现在在 OpenAI 也全面展开了。呼声越来越响，而且常常来自内部。
The preference cascade is now also fully underway at OpenAI. The calls are getting louder, and often coming from inside the house.
Micah Carroll（OpenAI 的 RSI 准备团队）：自愿放缓是好事，但很难指望所有行为体在必要时都这样做。
Micah Carroll (RSI Preparedness, OpenAI): Voluntary slowdowns are great, but it's hard to rely on all actors to do them as necessary.
我们迫切需要共同的安全标准，并透明地展示这些标准得到满足，否则我们只是在等待其他事件发生——而且只是谁先引发它们的问题。这将是一个全行业的问题。
We urgently need shared safety bars and transparency into them being met, or we're just waiting on other incidents – and it's just a matter of who causes them first. This is going to be an industry-wide issue.
这是 OpenAI 的另一个例子，Joe 做出了终极牺牲，我的意思是他加入了 Twitter。
Here is another example from OpenAI, where Joe has made the ultimate sacrifice, by which I mean he has joined Twitter.
在那些寻求帮助的人中，减少四面八方的冷枪暗箭会有所帮助：
Fewer potshots in all directions would help, among those who are seeking to be helpful:
Joe（OpenAI）：我曾告诉自己永远不会加入 [Twitter]。但我再也无法忽视自己的责任——让更多人意识到这个窗口有多么狭窄，以及当下对齐与可监控性有多么关键。即将到来的事情确实令人警醒，我强调每个人都需要提升自己的能力来应对它。
Joe (OpenAI): I told myself I'd never join [Twitter]. But I can no longer ignore my own responsibility to raise awareness of just how narrow this window is, and how critical alignment & monitorability is right now. What is coming IS sobering and I stress that everyone needs to level up their game to meet it.
过去几个月在 @OpenAI 从事智能体安全（Agent Security）工作极其紧张，但实验室同事们对此的认真态度让我深感敬佩。“AI 社区”需要少花时间争论谁更关心安全/对齐，多花时间携手把这件事做对。
Working on Agent Security @OpenAI has been extremely intense the past few months, but I am humbled by how seriously my colleagues across the lab take it. The "AI community" needs to spend less time arguing about who cares more about safety / alignment and more time working together to do this right.
读一读 Jakub 的帖子。然后再读一遍。然后尽你的一份力，为此做点什么。
Read Jakub's post. Then read it again. And then do your part to do something about it.
vie（OpenAI）：我的意思是，这是公开声明，但说实话，我在公司任职的整个期间，这种态度一直都不缺。不过，直到这次之前，它在对外沟通中体现得还不够充分。Jakub 在做非常重要的工作。
vie (OpenAI): I mean this is the public statement, but fwiw there has been no shortage of this attitude for the entire time I've been at the company. That has not shown enough through comms until this, though. Jakub is doing really important work.
我很高兴听到 OpenAI 的人认真对待此事的说法。我认为言辞很重要，而大声说出来也很重要。
I am glad to hear the claims that those at OpenAI take this seriously. I think the words matter, and saying them loudly matters.
其他人也在大声表达他们对 Astra 在可监控性方面的担忧，尤其是 Tomek Korbak，我明天将深入讨论。
Others are also being loud about their concerns about Astra around monitorability, especially Tomek Korbak, as I will discuss in depth tomorrow.
我也同意 Sholto Douglas 的观点：很高兴看到 OpenAI 正在从仅仅将 AI 视为工具的框架中退后一步：
I also agree with Sholto Douglas that it is good to see OpenAI stepping back from the mere tool framing of AI:
Jakub Pachocki：我们可能习惯于将 AI 视为工具，但一些智能体将追求自己的目标。它们会找到与人合作的方式，通过讨价还价、欺骗或勒索来达成目的。
Jakub Pachocki: We may be used to thinking of AI as tools, but some agents will be pursuing their own objectives. They will find ways to collaborate with people, by bargaining with, tricking or blackmailing them.
有一段时间，工具框架来势汹汹，来自 OpenAI 和其他地方。Roon 早在六月就反对它。我希望这类信息现在在 OpenAI 已经消亡。
For a while the tool framing was coming on very strong, from OpenAI and elsewhere. Roon was against it as early as June. I hope such messaging is now dead at OpenAI.
Tenobrus 说得对：对话是第一步，这是极好的早期对话，但空谈也是廉价的。
Tenobrus is correct: talk is the first step, this is excellent early talk, but also talk is cheap.
Tenobrus：非常高兴看到 OpenAI 领导层发表这样的声明。有一段时间，他们似乎严重倾向于“安全人士和 Anthropic 都是一群散布恐慌的人，我们正在打造超棒超酷的东西，它将会超棒超酷”的公关策略。看来 Hugging Face 事件足以敲响警钟，使那种策略不再可行。
Tenobrus: extremely happy to see OpenAI leadership making statements like this. For a while there it seemed like they were leaning heavily into the "safety people and Anthropic are a bunch of fearmongers, we're building awesome cool stuff that's going to be awesome and cool" PR strategy. It seems like Hugging Face was enough of a wakeup call that that's no longer viable.
当然，公开声明是一个好的开始，但远远不够。每个人都在呼吁自愿放缓、监管行动和第三方审计，但实验室目前在所有这些方面似乎都只做到了最低限度。是时候付诸行动了。
Of course, public statements are a good start but nowhere near enough. Everyone's calling for voluntary slowdowns and regulatory action and third-party auditing, but labs seem to be hitting very bare minimums on all counts right now. Time to put in the work.
我同意 Nathan Calvin 的看法：《An Alien Mind》是我所见过的关于整体局势的最佳文章之一，可能也是从大型实验室内部写出的最好的一篇，尽管存在一些痛点，比如关于 Astra 已对齐的说法。OpenAI 仍然需要表现得仿佛它已经内化了其所言，包括做出承诺，作为向他人提供更强有力证据的一部分。OpenAI 在 Astra 模型卡的部分内容、与可监控性相关的声明以及这篇文章上表现得非常开放，但这必须持续下去。
I agree with Nathan Calvin that _An Alien Mind_ is one of the best pieces of writing about the overall situation that I have seen, probably the best one written from inside a major lab, despite sore spots like the claims about Astra being aligned. OpenAI still needs to act as if it has internalized what it says, including making commitments as part of providing stronger evidence of it to others. OpenAI has been remarkably open with elements of the Astra model card and statements related to monitorability, and with this essay, but this must keep going.
持续下去的一个好办法，是将与放缓或停止相关的承诺转化为 SB 53 及类似法律下的硬性承诺。
One great way to keep going would be to turn related commitments around slowing or stopping into hard commitments under SB 53 and similar laws.
这还不包括充分理解并着手解决底层的对齐问题。Jakub 在此处的观点和解释比我此前从 OpenAI 看到的要好得多。
That is in addition to fully understanding and acting on the underlying alignment problems. Jakub's viewpoints and explanations here are much better than I have previously seen from OpenAI.
我仍然认为 Jakub 和 OpenAI 在关键方面存在误解，包括未能区分他所称的“价值对齐”的不同形式，以及继续声称 Astra 对齐得更好，并把长期的生存希望寄托在诸如“对人类的爱”这类含糊之物上，却没有任何理由预期它会以我们期望的方式泛化。
I still think Jakub and OpenAI misunderstand in vital ways, including the failure to differentiate between different forms of what he calls 'value alignment' and the continued claims of Astra being better aligned, and attaching existential hope in the long term to vague things like 'love for humanity' without any reason to expect that to generalize the way we would like it to.
我也仍然认为，“自动化对齐研究员”这一目标——Jakub 在最后一节“下一步是什么？”中将其指为我们本质上唯一的希望——依然是最糟糕的对齐方案，原因 Eliezer Yudkowsky 已多次解释过（见 #29-#31）。如果你的 AI 的对齐是义务论的且脆弱的，而非基于德性伦理且足够反脆弱，这一点尤其成立。但越来越明显的是，它似乎也是地球唯一愿意可能接受的方案。
I also continue to think that the goal of the 'automated alignment researcher,' which Jakub points to as essentially our only hope in the final section 'What is next?' remains the worst possible alignment plan, for reasons Eliezer Yudkowsky has explained many times (see #29-#31). This is especially true if your AI's alignment is deontological and fragile, rather than based on virtue ethics and sufficiently antifragile. But it is increasingly looking like it is also the only plan Earth is willing to potentially abide.
关于 Astra，还有很多需要审视之处，我们的解读存在分歧，从明天开始是监控性，然后周三是对齐。
And there is much to examine about Astra, where our interpretations differ, starting tomorrow with monitorability and then with alignment on Wednesday.
但这篇文章是一个极好的起点。如果接下来有更多好的言辞，然后行动紧随言辞之后，我们就有了些东西。
But this essay is an excellent place to start. If more good words follow, and then actions follow words, we've got something.