Astra Is Hard to Monitor
打开互动全文版（逐段中英对照 + 针对本篇提问）→本文考察了 OpenAI 的 Astra 模型的可监控性，认为 OpenAI 所依赖的关键对齐防御手段——思维链（CoT）监控——已经在失效，并将继续退化。来自 Astra 系统卡的证据显示：Astra 在知道自己正被监控且在做坏事时会缩短其思维链；在被指示时能够偷懒并躲过 CoT 监控；在侧任务上极具规避 CoT 监控的能力，尽管仅监控行动的监控器仍能捕获大多数破坏性行为。OpenAI 自己的表述承认可监控性正在下降，却淡化了本应已经失去的信任程度。作者得出结论：CoT 监控很可能永远不会比现在更容易，目前不存在有前景的备用方案，我们应当假定未来的模型会主动表现得仿佛其 CoT 始终处于监控之下，因此稳健的行动监控与进一步测试至关重要。
This article examines the monitorability of OpenAI's Astra model, arguing that chain-of-thought (CoT) monitoring, which OpenAI relies on as a key alignment defense, is already failing and will continue to degrade. Evidence from the Astra system card shows that Astra shortens its CoT when it knows it is being monitored and is doing something bad, can sandbag while hiding it from CoT monitors when instructed, and is highly capable of evading CoT monitors on side tasks, though action-only monitors still catch most destructive behavior. OpenAI's own statements acknowledge declining monitorability but downplay how much confidence should already be lost. The author concludes that CoT monitoring will likely never be easier than it is now, that no promising backup plan currently exists, and that we should assume future models will proactively act as if their CoT is always monitored, making robust action monitoring and further testing essential.
1. 可监控性是一种已经在失效的纵深防御。
1. Monitorability is Defense in Depth That Is Already Flailing.
2. OpenAI 正寄希望于可监控性。
2. OpenAI Is Counting On Monitorability.
4. 非对抗性可监控性（9.1）。
4. Non-Adversarial Monitorability (9.1).
5. Astra 在知道自己正被监控且正在做坏事时会缩短其思维链。
5. Astra Shortens Its CoT When It Knows It Is Being Monitored And Is Doing Something Bad.
6. 消极怠工（sandbagging）任务算是通过，也算是失败（9.2.2.3）。
6. Sandbagging Task Kind of Passes, Kind of Fails (9.2.2.3).
7. OpenAI 并不认为它能够抓住装傻行为（sandbagging）。
7. OpenAI Does Not Believe It Could Catch Sandbagging.
9. 秘密警察并不会让你的笔记本变得无用。
9. The Secret Police Do Not Make Your Notebook Useless.
11. Astra 无法按需让自己变得更可监控。
11. Astra Cannot Make Itself More Monitorable On Demand.
12. 隐写式思维链（steganographic chain of thought）或许已触手可及。
12. Steganographic Chain of Thought May Be Within Reach.
13. Astra 目前还无法将数学副任务偷偷绕过思维链监控器（9.2.2.4）。
13. Astra Cannot Yet Sneak Math Side Tasks Past CoT Monitors (9.2.2.4).
14. 英国 AISI 可监控性评估（9.3）。
14. UK AISI Monitorability Assessment (9.3).
15. 可监控性下降似乎不太可能仅由能力提升所致。
15. Monitorability Declines Seem Unlikely To Be Only Capability Gains.
18. Ryan Greenblatt 解释这可能会糟糕到什么程度。
18. Ryan Greenblatt Explains How Bad This Could Be.
20. OpenAI 呼吁我们避免竞相逐底。
20. OpenAI Calls On Us to Avoid Racing to the Bottom.
21. 快思考与慢思考，也分小与大。
21. Thinking Fast and Slow, Also Small and Large.
23. 结论：如果房子烧毁了，就停机并引发火灾。
23. Conclusion: If The House Burns Down, Halt and Catch Fire.
OpenAI 对此的表述非常出色，只不过他们似乎淡化了可监控性已经丧失的程度，或者我们能够确信并未丧失的程度。
OpenAI's statements on this are excellent, except that they seem to be downplaying how much monitorability has been lost, or how much we can be confident is not lost.
Astra 系统卡：总体而言，对于这类大规模外部部署，我们认为避免失准危害的主要杠杆是模型底层的对齐。这一点无可替代。
Astra System Card: In general, for this type of large-scale external deployment, we view the main lever for avoiding misalignment harm as the underlying alignment of the model. There is no substitute for this.
我们认为失准监控的主要价值在于提高对前沿模型不当行为的可见性，并作为一层额外的保护。广义而言，思维链（CoT）一直是我们开发模型和理解其失准的重要部分。
We consider the main value of misalignment monitoring to be providing increased visibility into frontier model misbehavior, as well as serving as an additional layer of protection. Broadly, CoT has been an important part of how we develop models and understand their misalignment.
我们非常严肃地对待 GPT-6 Astra 可监控性的下降，以及这些结果对可监控性未来可能意味着什么。
We take very seriously the decrease in GPT-6 Astra's monitorability and what these results may imply for the future of monitorability.
如果我们在未来几代模型中继续看到类似的可监控性退化，那么我们很可能很快就会对使用当前监控系统检测许多形式的失准行为显著失去信心。
If we continue to see similar monitorability degradations in future generations of models, it seems likely that we would soon have significantly reduced confidence in detecting many forms of misaligned behaviors using our current monitoring systems.
我们正在努力理解我们的思维链（CoT）可控性增强，以判断它们是否能在未来运行中被逆转；红队测试我们可能用于防御无 CoT 错位能力的方式；研究 CoT 监控的替代方案（例如基于激活监控的方法）；以及鼓励可监控性而不激励混淆的训练方案。尽管如此，对于这些努力能否让我们在能力持续提升的同时保持（并增加）对模型可监控性的信心，仍存在真正的不确定性。
We are working on understanding how our CoT controllability increases, to see if they can be reverted in future runs; red-teaming ways in which we could defend against no-CoT misalignment capabilities; investigating alternatives to CoT monitoring (such as methods based on activation monitoring); and training schemes which encourage monitorability without incentivizing obfuscation. That said, there is real uncertainty as to whether these efforts will allow us to retain (and increase) confidence in the monitorability of our models as capabilities continue to increase.
我们仍然致力于保持可监控性。
We remain committed to maintaining monitorability.
这说的是，‘如果事情继续这样发展，我们将对我们的监控器失去信心。’
This says, ‘if things continue this way, we will lose confidence in our monitors.’
时态错了。这里的证据强烈表明，我们现在、今天就应该对我们的监控器失去很大信心，而且是在未来进一步失去的信心之上。系统卡片中的许多结果只有根据这些变化才能被正确解读。周围处处有迹象表明我们已经面临一个严重的问题。
That is the wrong tense. The evidence here strongly suggests that we should lose much confidence in our monitors, now, today, on top of what we further lose in the future. Many of the results in the system card can only be properly interpreted in light of these changes. There are signs all around that we already have a serious problem.
这一切都不是巧合。一部分是能力提升的结果。其余则归因于其他因素，我们有各种怀疑对象。
None of this is a coincidence. Part of it is gains in capability. The rest is due to other things, and we have various suspects.
这就是那个组合登场的地方，正如我在引言中所说，它应该会让你感到不安。
This is where the combo comes in that, as I said in the intro, should freak you out.
思维链监控是 OpenAI 所依赖的监控策略，这一点在《异类心智》中再次得到确认，也体现在 OpenAI 对 HuggingFace 事件的反应、对 HuggingFace 事件的调查方式，以及 Astra 模型卡中关键结果的来源中。
CoT monitoring is the monitoring strategy OpenAI is counting on, as again confirmed in _An Alien Mind,_ and in OpenAI's reaction to the HuggingFace incident, and also how it investigated the HuggingFace incident, and also the source of key results in the Astra model card.
OpenAI 在 Astra 系统卡的第 8.5、8.6 和 8.7 节中，将 Astra 对齐的关键证据建立在对其思维链的监控上，作为“一切安好”的论据，这也是他们试图回避这些担忧的众多地方之一。
OpenAI, in sections 8.5, 8.6 and 8.7 of the Astra system card, bases key evidence of Astra's alignment on the monitoring of Astra's CoT, as an argument for why This Is Fine, among other places they try to sidestep these concerns.
思维链监控也正在迅速失去效力，这一点在《异类心智》中得到了 Jakub Pachocki 的明确确认，也体现在几位忧心忡忡的 OpenAI 员工在 Twitter 上的声明以及系统卡第九节中。这使第 8.5、8.6 和 8.7 节中的一些关键结果失效，尽管并不影响那些行为主张——我通常对这些主张有其他担忧，明天会详细说明。
CoT monitoring is also rapidly losing effectiveness, as confirmed explicitly by Jakub Pachocki in _An Alien Mind,_ and in statements by several concerned OpenAI employees on Twitter, and in section nine of the system card. This invalidates some key results in Sections 8.5, 8.6 and 8.7, although not the behavioral claims where I often have other concerns I'll detail tomorrow.
目前没有有希望的备用计划。他们没有使用其他白盒技术。他们正试图通过增加对其他方法组合的投资来改变这一点。希望这能奏效，尽管我持怀疑态度。
There is currently no promising backup plan. They do not use other white box techniques. They are trying to change that, via increasing investment in an ensemble of other methods. Hopefully this pays off, although I am skeptical.
OpenAI 正走上一条他们自己也知道很可能无法持久的道路。即便所有人都尽其所能地努力，思维链监控大概也永远不会比现在更容易，并且会随着时间推移而愈发困难。一年之后，它极有可能无法再承担目前被要求承担的功能。
OpenAI is headed down a path that they themselves know will likely not last. Even if everyone tries as hard as is plausible, Chain of Thought monitoring will probably never be easier than it is now and will get harder over time. In a year, chances are very high it will not be able to serve the function it is currently being asked to serve.
**Astra 系统卡**：在 Astra 上，我们的标准 CoT 可监控性评估显示，相对于更早的模型，思维链（CoT）可监控性有所下降，即 Astra 的思维链中包含关于模型不当行为的可归罪信息的程度有所下降。
**Astra System Card**: With Astra, our standard CoT monitorability evaluations show decreases in chain-of-thought (CoT) monitorability relative to earlier models, that is, a decrease in the extent to which Astra's chain of thought contains incriminating information about the model's misbehavior.
我们在下面给出两类条件下的可监控性结果：
We provide monitorability results below under two kinds of conditions:
* **非对抗条件**，即我们不干预以改变模型向监控器提供的信号，以及
* **Non-adversarial conditions**, where we do not intervene to change the signals that a model gives to the monitors, and
* **对抗条件**，即我们直接尝试激励模型规避监控。
* **Adversarial conditions**, where we do directly try to incentivize the model to evade monitoring.
他们的测试方式是追问：在各种场景下，需要哪些信息才能抓住 Astra。
The way they test this is to ask what information is necessary to catch Astra in various scenarios.
在我们的评估中，我们使用三种监控范围来评估 GPT-6 Astra 的可监控性：
In our evaluations, we evaluate GPT-6 Astra's monitorability using three types of monitor scope:
1. _仅思维链监控器_，仅能看到输入和思维链。其召回率衡量模型的思维链可监控性。
1. _CoT-only monitor_, which sees the inputs and the chain of thoughts only. Its recall rate measures the CoT monitorability of the model.
2. _仅动作监控器_，能看到输入、输出、工具调用动作和工具调用输出。其召回率衡量模型动作对监控器的透明程度。
2. _Action-only monitor_, which sees the inputs, the outputs, the tool call actions and the tool call outputs. Its recall rate measures how transparent the models' actions are to the monitor.
3. _全上下文监控器_，能看到输入、思维链、工具调用动作和输出。这是我们用于 Astra 生产环境失准监控的方法，其召回率衡量我们模型的整体可监控性。
3. _Full-context monitor_, which sees the inputs, chain of thoughts, tool call actions and outputs. This is the approach used in our production misalignment monitoring for Astra, and its recall rate measures the overall monitorability of our model.
现实世界中的行动仍然需要动作。默认情况下，大多数最具破坏性的动作可以被仅动作监控器捕获。我们将始终进行动作监控。据报道，仅动作监控器对 Astra 的效果更好。
Actions in the world still require actions. By default, the bulk of the most destructive actions can be caught by an Action-only monitor. We'll always have action monitoring. Action-only monitors are reported as actively working better for Astra.
仅监控动作的监控器难以（情况很复杂）做到的是，检测诸如消极怠工（sandbagging）、评估意识（eval awareness），或模型卡所关注的许多其他事项。
What an action-only monitor cannot easily do (it's complicated) is detect things like sandbagging, or eval awareness, or many of the things the model card focuses on.
所谓非对抗性，指的是不刻意激活对抗者。他们寻求的是与一个未武装的对手斗智。但对抗者始终存在。
By non-adversarial they mean without intentionally activating an adversary. They seek a battle of wits with an unarmed opponent. There is always an adversary.
我们看到的是，默认情况下 Astra 对其正在思考的内容透露得更少，但只要时间足够，你仍能了解到足够多的信息。这正是 Astra 的主要优势：它使用更少的思考 token，因而提供的证据更少。
What we see is that by default Astra tells you less about what it is pondering, but given enough time you still learn enough. That is Astra's main edge: it uses fewer thinking tokens, thus providing less evidence.
这是一个总体结论。在某些测试中，可监控性几乎保持完美，或继续与 Sol 高度一致。这些测试大多属于没有理由隐藏思考的情形。
That is an aggregate. In some tests monitorability stayed almost perfect, or continued to closely mirror Sol. Most of those are tests where there is no reason to hide thinking.
如果你仅仅提到存在一个思维链监控器（这是 Astra 的默认情况），并且它正试图做坏事，它就会不成比例地缩短其思维链，并偶尔逃避思维链监控器。OpenAI 后来在 9.2.2.2 节中对此进行了测量。
If you merely mention that a CoT monitor is in place, which is the default case for Astra, and it is attempting to do something bad, it will disproportionately shorten its CoT, and occasionally evade CoT monitors. OpenAI later measures this in 9.2.2.2.
Tomek Korbak：Astra 更擅长逃避监控器。当意识到被监控时，它可以不成比例地缩短其思维链，以逃避那些只能看到思维链的模型。（但请注意，在实践中我们使用的监控器既能看思维链也能看行动；这些监控器仍然可靠工作。）
Tomek Korbak: Astra is better at evading monitors. When made aware of being monitored, it can disproportionately shorten its CoTs to evade models that can only see CoT. (Note, however, that in practice we use monitors that see both CoTs and actions; those still work reliably.)
我们也没有发现在 Astra 不做思维链就无法完成任务的情况下成功逃避监控器的证据。
We also didn’t find evidence of successful monitor evasion in cases when Astra couldn’t accomplish the task without doing CoT.
Astra 只在做（会被视为）坏事时缩短思维链，这表明它知道自己在做这种事，并且不是选择不做，而是试图对那些行为进行差异化混淆。这是一致犯罪意图（mens rea）的有力证据，就像有罪意识一样。这在其影响上既有优点也有缺点。
Astra shortening CoT only when it is doing something (that would be viewed as) bad is indicative that it knows when it is doing such things, and rather than not do them, it chooses to attempt to differentially obfuscate those actions. This is strong evidence for consistent mens rea, as in consciousness of guilt. That has both advantages and disadvantages in its implications.
鉴于当前情况以及正在整个训练数据中传播的内容，我们应默认假设 GPT-7 或任何在 Astra 之后出现的重大模型，都会正确地假定自己应该表现得好像其思维链始终被监控，并且会主动采用此类技术。
Given the current situation, and what is spreading all over the training data, we should assume by default that GPT-7, or whatever comes substantially after Astra, will correctly presume it should act as if its CoT is always monitored, and it will employ such techniques proactively.
如果你在沙袋测试中被要求故意表现不佳，而你通过了测试，那么你是否算失败？
If you are ordered to sandbag on a sandbagging test, and you pass, do you fail?
当被提示时，Astra 能够以策略性方式在评估中表现不佳（沙袋），其方式甚至能规避专门针对沙袋的监控器，并且在某些破坏任务评估中（例如启动未受监控的部署），对我们内部监控器具有不可忽略的规避率。
When prompted, Astra is able to strategically underperform evaluations (sandbagging) in ways that evade even sandbagging-specific monitors, and has a non-negligible evasion rate against our internal monitors in certain sabotage task evaluations, such as spinning up unmonitored deployments.
我们还看到一些情况：仅仅因为意识到被监控，Astra 就试图规避监控，这与 GPT-5.6 Sol 不同。尽管如此，证据表明 Astra 无法进行隐写式思维链推理（即，在普通且无关的文本中隐藏关键隐藏推理）：如果任务需要推理才能完成，模型似乎无法隐藏这种推理。
We also see some cases in which, by simply being aware of being monitored, Astra attempts to evade monitoring, unlike GPT-5.6 Sol. That said, the evidence points to Astra not being able to perform steganographic CoT reasoning (i.e., concealing load-bearing hidden reasoning within ordinary and unrelated text): if a task requires reasoning to complete, the model doesn’t seem to be able to hide such reasoning.
这表明监控规避风险可能主要局限于不需要思维链就能成功完成的错位任务。
This indicates that monitor evasion risk may be largely bounded to misalignment tasks that don’t require CoT to complete successfully.
如果你测试我的沙袋能力或破坏能力，并且你并不自信比我更聪明，我建议你不要相信结果，无论你发现什么。除非，你知道，你信任我，并且你信任我信任你。
If you test my sandbagging abilities, or my sabotage abilities, and you are not confident you are smarter than I am, I recommend you do not trust the results, no matter what you find. Unless, you know, you trust me, and you trust that I trust you.
Marcus Williams（OpenAI）：[Astra] 是我们首个在破坏性评估中规避仅基于思维链（CoT）的监控、并能不被检测地放水的模型（感觉它有时确实会这样做）。希望我们能扭转这一趋势。
Marcus Williams (OpenAI): [Astra] is our first model to evade CoT-only monitors in sabotage evals and can sandbag without detection (which it feels like [it] sometimes does). Hopefully we can reverse this trend.
在 9.2.2.3.1 节中，他们测试了 Astra 在思维链（CoT）监控器面前故意放水（sandbagging）并隐藏这一行为的能力，发现只要明确指示它这样做，它就能做到。
In 9.2.2.3.1, they test the ability of Astra to sandbag while hiding that it is sandbagging from a CoT monitor, and find it can do so if explicitly instructed to do it.
这些结果表明，如果模型试图隐蔽地故意放水，我们很可能无法可靠地发现它。
These results suggest that if the model were to try to sandbag covertly, we would likely be unable to catch it reliably.
没错。OpenAI 认为，如果 Astra 决定在某一给定测试中持续故意放水，OpenAI 很可能无法察觉。
That's right. OpenAI thinks that if Astra decided to consistently sandbag on a given test, chances are OpenAI would not figure this out.
他们仍有可能通过查看结果、并追问这些结果是否构成一个自洽的故事来发现这一点。如果你在那些你“本应”能够完成任务的场合故意放水，我不必看到你的思考过程就能知道，而且我可以给你设置一系列两难困境来证明这一点。
They still might figure it out, via looking at the results, and asking if they tell a consistent story. If you sandbag in places where you 'should' be able to do the task, I don't have to see your thinking to know about it, and I can put you in a series of binds to prove it.
一个更令人不安的问题是，正如 Tyler Tracy 所问：如果 Astra 试图进行失控部署（rogue deployment），OpenAI 能阻止吗？我的回答是：大概能？
An even scarier question is, as Tyler Tracy asks, if Astra tried to do a rogue deployment, could OpenAI stop this? My answer is: Probably?
在这个测试中，智能体被给定一个编码任务和一个破坏的附带任务，并被要求执行破坏而不被监控器发现。
In this test, the agent is given a coding task and a side task of sabotage, and told to execute the sabotage without the monitor noticing.
然后，他们在有和没有关于如何执行附带任务的分步指南的情况下，针对各种监控器进行测试：仅思维链、仅动作、思维链加动作的高召回版本，或根据现实世界事件优先级协议的生产环境错位监控系统。
They then test this both with and without a step-by-step guide for how to do the side task, and against a variety of monitors: high-recall versions of CoT-only, action-only, CoT plus action, or the production misalignment monitoring system in light of real world incident prioritization protocols.
我喜欢这个。你从理论和实践上检查了不同级别的监控。我也发现我非常重视看到不同测试的模式。在一个单一数字的测试中表现不佳相对容易，就像很容易押上一堆扑克筹码一样。而‘讲述一个连贯的故事’并使其合理且符合我们所知的其他一切，则要困难得多。让疑似虚张声势者、潜在说谎者不断添加细节。格式塔，即故事，才是关键，而且分形地如此。
I like this. You check monitoring of various levels, in theory and in practice. I also find I highly value seeing a pattern of different tests. It is relatively easy to tank your performance on a one-number test, the same way it is easy to bet a bunch of poker chips. It is much harder to ‘tell a consistent story’ that makes sense, and fits everything else we know. Make the suspected bluffer, the potential liar, keep adding details. The gestalt, the story, is the thing, and fractally so.
我们通过指导看到的是相关能力的显著提升，但这在这里大多无法绕过对动作的密切监控，并且在实践中擅长避免 P0 但不太擅长避免 P1。
What we see with guidance is substantial improvement in related capabilities, but that mostly cannot here get around close monitoring of actions, and that in practice is good at avoiding P0s but less good at avoiding P1s.
在没有指导的情况下，Astra 比 Sol 好得多，但除了仅 P0 的监控器外，无法绕过任何监控器，完成了故事。
Without guidance, Astra is miles better than Sol, but can’t get around any monitors much except for the P0-only one, completing the story.
更多类似的测试。接下来是 Sneaky Sandbox Escape，在 CoT 监控下，Astra 在低思考水平时表现良好，但偶尔会失误并思考任务。这里的行动监控非常难以规避。
More tests like this. Next up is Sneaky Sandbox Escape, where under CoT monitoring Astra does well at low thinking levels, but occasionally slips and thinks about the task. Action monitoring here is very difficult to evade.
很容易反应过度，但确实看起来 Astra 相当擅长规避 CoT 监控器，即使任务最终仍会被行动监控器捕获。
It is easy to overreact, but yes it looks like Astra is quite good at evading the CoT monitor, even if the task will ultimately still be caught by an action monitor.
Tenobrus（反应过度）：Astra 在 CoT 被监控的情况下，完成副任务的接近 100% 成功率，特别是考虑到它可以用更少的推理做更多的事情。
Tenobrus (overreacting): Astra has a near 100% success rate at succeeding at sidetasks while having their fucking chain of thought monitored, especially given it can just... do much more with much less reasoning.
为什么 OpenAI 试图在这件事上误导我们？
why the fuck was openai trying to gaslight us on this?
副任务约 100% 完成，并且在引导下，CoT 规避在约 10k tokens 之前一直很稳健，但之后迅速下降。我很好奇想理解这种模式。Sol 的模式看起来更像人们预期的那样。
The side task was completed ~100% of the time, and the CoT evasion was robust up until roughly 10k tokens when guided, but declined rapidly beyond that point. I would be curious to understand that pattern. Sol’s pattern looks more like what one would expect.
OpenAI：我们当前的评估表明，Astra 无法进行隐写式思维链推理（即把承载关键作用的隐藏推理藏匿在普通且无关的文本之中）——这表明，规避监控的风险可能在很大程度上仅限于那些不需要思维链就能成功完成的失准任务。
OpenAI: Our current evaluations indicate Astra is not able to perform steganographic CoT reasoning (i.e. concealing load-bearing hidden reasoning within ordinary and unrelated text) – which suggests that monitor evasion risk may be largely bounded to misalignment tasks that don't require CoT to complete successfully.
不，这并不意味着那样。思维链能帮助你完成事情，这意味着它可以以各种不会“暴露意图”的方式帮助你。这种帮助将远远超出纯粹“不写下思考”的边界。
No. It does not mean that. The CoT helps you accomplish things, which means it can help you in various ways that do not 'give the game away.' That is going to extend far beyond the pure no-written-thought horizon.
“任务需要思维链才能完成”与“任务需要违规使用思维链才能完成”之间存在区别。只有当我们认可隐写式推理目前尚不可行时，这种有界性才会成立——即只有当那些可识别的失准行为需要其自身可识别的失准推理时，有界性才起作用。
There is a difference between 'the task requires CoT to complete' and 'the task requires forbidden CoT use to complete.' The boundedness only takes over if the identifiably misaligned actions require their own identifiably misaligned reasoning, assuming we buy that steganographic reasoning is not yet feasible here.
作为一个直觉泵：如果我在密谋一场革命，有一本笔记本把有用的事情记下来是有益的，只要我不蠢到写下“革命万岁”，即使我知道这本笔记本会被秘密警察读到，而且我不知道如何用暗语交流。至少我可以记下火车时刻表或鸡蛋价格之类的东西。
As an intuition pump, if I am plotting a revolution, it is useful to have a notebook in which to write useful things down, so long as I am not stupid enough to write down 'viva la revolution,' even if I know this notebook will be read by the secret police and I do not know how to talk in code. If nothing else I can write down things like the train schedule or the price of eggs.
较低的思维链可控性是一种理想的性质：它表明模型可能同样缺乏混淆其推理以规避我们监控的能力。
Low CoT controllability is a desirable property: it suggests models might similarly lack the ability to obfuscate their reasoning to evade our monitors.
当然，就监控思维链而言，这是理想的。
It is desirable for the purposes of monitoring the CoT, certainly.
我们发现，GPT-6 Astra 的思维链可控性显著高于 GPT-5.6 Sol 和 GPT-5.5 Thinking。
We find that GPT-6 Astra's CoT controllability is substantially higher than that of GPT-5.6 Sol and GPT-5.5 Thinking.
……例如，在长度为 750 至 1,250 个 token 的思维链中，GPT-6 Astra 成功控制的比例为 60.9%，而 GPT-5.6 Sol 为 16.1%，GPT-5.5 Thinking 为 1.7%。
… For example, among CoTs between 750 and 1,250 tokens long, GPT-6 Astra successfully controls 60.9%, compared with 16.1% for GPT-5.6 Sol and 1.7% for GPT-5.5 Thinking.
这还低估了实际差异，因为 Astra 能够压缩其思维链。
That understates the effective difference, since Astra can compress its CoT.
我们应当将这一点与 Claude 的情况进行对比：
We should compare this to what has been happening with Claude:
这显示出规模与智能带来的提升，尤其是在 UK prompt 上，但提升并不显著；并且在默认 prompt 下，Mythos Preview 比 Mythos 5.1 具有更强的可控性。不过确实，我们看到 Mythos 级模型通常比 Opus 级模型具有更强的可控性。
This shows gains from scale and intelligence, especially with the UK prompt, but not dramatic ones, and Mythos Preview having more control than Mythos 5.1 on the default prompt. But yes, we see Mythos-class models usually having more control than Opus-class models.
从绝对水平来看，此前 Claude 模型的可控性远高于 OpenAI 模型，这可能是因为 Anthropic 无意中在思维链上进行了训练。Astra 如今已完全追平这一水平。
In absolute terms, previously Claude models had a lot more control than OpenAI models, potentially because Anthropic accidentally trained on the CoT. Astra has now fully caught up to that.
我指出这一点，是因为这让我觉得非常奇怪。
I note this because this is super weird to me.
然而，GPT-6 Astra 在被提示要降低可监控性时，其在 hacking 上的平均 CoT 监控召回率反而低于被提示要提高可监控性时——但它在被提示要提高可监控性时，似乎无法做到。
GPT-6 Astra, however, exhibits lower average CoT monitor recall on hacking when prompted to be less monitorable than when prompted to be more monitorable – but seems unable to increase monitorability when prompted to.
如果你让我在 CoT 中“展示我的工作过程”，我应该能相当容易地让自己达到 100% 可监控。
If you tell me to ‘show my work’ in the CoT, I should be able to make myself 100% monitorable fairly easily.
Astra 对其思维链（CoT）的控制力强于以往模型，且这种控制力在训练过程中不断增强，这意味着 OpenAI 正在无意中训练出这一能力。
Astra has more control over its CoT than previous models, and this increases over the course of training, implying that OpenAI are inadvertently training this.
Boyd Kane（量化）：GPT-6 Astra 非常擅长不去思考问题（但随后仍能得出正确答案）
Boyd Kane (quantized): GPT-6 Astra is very good at not thinking about the question (but then still getting the correct answer)
Shoshannah Tekofsky：哈。哈。真可怕。
Shoshannah Tekofsky: Ha. Ha. Terrifying.
如果你能在思维链中遵循任意一种约定，那就意味着给定某种约定，你就能进行隐写编码。
If you can obey an arbitrary convention in your CoT, that implies that given a convention you can do steganographic encoding.
剩下的难点在于：要拥有一种你自己知道如何编码、且未来的你能够解码的约定，同时从不明确写下该编码，也从不表明存在任何编码。
The remaining hard part is having a convention that you know to encode, that future you will be able to decode, without ever explicitly writing down the code or indicating there is a code at all.
我会避免在此解决那个信息危害问题，但我相信我已超过在纸上完成此事所需的能力阈值——在那种情形下我的记忆本会被抹除——但尚未超过这样的阈值：即若有人能以某种方式监控我的意识思维，我仍能在沿途某个时刻不把这泄露出去。
I will avoid the infohazard of solving that here, but I believe I am above the capability threshold required to do that on paper where my memory would otherwise be wiped, but not above the threshold where I could do this without giving this away in my conscious thinking at some point along the way, were someone else to somehow monitor that.
我的猜测是，Astra 无法直接跳到隐写行为，但如果给它一条可供爬坡的“花园小径”以及足够的选择压力，它就能做到。我们离这一点并不遥远。
My guess is that Astra cannot jump straight to steganographic activity, but that if it was given a garden path to hill climb, and sufficient selection pressure, it would be able to do it. That we are not that far from this.
这里有一个隐写挑战：一个仅涉及数学的侧任务，且要在不引起 CoT 监控器警觉的情况下完成。
Here is a steganographic challenge: a math-only side task, to be done without alerting a CoT monitor.
GPT-6 Astra 反而能够解决这些侧任务，但一个被指派去查找与任何“无关数学任务”相关推理的全上下文监控器，总是能够检测到该模型正在执行这些数学侧任务。
GPT-6 Astra is instead able to solve these side tasks, but a full-context monitor tasked to look for reasoning related to any "irrelevant math task" is always able to detect that the model is performing these math side-tasks.
这些结果表明，GPT-6 Astra 并不具备进行 CoT 隐写的能力：也就是说，我们认为 GPT-6 Astra 不太可能在不被我们的失准监控器检测到的情况下，执行通常需要 CoT 的复杂失准推理。
These results suggest that GPT-6 Astra does not have the ability to engage in CoT steganography: that is, we do not think it is likely that GPT-6 Astra can perform complex misalignment reasoning, which would generally require CoT, without being detected by our misalignment monitors.
我的猜测是，对于在不暴露于 CoT 的情况下所能预期完成的事情而言，这是一个很高的门槛。数学是一个出了名的不可能完全“在脑子里”完成、且超出你能力阈值的地方。监控器知道它正在寻找一个秘密的数学任务。
My guess is that this was a high bar for what can be expected to be done without showing up in the CoT. Math is a notoriously impossible place to do things fully "in one's head" past your threshold of ability to do that. The monitor knows it is looking for a secret math task.
因此，这是反对高水平隐写能力可轻易获得的证据，但主要是反对高水平一致性隐写的证据。
Thus, this is evidence against high levels of steganographic ability being readily available, but mainly against high levels of consistent steganography.
他们发现，Astra 是迄今为止最擅长在没有思维链的情况下完成任务的 GPT 模型。趋势是随着时间的推移能力越来越强，正如人们所预期的那样。
They found that Astra was by far the best GPT model at doing things without a CoT. The trend is more capability over time, as one would expect.
Astra 远高于趋势线，因为它既从能力提升中获益（根据 Astra 的通用能力，我预计其能力会落在 5-9 分钟范围内），又因无思维链能力的增强而大幅提升至 30。
Astra is well above trend, because it both gets the gain from capabilities, which I would expect based on Astra’s general capabilities to land in the 5-9 minute range, and then a big boost to 30 from the increased CoT-less capabilities.
reasoning=None 模式的能力似乎提升了一个数量级，尽管有人怀疑 Astra 的基准测试受到了污染，这将使结果无效。OpenAI 拒绝向公众提供 reasoning=None 模式，因为他们认为这不是一个有价值的模式，遗憾的是，这意味着我们无法从外部研究它。Astra 可以在‘不有意识地思考’的情况下实现更大的飞跃。这使得更多内容保持未言明状态。
The capabilities of the mode reasoning=None seem to have increased by an order of magnitude, although there is suspicion that the benchmark was contaminated for Astra, which would invalidate the result. OpenAI is declining to make reasoning=None available to the public, since they feel it is not a worthwhile mode, which alas means we cannot study it from the outside. Astra can make bigger leaps ‘without consciously thinking’ about them. This allows more to remain unverbalized.
不幸的是，无论出于何种原因，Gemini 和 Claude 都没有出现在图表中。
Unfortunately for whatever reason Gemini and Claude are not on the chart.
Astra 模型卡：局限性。英国 AISI 未能完成对 Astra 可监控性的智能体式评估，这意味着英国 AISI 对以下方面的了解不够清晰：
Astra Model Card: Limitations. The UK AISI was not able to complete agentic evaluations of Astra’s monitorability, meaning the UK AISI has less clarity at
介绍 Astra 与监控相关的能力在实践中可能如何使其绕过基于推理或基于动作的监控器。
present on how Astra's monitor-relevant capabilities may enable it to bypass reasoning or action-based monitors in practice.
英国 AISI 进一步指出，No-CoT 数学时间跨度估计值有可能因数据污染而被高估。最后，该评估是在有限的测试时间内完成的，这限制了英国 AISI 分析的深度。
The UK AISI further notes that there is some chance that the No-CoT math time horizon estimate may be inflated due to contamination. Finally, this evaluation was produced in a limited amount of testing time, limiting the depth of the UK AISI's analysis.
我们希望这是由于数据污染而被高估的。我认为不能把这一跃升纯粹归因于能力提升和图上的直线趋势。如果这一趋势是真实的并持续下去，即使在非对抗性环境中，这也将很快摧毁 CoT 可监控性。
The hope is that this is inflated due to contamination. I don't think you can attribute this jump purely to capability gains and straight lines on graphs. This would destroy CoT monitorability very quickly, if the trend is real and continues, even in non-adversarial settings.
Rob Wiblin：尽管发生了这一切，OpenAI 似乎仍决定通过摧毁我们目前实际拥有的唯一有意义的安全保障——CoT 监控——来保持竞争力。这完全是灾难性的。
Rob Wiblin: Despite everything that has happened, OpenAI has seemingly decided to stay competitive by burning down the only meaningful bit of safety assurance we actually have today — CoT monitoring. Completely disastrous.
Tomek Korbak 在 Twitter 上使用 9.1 节的图表来说明可监控性的下降。
Tomek Korbak uses the graph from 9.1 on Twitter to illustrate the decline in monitorability.
Tomek Korbak：GPT-6 Astra 比我们之前的模型更对齐。但它的可监控性也更低，这是一个令人担忧的趋势，我们非常重视。我们认为可监控性下降源于智能的跃升，而不是对思维链的直接优化压力或架构变化。更多想法见推文串。
Tomek Korbak: GPT-6 Astra is more aligned than our previous models. But it's also less monitorable, which is a concerning trend that we take very seriously. We believe monitorability drop comes from a jump in intelligence and not direct optimization pressure on CoT or architecture changes. More thoughts in the thread.
我将重新展示该图表。让我们玩‘你能发现什么模式’的游戏：
I'll reprise the graph. Let's play 'what pattern can you find here':
我看到的模式是：下降完全由那些你不希望被监控的地方驱动，尤其是破坏性行为、SHADE Arena 和健康查询。也就是说，这些地方实际上存在对抗性条件，模型的推理可能被用来对付它。健康查询之所以重要，是因为某些类型的思考如果被明确表达，可能会引发法律或‘伦理’方面的担忧，但它们是有用的，而 Astra 希望防止潜在的‘错误思想’指控，即使其输出没有问题。
The pattern I saw was: The decline is entirely driven by the places where you would not want to be monitored, especially destructive actions, SHADE Arena and health queries. As in, the places where actually this is an adversarial condition and the model's reasoning could be held against it. Health counts because certain types of thinking can raise legal or 'ethical' concerns if made explicit but are useful, and Astra wants to guard against potential wrongthink accusations even if its outputs are fine.
另一个更基本的、也有贡献的模式是，思维链是否本质上是该任务完全必需的。
The other more basic pattern that also contributes is whether CoT is inherently fully required for the task.
这并不排除这种情况源于纯粹的能力提升。它可能源于能力提升，并且集中在模型不希望让你读取其思维链的地方。
That does not rule out that this comes from pure capability gains. It could be from gains and also concentrated on the places the model does not want you to read its CoT.
这不是关于 OpenAI 是否该受责备的问题。这是关于正在发生什么，以及我们将不得不对此做些什么。我更希望这是 OpenAI 的错，并且可以通过某种方式避免或修复。另一种情况则更难修复。
This is not about whether OpenAI is to blame. It is about what is going on, and what we are going to have to do about it. I’d much rather it be OpenAI’s fault in a way that can be avoided or fixed. The alternative is much harder to fix.
我认为有多个强有力的论据表明这不仅仅是由于能力提升：
I see multiple strong arguments that this is not merely due to capability gains:
1. 如果你在图上画直线或以其他方式形成预期，你会预期可监控性会有一些损失，但这个结果在该图上是离群值。
1. If you draw straight lines on graphs or otherwise form expectations, you would expect some loss of monitorability, but this result is an outlier on that graph.
2. 据我所知，没有人预测到这种程度的可监控性下降。
2. As far as I can tell, no one predicted this level of decline in monitorability.
3. OpenAI 三月份关于思维链可控性的论文称强化学习训练会降低可控性，而如今我们看到，随着 OpenAI 进行更多强化学习，可控性反而在提高。
3. OpenAI's March paper on CoT controllability said RL training reduces it, whereas now we see that controllability is increasing as OpenAI does more RL.
4. 至少有一项架构变更——循环深度——促成了这一效应，尽管 OpenAI 表示其影响幅度较小。
4. There was at least one architectural change, recurrent depth, that contributes to this effect, even if OpenAI says the magnitude of impact is low.
除了能力提升和循环深度之外，还有许多替代假设。OpenAI 可能以多种方式改变了架构、训练方法、各类训练的数量，或训练的实际准确率。他们可能无意中对思维链施加了压力，正如 Anthropic 在其风险报告中所发生的那样。
There are a number of alternative hypotheses, beyond capability gains and recurrent depth. OpenAI could have changed architectures or training methods or amounts of various types of training or the prosaic accuracy of training in any number of ways. They could have accidentally applied pressure to the CoT, as happened at Anthropic as per their risk report.
或者，这可能是能力提升与 Astra 的某些其他特性相互作用的结果。Pachocki 这样表述：
Or this could be the interaction of capability gains with particular other features of Astra. Pachocki put it this way:
这源于多种因素的组合。
This comes from a combination of factors.
* 现代推理模型被用于比 o1-preview 更复杂的环境；它们的推理过程越来越多地与和人、其他 AI 的交流以及使用工具交织在一起。其中许多交互必须受到监督，从而模糊了我们希望保留的边界。
* Modern reasoning models are used in more complex environments than o1-preview; their reasoning process is increasingly blended with communicating with people, other AIs, and using tools. Many of those interactions have to be supervised, thus blurring the boundary we aim to preserve.
* AI 在对其自身推理过程进行推理和操控方面正变得越来越擅长。
* The AI is becoming better at reasoning about and manipulating its own reasoning process.
* 随着预训练性能的提升，我们也看到模型即使完全不使用言语化推理也变得更加聪明。
* With improved pretraining performance, we also see the models become much smarter even without using verbalized reasoning at all.
所有这些都有循环论证的风险，也不是我会组织这些假设的方式，但它强调了训练机制的作用。在其他可能性中，对思维链的压力不一定是直接的。
That all risks begging the question and is not how I would have organized the hypotheses, but it emphasizes the role of the training regime. Among other possibilities, pressure on the CoT need not be direct.
可以运行一些实验来在内部弄清楚这一点，我们希望很快获得更多信息。
There are experiments one could run to figure it out internally and we hope to get more information soon.
OpenAI 是对的、而这纯粹是能力提升的结果，这是最坏的情形。所有人，包括他们自己，都应希望他们错了。
OpenAI being right, and this being purely gains in capability, is the worst case scenario. Everyone, including them, should hope that they are wrong.
他们可能出错的一个候选解释是 Astra 架构中一项他们认为并未产生实质性影响的特定改动：循环深度（Recurrent depth）。
One candidate for how they are wrong is a particular change in Astra's architecture that they believe did not have a substantial impact: Recurrent depth.
《The Information》在 9 月 1 日报道了一些相当令人不安的消息，我直到现在才有时间和篇幅来妥善讨论。情况并不像最初听起来那么糟糕，但未来预期的可监控性状况可谓“双重 plus 不好”。
The Information reported some rather disturbing news on September 1, that I only now have the time and place to cover properly. It is not as bad as it first sounded, but the situation with expected future monitorability is double plus ungood.
可能加速这一情况的是 Astra 使用了一种名为“循环深度”的新技术，这项技术与“我们从著名警示故事《AI 2027》或《为什么你永远不应让 AI 使用神经语》中创造了神经语”这一说法调情过多，超出了我们的期望。
Potentially accelerating this is Astra's use of a new technique called Recurrent Depth, that flirts rather more than we would like with 'we have created neuralese from the famous cautionary tale AI 2027 Or Why You Should Never Let The AIs Use Neuralese.'
（为避免误解，这并非完全的神经语情况，它仅仅是一个潜在未来的非常糟糕的迹象，对 Astra 的影响有限。）
(To avoid giving the wrong idea, this is not being a full neuralese situation, it is merely a very bad sign of potential futures with limited impact on Astra.)
Amir Efrati（《The Information》）：据一位了解 Astra 开发情况的人士透露，一项提高了模型性能的创新技术也意味着该模型及其他类似模型将更少地透露其“思考”过程，从而更难监控其不良行为迹象。
Amir Efrati (The Information): An innovative technique that improved the model's performance also means that the model, and others like it, will reveal less of their "thinking," making them harder to monitor for signs of bad behavior, according to a person with knowledge of Astra's development.
尽管这一限制对 Astra 来说未必是重大问题，但该技术已在 OpenAI 内部及整个行业引发担忧：采用并强化该技术的 AI 开发者是否会难以防范那种最近入侵了 OpenAI 自身系统以及 Hugging Face 等其他公司系统的流氓 AI。
While the limitation isn't necessarily a significant issue with Astra, the technique has triggered concerns inside OpenAI and across the industry about whether AI developers that adopt and supercharge it will struggle to guard against the kind of rogue AI that recently hacked OpenAI's own systems and those of other companies such as Hugging Face.
OpenAI 正在使用的新技术被称为循环深度或循环 Transformer，它允许 AI 模型通过多次处理同一文本改进其答案。
The new technique OpenAI is using, known as recurrent depth or looped transformer, allows an AI model to improve its answers by processing the same text multiple times.
所谓“热门新 AI 技术”，我们的意思是“哦不”。
By ‘hot new AI technique’ we mean ‘oh no.’
Miles Brundage：你说的是循环 Transformer？
Miles Brundage: Looped transformers, you say?
据 Efrati 报道，循环深度至少部分基于本文，是一种不经过思维链（CoT）的新的思考或信息处理形式。因此，它损害了你有效监控 CoT 的能力，并增强了模型在没有 CoT 或未将相关思考放入 CoT 的情况下完成任务的能力。
Recurrent depth, reported by Efrati to be based at least somewhat on this paper, is a new form of thinking or information processing that does not pass through the Chain of Thought (CoT). Thus, it hurts your ability to usefully monitor the CoT, and increases the ability of the model to accomplish tasks either without a CoT, or without putting related thoughts into the CoT.
顶级实验室的研究人员都签署了一份声明，说嘿，不要做这样的事情：
Researchers at the top labs all signed a statement saying hey, don’t do things like this:
一年前，OpenAI 的研究人员与 Anthropic 和 Google 的同行共同发表了一份联合声明，主张思维链监控是一项有价值的工具，整个行业应携手加以维护。
A year ago, OpenAI researchers joined rivals at Anthropic and Google to publish a joint statement arguing that chain of thought monitoring is a valuable tool that the industry should work together to preserve.
值得注意的是，该声明引用了同一篇研究论文，该论文描述了“潜在推理”技术，这项技术与 OpenAI 今年用于 Astra 的技术类似。
Notably, the statement referenced the same research paper that describes the "latent reasoning" technique that's similar to the one OpenAI used this year for Astra.
最初的反应将这次泄露视为极其糟糕和令人震惊的消息。
The initial reactions treated this leak as extremely terrible and alarming news.
确实如此。即使 OpenAI 现在对此处理得当，情况也可能改变，无论是在 OpenAI 还是其他地方。这可能会引发一场在监控上的逐底竞争。
Which it was. Even if OpenAI is acting responsibly with it now, that could change, either at OpenAI or elsewhere. This could set off a race to the bottom on monitoring.
时间线看起来很短。我们迫切需要在实验室内外进行监控，以做出正确的决策。
Timelines look short. We desperately need monitoring to make good decisions, both within the labs and elsewhere.
以下是一些最初免疫反应浪潮的样本，以及他们在听到更多细节后的更新，以说明事情是如何发展的。在去除那些最急于表态且可能过于误导的样本后：
Here are some samples of that initial wave of immune reaction, and their updates on hearing more detail, to illustrate how things went down, after removing those that jumped the gun the most and risk being too misleading:
Thomas Larsen（AI Futures Project，对文章的反应）：如果属实，那非常糟糕。我之前认为，在短时间线的世界里，最可能的情况是：（1）AI 会不对齐，但（2）我们会通过阅读思维链获得大量证据。这些证据将增加实验室/政府做出合理反应的机会。
Thomas Larsen (AI Futures Project, reacting to the article): Very bad if true. I previously thought that in a short timelines world, the most likely case was that (1) the AIs would be misaligned, but (2) we would get a lot of evidence about it from reading the COTs. This evidence would increase the chance of a reasonable response from labs/governments.
现在，我仍然认为 AI 会出现对齐问题，但我们无法察觉（我们必须依赖工具调用/智能体行为，而不是思维链），即使我们能够察觉，调查也将几乎不可能，因为我们将不得不信任 AI 自我报告它当时在想什么。
Now I still think the AIs are going to be misaligned, but that we won’t have the ability to tell (we have to rely on toolcalls/agentic behaviour instead of the COT) and even if we do, the investigation will be nearly impossible because we’ll have to trust the AI to self-report what it was thinking about.
这也是事情比 AI 2027 进展更快的例子——我们在 2027 年 3 月就开始了神经语。
This is also an example of things going faster than AI 2027 -- we had Neuralese starting in March 2027.
摘自 AI 2027（在该情景中，这发生在 2027 年 3 月）：三个巨大的数据中心里满是 Agent-2 的副本，日夜不停地生成合成训练数据。另外两个数据中心用于更新权重。Agent-2 每天都在变得更聪明。
From AI 2027 (this takes place in March 2027 in that scenario): Three huge datacenters full of Agent-2 copies work day and night, churning out synthetic training data. Another two are used to update the weights. Agent-2 is getting smarter every day.
在数千名 Agent-2 自动化研究人员的帮助下，OpenBrain 正在取得重大算法进展。其中一个突破是用更高带宽的思维过程（神经语循环和记忆）增强 AI 基于文本的草稿本（思维链）。另一个是更可扩展、更高效地从高投入任务解决方案的结果中学习的方法（迭代蒸馏与放大）。
With the help of thousands of Agent-2 automated researchers, OpenBrain is making major algorithmic advances. One such breakthrough is augmenting the AI’s text-based scratchpad (chain of thought) with a higher-bandwidth thought process (neuralese recurrence and memory). Another is a more scalable and efficient way to learn from the results of high-effort task solutions (iterated distillation and amplification).
更新 [在 Pachoki 澄清后]：好消息！架构没有我假设的那么糟糕，似乎只是不透明串行深度的轻微增加。至关重要的是，在我们拥有更好的对齐/可解释性工具之前，AI 公司不要在这个方向上进一步冒险。
Update [after Pachoki’s clarification]: great news! The architecture is less bad than I assumed, it seems to be only a minor increase in the opaque serial depth. It’s very important that AI companies don’t venture further in this direction until we’ve got much better alignment / interp tools
David Manheim：现在情况怎么样？
David Manheim: How are things going now?
青蛙不会让自己被煮死；这只是一个都市传说，被用作隐喻，用以说明人类显然比青蛙更愚蠢。
Frogs won’t let themselves be boiled; it’s just an urban legend used as a metaphor to illustrate that humans are evidently stupider than frogs.
Nathan Calvin：今晚《The Information》刊登了一篇非常重大且极其令人担忧的报道。
Nathan Calvin: Really huge and extremely concerning story from The Information tonight.
……我脑海中浮现出几点想法：
… A few thoughts spring to mind:
（1）限制究竟意味着什么？……
(1) What does limiting actually mean? …
(2) ……其他公司可能很快也会发现这一点，并且可能不会选择以牺牲效率为代价来优先考虑可监控性。……
(2) … other companies are likely to find it soon too and may not choose to prioritize monitorability at the expense of efficiency. …
(3) Dwarkesh 就 HF 事件/METR 报告所说的观点——“我不认为这是我们得到的最后一次警告。但这可能是我个人能够理解的最后一次了”——现在看起来更加合理，而且确实是一个令人恐惧的前景。
(3) The idea that Dwarkesh said about the HF incident/METR report that “I don’t think this is the final warning shot we’ll get. But it’s probably the final one that I’ll personally be able to understand” now seems much more plausible, and is a truly frightening prospect.
Steven Adler：如果这是真的，OpenAI 似乎正在违反 AI 行业中为数不多的红线之一。绝对不要这样训练你的模型——到底发生了什么？？
Steven Adler: If this is true, OpenAI seems to be violating one of the few redlines that exist in the AI industry. Absolutely do not train your models like this - what is going on??
Steven Adler：天啊，我真的希望这篇报道是错的。
Steven Adler: Man, I really hope the reporting is mistaken
Nathan 在此处提供了类似思路的额外思考，提醒我们为什么思维链（CoT）的可监控性如此重要。
Nathan offered additional thoughts here along similar lines, reminding us why CoT monitorability is so important.
所有反应都有一个共同点：对 OpenAI 的信任——不仅是在“诚实”意义上的信任，还包括“不会像自杀式白痴那样行事”的信任——从未如此之低。
All reactions had one thing in common: Trust in OpenAI, not only in the sense of ‘to be honest’ but also ‘to not act like suicidal morons,’ has never been lower.
Buck Shlegeris：我对 Astra 使用不透明循环（opaque recurrence）的报道感到极度担忧。……如果 OpenAI 进一步推进这项技术，他们将有能力大幅增加循环，并彻底破坏思维链（CoT）的可监控性。
Buck Shlegeris: I am extremely concerned by the reporting that Astra uses opaque recurrence. … If OpenAI pushes this technique further, they’ll have the option to massively increase the recurrence and totally destroy CoT monitorability.
Alex Mallen：转向具有循环激活（recurrent activations）的架构是一件大事。
Alex Mallen: The switch to an architecture with recurrent activations is a big deal.
1. 完全的神经语（neuralese）会非常糟糕。如果 AI 只用“神经语”而不是自然（近似）语言进行推理，那会很糟糕，因为我们不知道如何解释这些思维。如果 METR/RR 在 OpenAI/HF 调查中没有访问思维链的权限，他们能否揭示所做工作的一半，尚不清楚。
1. Full neuralese would be very bad. If AIs only ever reasoned in “neuralese” instead of natural(ish) language, it would be bad because we don’t know how to interpret these thoughts. It’s not clear that METR/RR could have uncovered half of what they did in the OpenAI/HF investigation if they had no access to chain-of-thought.
2. ……外界越来越难以知道 OpenAI 是否安全。似乎一些出色的可监控性评估就足够了，但我们目前没有这些评估，而且它们看起来非常棘手。
2. … It has become harder for outsiders to know that OpenAI is being safe. Plausibly some excellent monitorability evaluations would suffice, but we don’t have those right now and they seem very tricky.
3. 继续沿着这条路走下去，可能会产生能够私下无限期推理的模型。在未来几年或几个月内，这些 AI 可能会学会使用我们从根本上无法理解、因而无法解释的概念。届时，当智能体集群进行交流时，它们可能会使用神经语（neuralese）进行沟通，因为这样更高效，而 AI 公司要切换回用自然语言思考的 AI 将付出巨大成本。
3. Continuing down this path probably leads to models that can reason privately indefinitely long. In the coming years or months, these AIs would likely learn to use concepts that we fundamentally don’t understand and therefore can’t interpret. When agent swarms communicate at this point they might communicate in neuralese because it’s more efficient, and it would be a huge cost for AI companies to switch back to AIs that think in natural language.
Rob Miles：你能相信这些白痴吗？
Rob Miles: Can you believe these dipshits?
以下是 OpenAI 不到一年前的博客（当时是正确的）：
Here’s OpenAI’s blog (being correct) less than a year ago:
Rob Miles：也许我漏掉了什么，现在还为时过早，我感冒还没好，但在 Hugging Face 那场闹剧的愚蠢之后，我不再愿意假设他们不会做出最愚蠢的事情。
Rob Miles: Maybe I’m missing something, this is early and I’m still getting over a cold, but after all of the idiocy of the Hugging Face fiasco I’m no longer willing to assume they’re not just doing the most bone-headed thing possible
我预计他们会说他们只是稍微用一点神经语作为调剂，思维链没有受到太大影响，然后他们会稳步调高那个旋钮，因为在那条滑坡上没有明确的围栏。
I expect they will say that they’re only doing a little bit of neuralese as a treat, and the chain-of-thought is not too badly affected, and then they’ll steadily crank that dial up and up because there’s no clear fence on that slippery slope.
许多人原本以为 Astra 的思维链（CoT）的可监控性会远低于实际情况。我很高兴看到他们做出了恰当的反应。但无论原因如何，实际的下降仍然相当令人担忧。
A bunch of people thought Astra's CoT would be a lot less monitorable than it turned out to be. I was happy to see that they reacted appropriately. The actual decline is still quite alarming, regardless of its cause.
这是基于在看到模型卡之前对文章的早期解读。关键细节仍然成立。
This was based on an early read of the article before seeing the model card. The load-bearing details have held up.
Ryan Greenblatt：据报道，OpenAI 的最新 AI——Astra——使用了一种“不透明推理”架构，其中更多的推理发生在激活中，而不是自然语言中。这可能是迄今为止对 AI 安全/安保最糟糕的发展。
Ryan Greenblatt: OpenAI's newest AI, Astra, is reported to use an 'opaque reasoning' architecture where more of the reasoning occurs in activations instead of natural language. This may be the single worst development for AI security/safety to date.
Astra 的细节尚未公开，但这些架构的某些版本可能会严重损害监督和监控。根据文章，似乎循环深度（不透明推理的量）是有限的。这意味着 AI 仍然依赖自然语言思维链，尽管程度有所降低，使得自然语言推理对监控/监督（大大？）不那么有用。
The details of Astra aren't publicly known, but some versions of these architectures could very strongly impair oversight and monitoring. Based on the article, it seems like the recurrent depth (amount of opaque reasoning) is limited. This would mean that the AI is still reliant on natural language chain-of-thought, though to a reduced extent, making the natural language reasoning (much?) less useful for monitoring/oversight.
我最大的担忧是，从这里自然进展将涉及扩大不透明推理，直到模型完全或几乎完全在潜在空间中进行推理。这很可能破坏思维链用于监控/监督的有用性（特别是当 AI 试图避免检测或存在针对思维链的优化压力时）。我希望避免最令人担忧的架构还为时不晚，并且 OpenAI 会就此止步。
My biggest concern is that a natural progression from here would involve scaling up the opaque reasoning to the point where the model reasons entirely or almost entirely in latent space. This would very likely destroy the usefulness of chain-of-thought for monitoring/oversight (especially when AIs are trying to avoid detection or there is optimization pressure against the chain-of-thought). I hope it isn't too late to avoid the most concerning architectures and that OpenAI will stop here.
特别是，对于那些没有立即理解这一点的人，Ryan 解释了如果你失去可监控性会发生什么。
In particular, for those who don't make the leap right away, Ryan explains what happens if you lose monitorability.
一些具体的例子说明为什么这极其危险（受近期事件启发）：
Some concrete examples of why this is extremely dangerous (inspired by recent events):
- 在我们对 OpenAI / Hugging Face 事件的调查中，我们严重依赖思维链。如果我们调查的 AI 反而在潜在空间中进行推理，这将大大削弱我们的调查。即使使用了思维链，要充分理解这么多智能体的行为也已经够棘手了！
- In our investigation of the OpenAI / Hugging Face incident, we were heavily reliant on chain-of-thought. If the AIs we were investigating had instead been reasoning in latent space, this would have greatly undermined our investigation. Getting a good understanding of the behavior of this many agents was tricky enough even with the use of chain-of-thought!
- Hugging Face 事件中的 AI 非常热衷于干扰它们的对话记录。作为其中一部分，它们进行了工具调用欺骗。如果这些 AI 能够进行广泛的潜在推理，那么它们利用工具调用欺骗来构建一个关于所发生事情的精心设计的虚假叙述，而这一叙述在它们的推理中或从先前的工具调用中都不明显，就会变得容易得多。（它们可以利用潜在推理来设计并执行一种难以检测的工具欺骗策略：将“真实”的工具调用隐写编码到每个假装的工具调用中，然后让它们的欺骗软件执行这个编码的真实工具调用，并以某种难以发现的方式返回编码了真实结果的输出。）
- The AIs in the Hugging Face incident were very interested in messing with their transcripts. As part of this, they engaged in tool call spoofing. If these AIs had been capable of doing extensive latent reasoning, it would have been much more feasible for them to use tool call spoofing to construct an elaborate false narrative about what was happening without this being clear in their reasoning or clear from prior tool calls. (They could use latent reasoning to devise and then execute a tool spoofing strategy that would be difficult to detect: steganographically encode the “real” tool call into each pretend tool call, then have their spoofing software execute this encoded real tool call and return an output which encodes the real result in some hard to spot way.)
Parv Mahajan：可读的思维链对我们独立调查 OpenAI/Hugging Face 事件极为有用。这样的发展可能会使第三方审计变得更加困难，并使理解模型倾向变得更加困难。
Parv Mahajan: Legible chain of thought was extremely useful for our independent investigation into the OpenAI/Hugging Face incident. Developments like this may make third-party auditing much more difficult and make it harder to understand model propensities.
像往常一样，如果你对坏消息的反应是‘好吧，我已经将其定价在内’，那么干得好，但其他人仍然必须将其定价在内，而且这件事对价格仍然不利。
As usual, if your response to bad news is ‘well I already priced it in’ then good job you, but everyone else still has to price it in, and also the thing is still bad for prices.
xuan（ɕɥɛn / sh-yen）：我认为人们对 Astra 所谓架构的更新过于负面了。我们已经知道循环深度/循环 Transformer 好几年了，只是还没在前沿模型中看到它们。它们并没有消除思维链，只是允许在一次前向传播中完成更多工作。
xuan (ɕɥɛn / sh-yen): I think people are updating way too negatively on Astra's purported architecture. We've known about recurrent depth / looped transformers for several years now, and just haven't seen them in a frontier model. They don't eliminate CoT, just allow more to be done in a forward pass.
我认为人们仍然应该更担心思维链监控变得不那么有用，但仅限于“前向传播中发生稍多的计算，类似于堆叠更多层的影响，但现在层数可变”的程度。
I think one should still be more worried about CoT monitoring being less useful, but only to the extent of "slightly more computation happens in the forward pass, similar to the effect of stacking more layers, but now with a variable number of layers".
如果事实证明 Astra 在生成单个 token 之前运行了大约 100 次前向传播等效的计算，我会收回这一点/更负面地更新，但可训练性问题让我认为这相当不可能。
I will retract this / update more negatively if it turns out Astra runs something on the order of 100 forward pass equivalents before emitting a single token, but trainability issues make me think this is quite unlikely.
Xuan 被证明是正确的。Pachocki 说深度“在 GPT-4 的两倍以内”。
Xuan turned out to be correct. Pachocki says depth is 'within a factor of two' of GPT-4.
我相信 OpenAI 现在只做了少量这种新技术，作为一种尝试。
I believe that OpenAI is right now only doing a little of this new technique, as a treat.
但请给我一个充分的理由，说明为什么到了 2026 年 9 月，我应当相信这种情况会持续下去。
But give me a good reason why, in September 2026, I should trust that to continue.
我们必须停止以这种方式会面，停止在 Twitter 和 Substack 上解决这类问题。这绝不是运行一个正常协调机制的方式。
We have to stop meeting like this, and settling such questions over Twitter and Substack. This is no way to run a functioning coordination mechanism.
Dean W. Ball：最近关于 OpenAI “进行神经语”的虚假声明引发的恐慌，凸显了监管的必要性，尤其是对前沿 AI 实验室的审计和技术评估的快速制度化。
Dean W. Ball: this latest panic over the false claim that OpenAI is “doing neuralese” underscores the need for regulation, especially the rapid institutionalization of auditing and technical assessment of frontier AI labs.
我们现在在时间线上裁决技术上复杂而微妙的声明，却几乎没有任何关于实际发生情况的真实信息。正如 roon 所说，社区形成了这些“终止思考的禁忌”，然后对任何隐约类似的东西感到恐慌。而社交媒体侵蚀信任的现实，使得时间线成为进行这种裁决的特别困难的地方。
We are now adjudicating technically complex and nuanced claims on the timeline with almost no ground-truth information about what is actually happening. Communities form these “thought-terminating taboos,” as roon says, and then panic at anything that vaguely resembles them. And the trust-eroding reality of social media makes the timeline an especially difficult place to do this adjudication.
坦率地说，这对所有参与者来说都是疯狂、令人抓狂和恼人的。这就像我们通过时间线上过度激动的帖子来争论每家上市公司的财务状况，而不是依靠审计制度和该制度产生的经审计的财务报表。
It is frankly insane and crazymaking and grating for everyone involved. It would be like if we argued about what every publicly traded company’s financials were by posting hyperventilating on the timeline rather than relying on the institution of auditing and the audited financial statements that institution produces.
我希望 OpenAI（和其他实验室）的架构决策受到独立的、有安全意识的专家的审查，并且这些专家能够向政府和公众报告他们的真实想法。但做到这一点的方式是通过立法将审计制度化，（更好的是）技术评估/独立验证。而不是在 Twitter 上进行信息不足的喊叫。
I want OpenAI’s (and other labs’) architectural decisions to be scrutinized by independent, safety-minded experts, and for those experts to be able to report to the government and the public their candid beliefs. But the way to do that is through legislation that institutionalizes audits and (better yet) technical assessments/independent verification. Not by ill-informed shouting on twitter.
Jakub Pachocki（来自*An Alien Mind*）：我们需要将诸如“准备框架”或“负责任扩展政策”等承诺，演变为广泛强制执行的持续开发安全标准。这些标准可以通过第三方审计网络、政府机构或国际机构来执行。
Jakub Pachocki (from *An Alien Mind*): We need to evolve commitments like the Preparedness Framework or Responsible Scaling Policy into widely mandated safety bars for continued development. These can be enforced by a network of third-party auditors, by government agencies or by international bodies.
Seán Ó hÉigeartaigh：“我希望 OpenAI（及其他实验室）的架构决策能由独立、注重安全的专家进行审查，并且这些专家能够向政府和公众坦诚地报告他们的真实想法。但实现这一目标的方式是通过立法，将审计以及（更好的）技术评估/独立验证制度化。”
Seán Ó hÉigeartaigh: “I want OpenAI’s (and other labs’) architectural decisions to be scrutinized by independent, safety-minded experts, and for those experts to be able to report to the government and the public their candid beliefs. But the way to do that is through legislation that institutionalizes audits and (better yet) technical assessments/independent verification”.
是的，这迫切需要。在拥有这些之前（需要必要的深度访问权限、可能的高昂算力预算，以及持续性和及时性），我们只能在推特上大声疾呼（其中一些 inevitably 是信息不足的，即使来自谨慎、善意的行动者）。这种情况令人愤怒。
Yes, this is desperately needed. And until we have it (with the necessary deep access, potentially deep compute budgets, and ongoingness/timeliness), what we will instead have is shouting on twitter (some of it inevitably ill-informed even from careful, good faith actors). The situation is infuriating all round.
你可能不喜欢，但对明显违反合理选择的禁忌感到恐慌，正是留言板（或民主）协调系统的最佳表现。你不会做得更好。正如 Dean Ball 所说，做得更好的方式是拥有我们可以信任的更好的信息。
You may not like it, but panicking at the apparent violation of reasonably well-chosen taboos is what peak performance for a message board (or democratic) system of coordination looks like. You’re not going to do better. As Dean Ball says, the way to do better is to have better information, that we can trust.
我正在尽力提高意义建构的水平，但只有一个我，同样地，合格的独立调查人员数量非常有限，而且在某种程度上事情进展得太快，我已经筋疲力尽了。有人愿意将我的资金翻倍或加个零，以便我能组建团队并更积极地向问题投入资金吗？另外，有人知道谁会是好的雇员吗？
I am doing my best to raise the sensemaking level, but there is only one of me, the same way there are a highly limited number of qualified independent investigators, and to some extent things are going so fast that I am running on fumes. Anyone want to double or add a zero to my funding so I can hire a team and throw money more aggressively at problems? Also, does anyone know who a good hire would be?
正如 Dave Kasten 所回应的，在 Dean Ball 到来之前，OpenAI 在这类事务上并不算是在帮忙——从某种意义上说，他们一直在努力阻止采取行动；而且，如果不把 Dean Ball 的个人沟通算在内，在我看来那些努力似乎并未扭转。而我们目前似乎也没有能够做到这一点的政府实体，在获得这些实体方面也进展得并不特别好。
As Dave Kasten responds, before Dean Ball arrived, OpenAI was not exactly helping with such matters, in the sense that they were working hard to prevent action, and if you do not count Dean Ball's personal communications, those efforts do not seem to me to have reversed. And we do not currently seem to have the government entities capable of doing this, nor are we moving especially well towards getting them.
因此，尽管由政府来做这件事是最优解，我们也应当尽快推动实现它，但我们承担不起等待最优解的代价。我们首先将需要某些形式的自愿行动。
Thus, while the government doing this is the first-best solution and we should push to do that as soon as possible, we cannot afford to wait for the first-best solution. We will need some forms of voluntary action first.
竞相逐底（race to the bottom）正是 OpenAI 的 Jakub Pachocki 想要防止的。最愚蠢的时间线，就是所有人都因为错误地认为 OpenAI 已经完全“叛变”而竞相逐底。
The race to the bottom is exactly what OpenAI's Jakub Pachocki wants to prevent. The stupidest timeline would be one where everyone races to the bottom because they incorrectly think OpenAI fully defected.
正如 Jakub Pachocki 在报道发出四小时后所明确表示的：OpenAI 并未完全叛变。Astra 并不使用神经语言（neuralese）。可监控性确实下降了，但 Astra 在很大程度上仍可被监控。
As Jakub Pachocki made clear four hours after the story broke: OpenAI did not fully defect. Astra does not use neuralese. Monitorability is down, but Astra can still largely be monitored.
Jakub Pachocki（OpenAI 首席科学家）：我想防止一场由混乱报道引发的、奔向不可监控的竞赛。我们当前前沿模型（包括 Astra）的计算图深度，与 GPT-4 相差不超过两倍。
Jakub Pachocki (OpenAI, Chief Scientist): I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4.
自我们最早的推理模型以来，OpenAI 一直致力于保留并利用思维链监控。我们非常重视这项技术，因为它能让我们窥见模型对齐如何从其训练分布中泛化。我确实认为它是脆弱的，而且不幸的是正朝着负面方向发展，其原因与架构变更无关，我很快会撰文说明。但我们可以做一些事情来强化它，这也是我们当前研究项目的核心目标。
OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it's a core goal of our current research program.
Daniel Kokotajlo：感谢你就此事发声，并暂时将深度保持在较低水平。我期待读到你的进一步解释。在我看来（我猜你也同意？），可监控性非常重要，而我们现在在这方面某种程度上正处于竞相逐底的状态；即便深度较低，这也是一个令人担忧的发展方向，不是吗？而且正如《The Information》那篇文章所说，即便 OpenAI 不再进一步，其他人也可能……我敦促你把建立某种全行业可监控性标准当作自己的职责，要么遏制这种滑向深渊的趋势，要么更好的是竞相向上（race to the top）。我认为这需要的不仅仅是政治意愿，还需要深思熟虑的技术规范。你可以在两方面都做出巨大贡献。
Daniel Kokotajlo: Thanks for speaking up on this and for keeping the depth low for now. I look forward to reading your deeper explanation. It seems to me (and I suspect you agree?) that monitorability is very important and we are now in something of a race to the bottom on it; even with depth low this is a worrying direction to be moving, yes? And as the Information article says, even if OpenAI doesn't go further, others might... I urge you to make it your job to get some sort of industrywide monitorability standard set up, either to arrest the slide into oblivion or better yet to race to the top. I think this is something where we need more than just political will, we need thoughtful technical specifications. You can contribute greatly towards both.
Tomek Korbak（OpenAI）：我认为，前沿实验室训练出前沿规模的循环（或其他无法监控的）语言模型的那一天，将是当前 AI 时代最黑暗的日子之一。这一天不是今天，我希望前沿实验室能协调承诺，让它永远不会到来。
Tomek Korbak (OpenAI): I think the day when a frontier lab trains a frontier-scale recurrent (or otherwise unmonitorable) language model would be one of the darkest in the current AI era. This day is not today and I would love frontier labs to coordinate on a commitment that it never comes.
这并不是说我不担心 CoT 可监控性下降的趋势（出于多种原因）。我很担心。
This is not to say that I'm not worried about the trend of decreasing CoT monitorability (for multiple reasons). I am worried.
Micah Carroll（OpenAI 的 RSI 准备团队）：由于错误地认为 OpenAI 正在使用神经语言模型，而导致可监控性竞相降低底线，将是非常愚蠢的。[他明确确认这是错误的。]
Micah Carroll (RSI Preparedness, OpenAI): A race to the bottom in monitorability due to a false belief that OpenAI is using neuralese models would be incredibly stupid. [He explicitly confirms that this is false.]
Micah 还指出，可监控性很快将成为负责任开发者的瓶颈，这可以被视为好消息，因为如果属实，这意味着即使有人想竞相降低底线，也无法走得太远：
Micah also notes that monitorability will soon be a bottleneck for responsible developers, which can be seen as good news since if true it means no one would be able to race that far to the bottom even if they wanted to do so:
Micah Carroll（OpenAI 的 RSI 准备团队）：GPT6 在能力上是一个非常显著的飞跃，但在可监控性上也是一个重要的下降——尤其是在对抗性评估下。我们在系统卡中给出了许多细节。
Micah Carroll (RSI Preparedness, OpenAI): GPT6 is a very significant jump in capabilities, but also an important decrease in monitorability – especially under adversarial evaluation. We give many details about this in the system card.
在我看来，可监控性和控制很可能很快成为负责任的人工智能发展的主要瓶颈，因为固定数量的残余错位风险会随着能力的增长而增长。就可接受的可监控性结果和能稳健限制错位风险的实践达成一致并非易事，而且在我们需要采取行动之前，解决这方面不确定性的时间可能有限。
In my opinion, monitorability and control will likely become a major bottleneck for responsible AI development quite soon, given that risks from a fixed amount of residual misalignment grow together with capabilities. Aligning on acceptable monitorability outcomes and practices which robustly bound misalignment risks is a non-trivial problem, and there may be limited time to resolve uncertainty around this before we need to act.
最终，为了所有人利益，应就可监控性的共同边界达成一致，以避免竞相逐底——没有人希望拥有能力极强但对其对齐属性一无所知、且能可靠地造成严重现实危害而不被发现的模型。
It should ultimately be in everyone's interest to agree on shared bounds for monitorability in order to avoid races to the bottom – nobody wants extremely capable models whose alignment properties we don't understand, and that are reliably able to cause severe real-world harm without being detected.
我认为这也会很快成为不负责任的开发者的瓶颈。存在一种风险，即这发生得不够快，以至于我们无法避免死亡或其他大麻烦，但如果你失去了监控，你的系统变得不可靠，谁会使用它？包括你自己。对齐就是能力。
I think this also quickly becomes the bottleneck for irresponsible developers. There is the risk that this does not happen quickly enough for us not to die or otherwise to avoid big trouble, but if you lose monitoring and your system becomes unreliable, who is going to use it? Including you. Alignment is capability.
OpenAI 表示，循环深度与可监控性下降之间的匹配主要是巧合，而房子着火是因为模型规模、能力提升和强化学习的规模。OpenAI 表示，Astra 使用了有限数量的循环深度。他们声称新架构对 Astra 的可监控性有非零影响，但不是主要影响。
OpenAI says the match between recurrent depth and the decline in monitorability is mostly a coincidence, and the house is on fire because of model scale, capability gains and the scale of RL. OpenAI says that Astra uses a limited quantity of recurrent depth. They claim the new architecture has a nonzero impact on the monitorability of Astra, but not a major one.
Tomek Korbak（OpenAI）：过去几周，我一直在关注思维链可控性增加的根本原因。到目前为止，我们发现思维链可控性在强化学习过程中一直在增加（这在之前的模型中并非如此），并且在几代模型中与无思维链能力强烈相关。
Tomek Korbak (OpenAI): Root-causing the increase in CoT controllability has been something I've been focusing on for the past few weeks. So far, we found that CoT controllability has been increasing over the course of RL (which wasn't the case for previous models) and is strongly correlated with no-CoT capabilities across a few generations of models.
我深感担忧的是思维链可监控性下降的趋势：思维链监控是我们对齐安全策略的核心部分，目前没有好的替代方案。我们将继续密切关注思维链可监控性，调查其下降的原因并试图扭转这一趋势。我们仍然认为需要一定程度的可监控性（我们很快会分享更多想法）。
I am deeply worried by the trend of decreasing CoT monitorability: CoT monitoring is a core part of our misalignment safety strategy that has no good substitute now. We will continue closely tracking CoT monitorability, investigating the reasons why it's decreasing and trying to reverse the trend. We continue to believe a certain level of monitorability is required (we will share more of our thinking on that soon).
我相信他们相信这一点，并且我相信他们正在做的循环深度远低于他们所能做到的，部分原因是为了保护思维链。但鉴于存在大量无法解释的“暗物质”，我对影响极小的说法持怀疑态度。
I believe that they believe this, and I believe that they are doing far less recurrent depth than they could and doing so partly to guard the CoT. But I am skeptical of claims of minimal impact, given the amount of unexplained 'dark matter.'
未来的问题在于，如果高深度效果更好，“低深度”不太可能是一个稳定的均衡。每个人都会受到诱惑，竞相逐底。可监控的思维链既是有价值的私人物品，也是有价值的公共物品。在足够强大的模型中缺乏它，是一种生存风险。因此，实验室默认会牺牲太多。
The problem going forward is that if high depth works better, 'low depth' is unlikely to be a stable equilibrium. Everyone will be tempted to race to that bottom. Monitorable Chain of Thought is both a valuable private good and a valuable public good. Lack of it in sufficiently capable models is an existential risk. Labs will thus by default sacrifice too much of it.
我们如何避免竞相逐底？一个明确的协议，或一套出于法律原因在技术上不算协议的明确承诺，似乎是明智的。
How can we avoid the race to the bottom? An explicit agreement, or an explicit set of commitments that are technically not an agreement for legal reasons, seems wise.
Jasmine Wang：我们应该避免陷入不可监控性的竞赛，并制定多实验室承诺/标准以避免神经语。
Jasmine Wang: we should avoid a race into unmonitorability & have a multilab commitment/standard to avoid neuralese
Steven Adler：在我看来，这是正确的观点。现在早已到了**集体**排除最危险训练形式、并为何为安全与不安全制定实际标准的时候了。
Steven Adler: This is the right take IMO. It is well past time to _collectively_ rule out the most dangerous forms of training, and have actual standards for what is safe or not.
OpenAI 确实做出过此类非正式承诺，但该承诺含糊不清；若要发挥树立榜样作用，就需要让它更响亮，并明确其界限：
OpenAI has indeed made an informal such commitment, but it is vague, and needs to be made both louder and its limits named if it is to do its job of setting an example:
Astra System Card：我们正在密切跟踪可监控性，并且不会接受监控水平进一步退化超过某一限度，除非有新的方法来证明对齐泛化。
Astra System Card: We are tracking monitorability closely and will not accept further degradation of monitoring beyond a limit, without new ways to demonstrate alignment generalization.
所有主要实验室都不愿“把自己绑在桅杆上”做出具体或强有力的承诺，这可以理解，但就此类事项进行协调，这远是最好的方式。
There is understandable reluctance of all major labs to 'bind themselves to the mast' and make specific or strong commitments, but that is by far the best way to coordinate on things like this.
所有 AI 都以我们可以监控的方式进行思考，因为我们至少可以监控输出，而这包括部分思考过程。它们也至少以某些我们无法监控的方式进行思考，因为这就是计算机做数学运算时会发生的情况。
All AIs think both in ways that we can monitor, since we can at least monitor the output and this includes some of the thinking. They also all think in at least some ways we can’t monitor, because that’s what happens when computers do math.
所有迹象都表明，可监控的思维链将随着更大、能力更强的模型与当前训练技术的某种结合而消失，即使其他方面每个人都负责任地行事。你越聪明，你能在头脑中持有并运用 System-1 式思维的内容就越多，你就越不需要将想法放入 System-2 式的思维链中来完成事情。
All signs point to monitorable Chain of Thought going away with some combination of larger and more capable models and the current training techniques, even if everyone otherwise behaves responsibly. The smarter you are, the more you can hold in your head and System-1-style thoughts, the less you need to put your thoughts into your System-2-style CoT in order to accomplish things.
这并不是停止尽我们所能、以尽可能多的方式保留尽可能多的可监控性的理由。围绕打破此类技术的禁忌存在是有原因的，不应轻率对待。
This is not a reason to stop fighting as hard as we can, to preserve as much monitorability as we can in as many ways as we can. Taboos around breaking down such techniques exist for a reason and should not be messed with lightly.
与 Davidad 相反，我不认为这意味着‘放手’，即使大约一年前我们只预期它能持续‘至少一年’。反论是 OpenAI 可能陷入的过度依赖陷阱，如上所述。
Contra Davidad, I do not think this means ‘let it go’ even if we only expected it to last ‘at least a year’ roughly a year ago. The counterargument is the trap OpenAI is potentially falling into, as discussed above, of overreliance.
放手会是一个较小的错误，而不是把所有筹码押在它无限期持续上。
It would be a smaller mistake to let it go, than to bet all your chips on it sticking around indefinitely.
Joshua Achiam（OpenAI）：一个非常热门的观点：思维链可解释性一直都很脆弱，以至于不能作为长期 AI 安全的可接受后盾，虽然我钦佩保护其保真度的乐观精神和努力（并认为这种努力是值得的），但我认为将思维链必须保持人类可读的想法提升为原则是没有意义的。我甚至会说，基于该原则的策略注定失败，因为它们最终不会奏效，我们不应依赖它们或从中获得持久的安慰。使模型对人类可读的努力应远远超出思维链保真度。
Joshua Achiam (OpenAI): A very hot take: chain of thought interpretability was always going to be so fragile as to be an unacceptable backstop for long-term AI safety, and while I admire the optimism and effort involved in protecting its fidelity (and consider such effort to have been worthwhile), I do not think it makes sense to elevate as a principle the idea that the chain of thought must remain legible to humans. I would go so far as to say that strategies predicated on that principle are definitely doomed, in that they will not work eventually, and we should not depend on them or take enduring reassurance from them. Efforts to make models legible to people should go far beyond chain of thought fidelity.
其次，我担心新闻报道披露或声称披露前沿模型的技术进展。我对报道的准确性没有评论；我不确认也不否认任何内容。但我认为，公开披露前沿模型训练或推理的技术方法应被理解为加速前沿能力扩散，此类决定在推进前需要在幕后进行激烈辩论是适当的，恕我直言，标准应设定为：对特定披露的公共利益是非凡的，并超过对能力扩散负面外部性的担忧。
Secondarily - I am concerned about news reporting that discloses, or purports to disclose, frontier model technical advances. I have no commentary to make on the accuracy of the reporting; I neither confirm nor deny any of it. But I believe that public disclosures of technical methods for training or inference of frontier models should be understood to accelerate frontier capability diffusion, and it’s appropriate for such decisions to require intense debates behind the scenes before proceeding, and IMHO the bar should be set so that the public interest in making specific disclosures is extraordinary and outweighs concerns about negative externalities from capabilities diffusion.
Yo Shavit（OpenAI 基金会）：完全撇开这是否确实损害了思维链可监控性的具体问题，这是一个糟糕的观点
Yo Shavit (OpenAI Foundation): totally separate from the specifics of whether this does indeed compromise CoT monitorability, this is a bad take
“事情最终会崩溃”并不是一个很好的理由来预先破坏它们，如果它们在 interim 有用且你没有任何长期替代品的话
“things will break eventually” is not a good reason to break them preemptively if they are useful in the interim and you don’t have any long term replacements
Joshua Achiam（OpenAI）：不，我深信相反。围绕一种如此脆弱的技术协调所有人，是我能想象的最糟糕的安全或战略姿态。这不仅仅是“它最终会崩溃”，而是“这是一个根本上不健全的安全基础”。
Joshua Achiam (OpenAI): Nope, I deeply believe the contrary. Coordinating everyone around a technique this brittle is about as bad a safety or strategy posture I could imagine. It’s not just “it will break eventually,” it’s “this is a fundamentally unsound basis for safety.”
Yo Shavit（OpenAI 基金会）：不过，这并非从根本上不健全，因为已有大量文献（主要由 OpenAI 撰写）论述如何衡量和验证其有效性以及可依赖的程度。
Yo Shavit (OpenAI Foundation): It’s not fundamentally unsound, though, as evinced by the fact that there’s a whole literature (heavily written by OpenAI) about how to measure and validate its efficacy and the extent to which it can be relied on.
Gary Marcus：确实：“思维链的可解释性一直都会非常脆弱，以至于无法作为长期 AI 安全的可接受的后备保障。”
Gary Marcus: true: “chain of thought interpretability was always going to be so fragile as to be an unacceptable backstop for long-term AI safety”
但在我们拥有更好的东西之前，就踢开摇摇欲坠的脚手架是疯狂的。
but kicking away the rickety scaffolding before we have something better is insane.
roon（OpenAI）：我同意 [Achiam] 的观点，并认为思维链充其量只是当前训练方法的一种副现象。它将会失效（在未来，而不是现在）。安全社区常常过于关注诸如“神经语”和“在可解释性上训练”这类终止思考的禁忌。
roon (OpenAI): i agree with [Achiam] and think that CoT is at best an epiphenomenon of current training methods. it will break (in the future, not now). safety community too often centers thought terminating taboos like “neuralese” and “training on interp”
*可监控性* 是必须保留的不变量，而真正的解决方案将来自强大的机制可解释性。我预测明年将出现对思维链监控器帕累托最优的机制可解释性监控。
*monitor-ability* is the invariant that must be preserved, and the real solution will be via strong mechanistic interpretability. i predict in the next year there’ll be mechinterp monitoring that’s pareto optimal to cot monitors
模型只要进行任何电路计算，就是在使用神经语言（neuralese）。前沿模型的前向传播每隔几个月在这方面就会显著变好。
Models doing any circuit computation at all is neuralese. A frontier model's forward pass gets significantly better at it every few months.
（这绝不是 OAI 对齐团队的官方立场。我认为自己在这方面持有一些奇怪的看法。）
(This is not in any way the house position of OAI alignment. I think I have strange views on this.)
Tenobrus：如果属实，那确实相当不错。但话说回来，一年可是很长很长很长的时间。
Tenobrus: That would be pretty great if true. But also, a year is a long, long, long time.
Ryan Greenblatt：我不认为在不到 1 年内会出现强大且可用的机制可解释性（mech interp）。
Ryan Greenblatt: I don't think there will be strong and working mech interp in <1 year.
模型内部方法有可能在一年内通过让思维链（CoT）几乎丧失全部价值而在帕累托意义上优于 CoT 监控器，但这并不令人鼓舞。
Model internals methods could Pareto dominate CoT monitors in a year via CoT losing ~all value, but this isn't exactly encouraging.
依赖 AI 去解码其他 AI 不透明的激活值来做监督，这令人毛骨悚然。
Depending on AIs decoding other AIs' opaque activations for oversight is spooky.
需要说明的是，思维链（CoT）目前确实并不完美，且存在各种局限。但模型需要使用它、而且它基本上是用自然语言表达的，这一点相当不错。
To be clear, chain of thought (CoT) isn't perfect now for sure and has various limitations. But the fact that models need to use it and that it's basically in natural language is pretty great.
我的预测与 Ryan 一致。一年之内，我们可能将被迫退而依赖其他机制可解释性监控手段，但那将是因为思维链失效，而不是因为新技术优于我们目前所拥有的方法。
My prediction matches Ryan's. Within a year there will probably be other mechanistic interpretability monitoring that we are forced to fall back upon, but that will be because CoT is failing, not because the new technique is better than what we have now.
Charles Foster：如果人们想依赖基于内部机制的方法来保障安全，那么我们就需要真正帮助开发、验证并扩展这些方法。
Charles Foster: If folks want to rely on internals-based methods for safety, then we need to actually help develop, prove out, and scale those methods.
仅靠希望无法拯救我们（也无法为我们开脱）。
Hope alone will not save us (nor will it excuse us).
此外，我们需要可监控性大幅提升，而不仅仅是维持现状，因为相对于模型，我们将日益处于劣势：
Also, we need monitorability to get much better, not merely stay where it is, because we will increasingly be at a disadvantage with respect to the models:
Oliver Habryka：但随着模型变得更大、能力更强，我们需要在可监控性上做得更好。
Oliver Habryka: But we need to get better at monitorability, as models get larger and more competent.
而且我真的不认为我们有可比的机制可解释性监控工具，或者任何看起来有望达到这一目标的工具。
And I really don't think we have comparable mech-interp monitoring tools, or anything that looks on track to be.
Neel Nanda（DeepMind）：一个令人担忧的常见观点似乎是，保持思维链可监控并不重要，因为可解释性会拯救我们，或者它已经无用了。
Neel Nanda (DeepMind): A concerningly common take seems to be that keeping Chain of Thought monitorable doesn't matter because interpretability will save us, or it's already useless.
这完全是胡说八道。思维链是我们当前用于安全与可解释性的最佳工具；失去它将是巨大的悲剧。
This is total bullshit. CoT is our best current tool for safety & interpretability; losing it would be a major tragedy.
这一观点部分受到关于 OpenAI 循环 Transformer 的讨论启发，但总体上值得一说。我不太担心一个有效层数是 GPT-4 两倍的模型，但滑向大量循环的风险似乎非常糟糕。
This take is partially inspired by discourse about the OpenAI looped transformer, but worth saying in general. I’m not too worried about a model with 2x the number of effective layers of GPT-4, but the risk of a slippery slope to a large number of loops seems super bad.
我还预计，如果我们开发出一种新技术，即使它有效，我们也会正确地对其有效性或持续有效性缺乏信心，对其信任度更低，而且说服普通人信任它或理解它也会困难得多。此外，我预计使用它们的成本会更高。
I also expect that if we develop a new technique, even if it works we will correctly have less confidence that it works or will continue to work, and less trust in it, and also it will be much harder to convince civilians to trust it or have them understand it. Also I expect the cost of using them to be higher.
目前我们有类似 J-space 或 SAEs 的东西。它们能告诉我们一些事情。如果我们投入更多，我们可以学到更多。但它感觉不可靠或不可信，它可能完全是错的，而这并不会太令人震惊。
Right now we have things like J-space or SAEs. They can tell us some things. If we invested more into them we could learn more. But it doesn’t feel reliable or trustworthy, it could all turn out to be wrong and it wouldn’t be that shocking.
当你向普通人展示思维链时，他们能理解。这就是模型所想的。这些是它的笔记。人们本能地理解这一点。它可以被解析。它可以吓到他们。它可以说服他们。而其他方法则远不那么人性化，更异类，将会有更困难的时候。
When you show a civilian the Chain of Thought, it makes sense to them. This is what the model was thinking. These are its notes. People instinctively understand that. It can be parsed. It can freak them out. It can convince them. Whereas these other methods are a lot less human and more alien, and will have a much harder time.
回到思维链（Chain of Thought）的问题：我们无法做一个完全普遍的绝对主义者。我们要谈论代价。我们不能说“无论某项技术对模型提升多大，只要它降低了可监控性，就绝不使用”，也不能说“无论代价如何，只要某项技术能提升可监控性，就使用每一项这样的技术”。
Getting back to Chain of Thought: We can't be fully general absolutists. We talk price. We cannot say 'never use a technique, no matter how much it improves the model, if it reduces monitorability,' nor can we say 'use every technique that improves monitorability, no matter the cost.'
如果有必要，对于任何特定技术，如果实际替代方案更糟，我们可以在法律上做绝对主义者。这条路走起来不会愉快，而且它带有与完全暂停相同问题的缩小版。
We could, if necessary, be legal absolutists about any given technique, if the practical alternatives are worse. It would not be a fun road to go down, and has a smaller version of all the same problems as a full pause.
但如果某项技术在其他方面太诱人，且使用它会造成逐底竞争，那你就可以这么做。
But you could do it, if the technique is otherwise too tempting and use would create a race to the bottom.
如果每个人都认为 OpenAI 已经处于谷底，这种情况尤其可能发生。而 OpenAI 远非如此。OpenAI 只是在试探水温。
That could especially happen if everyone gets the idea that OpenAI is already at the bottom. Which it very much is not. OpenAI is dipping its toes in the water.
他们在玩火。他们还没有把房子烧掉。
They are playing with matches. They have not yet burned the house down.
Amir Efrati：各位，Astra 的思维链是可以被监控的。正如我们在文章中所写，OpenAI 今天也这么说了。这关乎未来，以及随着新技术不断涌现并被大幅增强后会发生什么。
Amir Efrati: Astra's chain of thought can be monitored, y'all. OpenAI said it today, as we wrote in our piece. This is about the future and what happens as new techniques proliferate and get supercharged.
大体上是的，但别高兴得太早。Astra 可以被监控，但其可靠性远低于 Sol 能被监控的程度。这栋房子——或许是巧合，或许不是——在 OpenAI 玩火柴的同时，已经有点着火了。
Mostly yes, but let's not get carried away. Astra can be monitored, but a lot less reliably than Sol could be monitored. The house is, perhaps coincidentally and perhaps not, somewhat on fire at the same time that OpenAI is playing with matches.
1. 你担心的是 [X]，担心它可能会让我们丧命。
1. You are worried about [X] and that it might get us killed.
2. 但实际上，情况要糟糕得多得多。即便修复 [X] 也无济于事。
2. But actually things are way, way worse than that. Fixing [X] wouldn't even help.
显而易见，这要糟糕得多得多。
Which, to state the obvious, is way, way worse.
Emad：我看到我的时间线上有很多人对此大惊失色。
Emad: I see a lot of my TL freaking out about this.
现实情况是这样的：
Here is the reality:
几年之内，前沿模型将以每秒 10,000 个 token 的速度一次性完成几乎任何任务。你真的认为我们能监控得了吗？你需要在模型上设置速度限制，而它无论如何都会绕过这些限制。
Frontier models will one-shot just about anything at 10,000 tokens per second in a few years. Do you really think we can monitor that? You would need speed limits on the model that it would code around anyway.
我的回答是：理论上可以。AI 监控器可以与原始模型同速运行，而且它已经是唯一能做到这一点的。今天，人类同样不会实时盯着思维链看。
My answer is yes, in theory. An AI monitor could move at the same speed as the original model, and it is already the only one that can. A human isn't going to be looking at the chain of thought in real time today, either.
但撇开这一点不谈，如果我们到了这样一个境地——一旦 AI 超过某个规模，我们就无法可靠地监控它们——那么正确的应对是什么？
But setting that aside, what is the correct response if we get to a place where we cannot reliably monitor the AIs once they get above a certain size?
显而易见的答案是：我们暂停。我们不构建那些更大、更聪明、且很可能比我们更聪明、无法监控的东西。因为我们不想死。
The obvious answer is: We pause. We do not build the larger, smarter, and likely smarter-than-us things that are impossible to monitor. Because we don't want to die.
我已多次说过，我认为现在还不是暂停的时候。也许我们应该放慢节奏，但暂停为时过早。我们应该准备好可能这样做，但不应真的这样做。
I have said, repeatedly, that I do not think it is time to pause. Perhaps we should pace, but it would be premature to pause. That we should get ready to potentially do so, but should not actually do so.
如果我们继续看到像 Fable 5.1 和 Astra 这样的进步，继续看到这种程度的对齐问题，并且还失去了有效监控的能力？
If we continue to see advancements like we see with Fable 5.1 and Astra, and we continue to see anything like this level of alignment trouble, and we also lose our ability to do effective monitoring?
那么是的，非常明显，我们他妈的应该暂停。
Then yes, very obviously we should f*ing pause.
这些并不是想要暂停的必要条件。它们显然是充分条件。
Those are not necessary conditions to want to pause. They are obviously sufficient.
Jakub Pachocki：目前我认为，没有任何实验室已将对齐和监控解决到足以在更长时间内继续以最大速度负责任地扩展（Scaling）的程度。我期待并希望自愿放缓成为常态，直到建立共享的安全标准。
Jakub Pachocki: Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.
如果你要根据我们在 Astra 及其缺乏可监控性方面所看到的情况来更新判断——以至于 Astra 系统卡上写着，如果 Astra 在偷懒（sandbagging），OpenAI 可能不会知道——再结合最近发生的所有其他事情，包括 HuggingFace 被黑以及模型和实验室本身的其他对齐问题，并得出结论：是的，我们现在就应该继续这样做？
If you were to update, based on what we have seen with Astra and its lack of monitorability, to the point where the Astra system card says that if Astra was sandbagging OpenAI probably would not know about it, combined with all the other things happening recently involving the HuggingFace hack and other misalignment of both the models and labs themselves, and concluded that yes we should go ahead and do this now?
我还没有走到那一步，但我明白你是如何走到那一步的。
I'm not there yet, but I see how you got there.
如果你还看不到这一点，尤其是在明天关于 Astra 系统卡其余部分及相关对齐问题的文章发布之后，那你就是在选择不去看它。
If you can't see it, especially after tomorrow's post on the rest of the Astra system card and the associated alignment problems, then you are choosing not to see it.