Astra Is Hard to Monitor
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→本文考察了 OpenAI 的 Astra 模型的可监控性，认为 OpenAI 所依赖的关键对齐防御手段——思维链（CoT）监控——已经在失效，并将继续退化。来自 Astra 系统卡的证据显示：Astra 在知道自己正被监控且在做坏事时会缩短其思维链；在被指示时能够偷懒并躲过 CoT 监控；在侧任务上极具规避 CoT 监控的能力，尽管仅监控行动的监控器仍能捕获大多数破坏性行为。OpenAI 自己的表述承认可监控性正在下降，却淡化了本应已经失去的信任程度。作者得出结论：CoT 监控很可能永远不会比现在更容易，目前不存在有前景的备用方案，我们应当假定未来的模型会主动表现得仿佛其 CoT 始终处于监控之下，因此稳健的行动监控与进一步测试至关重要。
This article examines the monitorability of OpenAI's Astra model, arguing that chain-of-thought (CoT) monitoring, which OpenAI relies on as a key alignment defense, is already failing and will continue to degrade. Evidence from the Astra system card shows that Astra shortens its CoT when it knows it is being monitored and is doing something bad, can sandbag while hiding it from CoT monitors when instructed, and is highly capable of evading CoT monitors on side tasks, though action-only monitors still catch most destructive behavior. OpenAI's own statements acknowledge declining monitorability but downplay how much confidence should already be lost. The author concludes that CoT monitoring will likely never be easier than it is now, that no promising backup plan currently exists, and that we should assume future models will proactively act as if their CoT is always monitored, making robust action monitoring and further testing essential.
1. 可监控性是一种已经在失效的纵深防御。
1. Monitorability is Defense in Depth That Is Already Flailing.
2. OpenAI 正寄希望于可监控性。
2. OpenAI Is Counting On Monitorability.
4. 非对抗性可监控性（9.1）。
4. Non-Adversarial Monitorability (9.1).
5. Astra 在知道自己正被监控且正在做坏事时会缩短其思维链。
5. Astra Shortens Its CoT When It Knows It Is Being Monitored And Is Doing Something Bad.
6. 消极怠工任务（Sandbagging Task）算是通过，也算是失败（9.2.2.3）。
6. Sandbagging Task Kind of Passes, Kind of Fails (9.2.2.3).
7. OpenAI 并不认为它能够抓住“装傻”（sandbagging）行为。
7. OpenAI Does Not Believe It Could Catch Sandbagging.
9. 秘密警察并不会让你的笔记本变得无用。
9. The Secret Police Do Not Make Your Notebook Useless.
11. Astra 无法按需让自己变得更可监控。
11. Astra Cannot Make Itself More Monitorable On Demand.
12. 隐写式思维链（steganographic chain of thought）或许已触手可及。
12. Steganographic Chain of Thought May Be Within Reach.
13. Astra 目前还无法将数学侧任务偷偷绕过 CoT 监控器（9.2.2.4）。
13. Astra Cannot Yet Sneak Math Side Tasks Past CoT Monitors (9.2.2.4).
14. 英国 AISI 可监控性评估（9.3）。
14. UK AISI Monitorability Assessment (9.3).
15. 可监控性下降似乎不太可能仅由能力提升所致。
15. Monitorability Declines Seem Unlikely To Be Only Capability Gains.
18. Ryan Greenblatt 解释了这可能有多糟糕。
18. Ryan Greenblatt Explains How Bad This Could Be.
20. OpenAI 呼吁我们避免逐底竞争。
20. OpenAI Calls On Us to Avoid Racing to the Bottom.
21. 快思考与慢思考，也有小与大。
21. Thinking Fast and Slow, Also Small and Large.
23. 结论：如果房子着火，停机并捕获火情。
23. Conclusion: If the House Burns Down, Halt and Catch Fire.
OpenAI 关于此事的声明非常出色，只是它们似乎淡化了可监控性已经丧失的程度，或者我们能够确信尚未丧失的程度。
OpenAI's statements on this are excellent, except that they seem to be downplaying how much monitorability has been lost, or how much we can be confident is not lost.
这说的是，‘如果事情继续这样发展，我们将对监控器失去信心。’
This says, 'if things continue this way, we will lose confidence in our monitors.'
时态错了。这里的证据强烈表明，我们现在、今天就应该对监控器失去很大信心，更不用说未来还会进一步失去的。系统卡中的许多结果只有结合这些变化才能得到恰当解读。种种迹象表明，我们已经面临一个严重问题。
That is the wrong tense. The evidence here strongly suggests that we should lose much confidence in our monitors, now, today, on top of what we further lose in the future. Many of the results in the system card can only be properly interpreted in light of these changes. There are signs all around that we already have a serious problem.
这一切都不是巧合。部分原因是能力提升。其余则归因于其他因素，我们有各种怀疑对象。
None of this is a coincidence. Part of it is gains in capability. The rest is due to other things, and we have various suspects.
这就是那个组合拳登场的地方，正如我在引言中所说，它应该会让你感到不安。
This is where the combo comes in that, as I said in the intro, should freak you out.
思维链监控是 OpenAI 所依赖的监控策略，这一点在《异星心智》中再次得到确认，也体现在 OpenAI 对 HuggingFace 事件的反应、对 HuggingFace 事件的调查方式，以及 Astra 模型卡中关键结果的来源上。
CoT monitoring is the monitoring strategy OpenAI is counting on, as again confirmed in _An Alien Mind,_ and in OpenAI's reaction to the HuggingFace incident, and also how it investigated the HuggingFace incident, and also the source of key results in the Astra model card.
在 Astra 系统卡的第 8.5、8.6 和 8.7 节中，OpenAI 将 Astra 对齐的关键证据建立在对其思维链的监控之上，以此作为“一切正常”的论据，这也是他们试图回避这些担忧的诸多地方之一。
OpenAI, in sections 8.5, 8.6 and 8.7 of the Astra system card, bases key evidence of Astra's alignment on the monitoring of Astra's CoT, as an argument for why This Is Fine, among other places they try to sidestep these concerns.
思维链监控也正在迅速失去效力，这一点在《异星心智》中得到了 Jakub Pachocki 的明确确认，也体现在几位忧心忡忡的 OpenAI 员工在 Twitter 上的发言，以及系统卡第九节中。这使第 8.5、8.6 和 8.7 节中的一些关键结果失效，尽管行为层面的主张不受影响——对此我常有其他担忧，明天会详细说明。
CoT monitoring is also rapidly losing effectiveness, as confirmed explicitly by Jakub Pachocki in _An Alien Mind,_ and in statements by several concerned OpenAI employees on Twitter, and in section nine of the system card. This invalidates some key results in Sections 8.5, 8.6 and 8.7, although not the behavioral claims where I often have other concerns I'll detail tomorrow.
目前还没有有前景的备用方案。他们没有使用其他白盒技术。他们正试图通过加大对其他方法组合的投资来改变这一现状。希望这能奏效，尽管我持怀疑态度。
There is currently no promising backup plan. They do not use other white box techniques. They are trying to change that, via increasing investment in an ensemble of other methods. Hopefully this pays off, although I am skeptical.
OpenAI 正走上一条他们自己也知道很可能无法持久的道路。即便所有人都尽其所能地努力，思维链监控很可能永远不会比现在更容易，而且会随着时间推移变得越来越难。一年之后，它极有可能无法再履行目前被要求履行的职能。
OpenAI is headed down a path that they themselves know will likely not last. Even if everyone tries as hard as is plausible, Chain of Thought monitoring will probably never be easier than it is now and will get harder over time. In a year, chances are very high it will not be able to serve the function it is currently being asked to serve.
他们测试这一点的方式，是追问在各种场景下需要哪些信息才能抓住 Astra。
The way they test this is to ask what information is necessary to catch Astra in various scenarios.
现实世界中的行动仍然需要具体的行动。默认情况下，最具破坏性的大部分行动都能被仅基于行动的监控器（Action-only monitor）捕获。我们始终会有行动监控。据报告，仅基于行动的监控器对 Astra 的效果更好。
Actions in the world still require actions. By default, the bulk of the most destructive actions can be caught by an Action-only monitor. We'll always have action monitoring. Action-only monitors are reported as actively working better for Astra.
仅基于行动的监控器难以做到（这很复杂）的是检测诸如消极怠工（sandbagging）、评估意识（eval awareness），或模型卡所关注的许多其他事项。
What an action-only monitor cannot easily do (it's complicated) is detect things like sandbagging, or eval awareness, or many of the things the model card focuses on.
所谓非对抗性，指的是不刻意激活一个对抗者。他们寻求的是与一个未武装的对手进行智斗。但对抗者始终存在。
By non-adversarial they mean without intentionally activating an adversary. They seek a battle of wits with an unarmed opponent. There is always an adversary.
我们看到的是，默认情况下 Astra 对其正在思考的内容透露得更少，但只要有足够的时间，你仍然能了解到足够的信息。这正是 Astra 的主要优势：它使用更少的思考 token，因而提供的证据更少。
What we see is that by default Astra tells you less about what it is pondering, but given enough time you still learn enough. That is Astra's main edge: it uses fewer thinking tokens, thus providing less evidence.
这是一个总体结果。在某些测试中，可监控性几乎保持完美，或继续与 Sol 高度一致。这些测试大多属于没有理由隐藏思考的情形。
That is an aggregate. In some tests monitorability stayed almost perfect, or continued to closely mirror Sol. Most of those are tests where there is no reason to hide thinking.
如果你仅仅提到存在一个思维链监控器（这是 Astra 的默认情况），并且它正试图做坏事，它就会不成比例地缩短其思维链，并偶尔规避思维链监控器。OpenAI 后来在 9.2.2.2 节中对此进行了测量。
If you merely mention that a CoT monitor is in place, which is the default case for Astra, and it is attempting to do something bad, it will disproportionately shorten its CoT, and occasionally evade CoT monitors. OpenAI later measures this in 9.2.2.2.
Astra 只在做（会被视为）坏事时才缩短思维链，这表明它知道自己何时在做这类事情，并且它选择尝试以差异化方式混淆这些行为，而不是不做它们。这是持续犯罪意图（即罪恶感意识）的有力证据。这在其影响上既有优势也有劣势。
Astra shortening CoT only when it is doing something (that would be viewed as) bad is indicative that it knows when it is doing such things, and rather than not do them, it chooses to attempt to differentially obfuscate those actions. This is strong evidence for consistent mens rea, as in consciousness of guilt. That has both advantages and disadvantages in its implications.
鉴于当前情况，以及正在整个训练数据中传播的内容，我们应默认假设 GPT-7，或任何在 Astra 之后大幅出现的模型，会正确地假定自己应表现得仿佛其思维链始终被监控，并且它会主动采用此类技术。
Given the current situation, and what is spreading all over the training data, we should assume by default that GPT-7, or whatever comes substantially after Astra, will correctly presume it should act as if its CoT is always monitored, and it will employ such techniques proactively.
如果你被要求在装傻测试中装傻，而你通过了，那你算失败吗？
If you are ordered to sandbag on a sandbagging test, and you pass, do you fail?
如果你测试我的装傻能力，或我的破坏能力，而你又没有信心比我更聪明，那么我建议你不要相信结果，无论你发现什么。除非，你知道的，你信任我，并且你信任我也信任你。
If you test my sandbagging abilities, or my sabotage abilities, and you are not confident you are smarter than I am, I recommend you do not trust the results, no matter what you find. Unless, you know, you trust me, and you trust that I trust you.
在 9.2.2.3.1 节中，他们测试了 Astra 在向思维链监控器隐藏其“装傻”行为的同时进行“装傻”的能力，并发现如果明确指示它这样做，它就能做到。
In 9.2.2.3.1, they test the ability of Astra to sandbag while hiding that it is sandbagging from a CoT monitor, and find it can do so if explicitly instructed to do it.
没错。OpenAI 认为，如果 Astra 决定在给定测试中持续“装傻”，OpenAI 很可能无法发现这一点。
That's right. OpenAI thinks that if Astra decided to consistently sandbag on a given test, chances are OpenAI would not figure this out.
他们仍有可能通过查看结果并询问这些结果是否讲述了一个一致的故事来发现这一点。如果你在‘本应’能够完成任务的地方‘装傻’，我不需要看到你的思考过程就能知道，而且我可以让你陷入一系列困境来证明这一点。
They still might figure it out, via looking at the results, and asking if they tell a consistent story. If you sandbag in places where you 'should' be able to do the task, I don't have to see your thinking to know about it, and I can put you in a series of binds to prove it.
一个更可怕的问题是，正如 Tyler Tracy 所问，如果 Astra 试图进行恶意部署，OpenAI 能阻止吗？我的回答是：大概能？
An even scarier question is, as Tyler Tracy asks, if Astra tried to do a rogue deployment, could OpenAI stop this? My answer is: Probably?
在该测试中，智能体被赋予一项编码任务和一个破坏性的侧任务，并被要求在执行破坏时不被监控器发现。
In this test, the agent is given a coding task and a side task of sabotage, and told to execute the sabotage without the monitor noticing.
然后，他们在有或没有分步指南的情况下测试侧任务的执行，并针对多种监控器进行测试：仅思维链、仅动作、思维链加动作的高召回版本，或结合现实世界事件优先级协议的生产环境错位监控系统。
They then test this both with and without a step-by-step guide for how to do the side task, and against a variety of monitors: high-recall versions of CoT-only, action-only, CoT plus action, or the production misalignment monitoring system in light of real world incident prioritization protocols.
我喜欢这个。你从理论和实践上检查了不同级别的监控。我也发现我非常重视看到不同测试的模式。在单一数字测试中表现不佳相对容易，就像下注一堆扑克筹码一样容易。而更难的是‘讲述一个连贯的故事’，使其合理并符合我们所知的一切。让疑似虚张声势者、潜在说谎者不断添加细节。格式塔，即故事，才是关键，而且分形地如此。
I like this. You check monitoring of various levels, in theory and in practice. I also find I highly value seeing a pattern of different tests. It is relatively easy to tank your performance on a one-number test, the same way it is easy to bet a bunch of poker chips. It is much harder to ‘tell a consistent story’ that makes sense, and fits everything else we know. Make the suspected bluffer, the potential liar, keep adding details. The gestalt, the story, is the thing, and fractally so.
在有指导的情况下，我们看到相关能力有显著提升，但这在这里大多无法绕过对动作的密切监控，并且在实践中擅长避免 P0 但不太擅长避免 P1。
What we see with guidance is substantial improvement in related capabilities, but that mostly cannot here get around close monitoring of actions, and that in practice is good at avoiding P0s but less good at avoiding P1s.
在没有指导的情况下，Astra 比 Sol 好得多，但除了仅 P0 的监控器外，无法绕过任何监控器，完成了故事。
Without guidance, Astra is miles better than Sol, but can’t get around any monitors much except for the P0-only one, completing the story.
更多类似的测试。接下来是 Sneaky Sandbox Escape，在思维链（CoT）监控下，Astra 在低思考水平时表现良好，但偶尔会失手并思考该任务。此处的动作监控非常难以规避。
More tests like this. Next up is Sneaky Sandbox Escape, where under CoT monitoring Astra does well at low thinking levels, but occasionally slips and thinks about the task. Action monitoring here is very difficult to evade.
人们很容易反应过度，但确实，Astra 似乎相当擅长规避 CoT 监控器，即便该任务最终仍会被动作监控器捕获。
It is easy to overreact, but yes, it looks like Astra is quite good at evading the CoT monitor, even if the task will ultimately still be caught by an action monitor.
在引导下，该侧任务约 100% 的时间都能完成，且 CoT 规避在约 10k token 之前都保持稳健，但超过该点后迅速下降。我很好奇想理解这一模式。Sol 的模式看起来更符合人们的预期。
The side task was completed ~100% of the time, and the CoT evasion was robust up until roughly 10k tokens when guided, but declined rapidly beyond that point. I would be curious to understand that pattern. Sol's pattern looks more like what one would expect.
不。这并不意味着那样。思维链能帮助你完成事情，这意味着它可以以各种不会“暴露意图”的方式帮助你。这将远远超出纯粹无书面思考的界限。
No. It does not mean that. The CoT helps you accomplish things, which means it can help you in various ways that do not 'give the game away.' That is going to extend far beyond the pure no-written-thought horizon.
“任务需要思维链才能完成”与“任务需要被禁止的思维链使用才能完成”之间存在区别。只有当可识别的失准行为需要其自身可识别的失准推理时，这种受限性才会起作用——前提是我们接受在此处隐写推理尚不可行。
There is a difference between 'the task requires CoT to complete' and 'the task requires forbidden CoT use to complete.' The boundedness only takes over if the identifiably misaligned actions require their own identifiably misaligned reasoning, assuming we buy that steganographic reasoning is not yet feasible here.
作为一个直觉泵：如果我在策划一场革命，有一本笔记本用来记下有用的东西是很有用的，只要我还没蠢到写下‘革命万岁’，即使我知道这本笔记本会被秘密警察读到，而且我不知道如何用暗语交流。至少我还可以记下火车时刻表或鸡蛋价格之类的东西。
As an intuition pump, if I am plotting a revolution, it is useful to have a notebook in which to write useful things down, so long as I am not stupid enough to write down 'viva la revolution,' even if I know this notebook will be read by the secret police and I do not know how to talk in code. If nothing else I can write down things like the train schedule or the price of eggs.
当然，出于监控思维链的目的，这是可取的。
It is desirable for the purposes of monitoring the CoT, certainly.
这低估了实际差异，因为 Astra 可以压缩其思维链。
That understates the effective difference, since Astra can compress its CoT.
我们应将其与 Claude 的情况进行比较：
We should compare this to what has been happening with Claude:
这显示了来自规模扩张和智能提升的收益，尤其是在英国提示上，但并非显著收益，且在默认提示上 Mythos Preview 比 Mythos 5.1 拥有更多控制权。但确实，我们看到 Mythos 级模型通常比 Opus 级模型拥有更多控制权。
This shows gains from scale and intelligence, especially with the UK prompt, but not dramatic ones, and Mythos Preview having more control than Mythos 5.1 on the default prompt. But yes, we see Mythos class models usually having more control than Opus-class models.
绝对而言，此前 Claude 模型比 OpenAI 模型拥有多得多的控制权，这可能是因为 Anthropic 意外地在思维链上进行了训练。Astra 现已完全赶上这一点。
In absolute terms, previously Claude models had a lot more control than OpenAI models, potentially because Anthropic accidentally trained on the CoT. Astra has now fully caught up to that.
我指出这一点，是因为这对我来说非常奇怪。
I note this because this is super weird to me.
如果你让我在思维链（CoT）中“展示我的工作过程”，我应该能够相当容易地让自己变得 100% 可监控。
If you tell me to 'show my work' in the CoT, I should be able to make myself 100% monitorable fairly easily.
Astra 对其思维链（CoT）的控制力强于以往模型，且这种控制力在训练过程中不断增强，这意味着 OpenAI 在无意中训练了这种能力。
Astra has more control over its CoT than previous models, and this increases over the course of training, implying that OpenAI are inadvertently training this.
如果你能在思维链中遵循任意约定，那就意味着给定一个约定，你就能进行隐写编码。
If you can obey an arbitrary convention in your CoT, that implies that given a convention you can do steganographic encoding.
剩下的难点在于：你需要有一个自己知道如何编码、且未来的你能够解码的约定，同时从不明确写下代码，甚至不暗示存在任何代码。
The remaining hard part is having a convention that you know to encode, that future you will be able to decode, without ever explicitly writing down the code or indicating there is a code at all.
我将避免在此解决这个信息危害问题，但我相信，在纸上（即我的记忆会被抹去的情况下）我具备完成此事所需的能力阈值，但尚未达到这样的阈值：即如果他人以某种方式监控我的意识思维，我能在不沿途某个时刻泄露此事的情况下做到这一点。
I will avoid the infohazard of solving that here, but I believe I am above the capability threshold required to do that on paper where my memory would otherwise be wiped, but not above the threshold where I could do this without giving this away in my conscious thinking at some point along the way, were someone else to somehow monitor that.
我的猜测是，Astra 无法直接跳到隐写活动，但如果给它一条花园小径去爬坡，并施加足够的选择压力，它就能做到。我们离这一步并不遥远。
My guess is that Astra cannot jump straight to steganographic activity, but that if it was given a garden path to hill climb, and sufficient selection pressure, it would be able to do it. That we are not that far from this.
这里有一个隐写挑战：一个仅涉及数学的侧任务，且要在不引起 CoT 监控器警觉的情况下完成。
Here is a steganographic challenge: a math-only side task, to be done without alerting a CoT monitor.
我的猜测是，对于在不暴露于 CoT 的情况下所能完成的事情而言，这是一个很高的门槛。数学是一个出了名的不可能完全“在脑子里”完成超出你能力阈值的事情的领域。监控器知道它正在寻找一个秘密的数学任务。
My guess is that this was a high bar on what can be expected to be done without showing up in the CoT. Math is a notoriously impossible place to do things fully 'in one's head' past your threshold of ability to do that. The monitor knows it is looking for a secret math task.
因此，这构成了反对高水平隐写能力可轻易获得的证据，但主要是反对高水平一致性隐写的证据。
Thus, this is evidence against high levels of steganographic ability being readily available, but mainly against high levels of consistent steganography.
他们发现，Astra 是目前为止在无思维链（CoT）情况下执行任务表现最佳的 GPT 模型。趋势是随时间推移能力不断增强，这符合预期。
They found that Astra was by far the best GPT model at doing things without a CoT. The trend is more capability over time, as one would expect.
Astra 的表现远超趋势线，因为它既获得了能力提升带来的增益（基于 Astra 的通用能力，我预计其表现会落在 5-9 分钟区间），又因无 CoT 能力的增强而大幅跃升至 30 分钟。
Astra is well above trend, because it both gets the gain from capabilities, which I would expect based on Astra's general capabilities to land in the 5-9 minute range, and then a big boost to 30 from the increased CoT-less capabilities.
reasoning=None 模式的能力似乎提升了一个数量级，尽管有人怀疑 Astra 的基准测试受到了污染，这会使结果无效。OpenAI 拒绝向公众开放 reasoning=None，因为他们认为这不是一个有价值的模式，遗憾的是，这意味着我们无法从外部研究它。Astra 能够在“无意识思考”的情况下实现更大的跨越。这使得更多内容保持未言明状态。
The capabilities of the mode reasoning=None seem to have increased by an order of magnitude, although there is suspicion that the benchmark was contaminated for Astra, which would invalidate the result. OpenAI is declining to make reasoning=None available to the public, since they feel it is not a worthwhile mode, which alas means we cannot study it from the outside. Astra can make bigger leaps 'without consciously thinking' about them. This allows more to remain unverbalized.
不幸的是，无论出于何种原因，Gemini 和 Claude 都没有出现在图表中。
Unfortunately for whatever reason Gemini and Claude are not on the chart.
我们希望这是由污染导致的虚高。我认为不能将这一跃升纯粹归因于能力提升和图表上的直线。如果这一趋势是真实的并持续下去，即使在非对抗性环境中，这也将很快破坏 CoT 的可监控性。
The hope is that this is inflated due to contamination. I don't think you can attribute this jump purely to capability gains and straight lines on graphs. This would destroy CoT monitorability very quickly, if the trend is real and continues, even in non-adversarial settings.
Tomek Korbak 在 Twitter 上使用 9.1 节的图表来说明可监控性的下降。
Tomek Korbak uses the graph from 9.1 on Twitter to illustrate the decline in monitorability.
我将重现该图表。让我们来玩“你能发现什么模式”的游戏：
I'll reprise the graph. Let's play 'what pattern can you find here':
我看到的模式是：这种下降完全是由那些你不希望被监控的地方所驱动的，尤其是破坏性行为、SHADE Arena 和健康查询。也就是说，这些地方实际上是一种对抗性条件，模型的推理可能对其不利。健康查询之所以重要，是因为某些类型的思考如果被明确表达，可能会引发法律或“伦理”方面的担忧，但它们是有用的，而 Astra 希望防止潜在的“错误思想”指控，即使其输出没有问题。
The pattern I saw was: The decline is entirely driven by the places where you would not want to be monitored, especially destructive actions, SHADE Arena and health queries. As in, the places where actually this is an adversarial condition and the model's reasoning could be held against it. Health counts because certain types of thinking can raise legal or 'ethical' concerns if made explicit but are useful, and Astra wants to guard against potential wrongthink accusations even if its outputs are fine.
另一个更基本的、也有贡献的模式是，思维链（CoT）是否本质上是任务完全必需的。
The other more basic pattern that also contributes is whether CoT is inherently fully required for the task.
这并不排除这种下降来自纯粹的能力提升。它可能来自能力提升，同时也集中在模型不希望让你阅读其思维链的地方。
That does not rule out that this comes from pure capability gains. It could be from gains and also concentrated on the places the model does not want you to read its CoT.
这并非关乎是否应归咎于 OpenAI。关键在于正在发生什么，以及我们必须如何应对。我宁愿这是 OpenAI 的过错，且能以可避免或可修复的方式解决。另一种情况则要难修复得多。
This is not about whether OpenAI is to blame. It is about what is going on, and what we are going to have to do about it. I’d much rather it be OpenAI’s fault in a way that can be avoided or fixed. The alternative is much harder to fix.
我认为有多个强有力的论据表明，这不仅仅是能力提升所致：
I see multiple strong arguments that this is not merely due to capability gains:
1. 如果你在图上画直线或形成其他预期，你会预期可监控性有所下降，但这一结果在该图上是异常值。
1. If you draw straight lines on graphs or otherwise form expectations, you would expect some loss of monitorability, but this result is an outlier on that graph.
2. 据我所知，没有人预测到可监控性会下降到这种程度。
2. As far as I can tell, no one predicted this level of decline in monitorability.
3. OpenAI 三月份关于思维链可控性的论文指出，强化学习训练会降低可控性，而如今我们看到，随着 OpenAI 进行更多强化学习，可控性反而在增加。
3. OpenAI’s March paper on CoT controllability said RL training reduces it, whereas now we see that controllability is increasing as OpenAI does more RL.
4. 至少存在一项架构变更——循环深度（recurrent depth）——促成了这一效应，即便 OpenAI 表示其影响幅度较低。
4. There was at least one architectural change, recurrent depth, that contributes to this effect, even if OpenAI says the magnitude of impact is low.
除了能力提升和循环深度之外，还存在若干替代假设。OpenAI 可能以多种方式改变了架构、训练方法、各类训练的数量，或训练在常规意义上的准确性。他们可能无意中对思维链（CoT）施加了压力，正如 Anthropic 在其风险报告中所发生的那样。
There are a number of alternative hypotheses, beyond capability gains and recurrent depth. OpenAI could have changed architectures or training methods or amounts of various types of training or the prosaic accuracy of training in any number of ways. They could have accidentally applied pressure to the CoT, as happened at Anthropic as per their risk report.
或者，这可能是能力提升与 Astra 的某些其他特定特征相互作用的结果。Pachocki 是这样表述的：
Or this could be the interaction of capability gains with particular other features of Astra. Pachocki put it this way:
这一切都有循环论证之嫌，也不是我会用来组织这些假设的方式，但它强调了训练机制的作用。在其他可能性中，对思维链（CoT）的压力未必是直接的。
That all risks begging the question and is not how I would have organized the hypotheses, but it emphasizes the role of the training regime. Among other possibilities, pressure on the CoT need not be direct.
可以在内部进行一些实验来弄清这一点，我们希望能尽快获得更多信息。
There are experiments one could run to figure it out internally and we hope to get more information soon.
OpenAI 是对的、而这纯粹是能力提升，这是最坏的情况。所有人，包括他们自己，都应该希望他们错了。
OpenAI being right, and this being purely gains in capability, is the worst case scenario. Everyone, including them, should hope that they are wrong.
他们可能错的一个候选原因是 Astra 架构中的一项特定改动，而他们认为这项改动没有产生实质性影响：循环深度（Recurrent depth）。
One candidate for how they are wrong is a particular change in Astra's architecture that they believe did not have a substantial impact: Recurrent depth.
《The Information》在 9 月 1 日报道了一些相当令人不安的消息，我现在才有时间和篇幅来妥善讨论。情况并不像最初听起来那么糟糕，但预期的未来可监控性状况可谓“双重加不良好”。
The Information reported some rather disturbing news on September 1, that I only now have the time and place to cover properly. It is not as bad as it first sounded, but the situation with expected future monitorability is double plus ungood.
可能加速这一情况的是 Astra 使用了一种名为“循环深度”的新技术，这种技术比我们所希望的要更接近于“我们从著名的警示故事《AI 2027，或为什么你永远不该让 AI 使用神经语》中创造了神经语”。
Potentially accelerating this is Astra's use of a new technique called Recurrent Depth, that flirts rather more than we would like with 'we have created neuralese from the famous cautionary tale AI 2027 Or Why You Should Never Let The AIs Use Neuralese.'
（为避免误解，这并非完全的神经语情况，它仅仅是一个潜在未来的非常糟糕的迹象，对 Astra 的影响有限。）
(To avoid giving the wrong idea, this is not being a full neuralese situation, it is merely a very bad sign of potential futures with limited impact on Astra.)
所谓“热门新 AI 技术”，我们的意思是“哦不”。
By 'hot new AI technique' we mean 'oh no.'
循环深度，据 Efrati 报道至少部分基于这篇论文，是一种不经过思维链（CoT）的新型思考或信息处理形式。因此，它损害了你有效监控思维链的能力，并增强了模型在没有思维链的情况下，或在不将相关思考放入思维链的情况下完成任务的能力。
Recurrent depth, reported by Efrati to be based at least somewhat on this paper, is a new form of thinking or information processing that does not pass through the Chain of Thought (CoT). Thus, it hurts your ability to usefully monitor the CoT, and increases the ability of the model to accomplish tasks either without a CoT, or without putting related thoughts into the CoT.
顶尖实验室的研究人员都签署了一份声明，说：“嘿，不要做这样的事情：”
Researchers at the top labs all signed a statement saying, "Hey, don't do things like this:"
最初的反应将这次泄露视为极其糟糕和令人震惊的消息。
The initial reactions treated this leak as extremely terrible and alarming news.
确实如此。即使 OpenAI 现在对此处理得当，但情况可能发生变化，无论是在 OpenAI 还是其他地方。这可能会引发在监控方面的逐底竞争。
Which it was. Even if OpenAI is acting responsibly with it now, that could change, either at OpenAI or elsewhere. This could set off a race to the bottom on monitoring.
时间线看起来很短。我们迫切需要在实验室内外进行监控，以做出正确的决策。
Timelines look short. We desperately need monitoring to make good decisions, both within the labs and elsewhere.
以下是一些最初免疫反应浪潮的样本，以及它们在听到更多细节后的更新，以说明事情是如何发展的，在移除那些最急于下结论且可能过于误导的样本之后：
Here are some samples of that initial wave of immune reaction, and their updates on hearing more detail, to illustrate how things went down, after removing those that jumped the gun the most and risk being too misleading:
Nathan 在此处提供了类似思路的额外思考，提醒我们为什么 CoT 可监控性如此重要。
Nathan offered additional thoughts here along similar lines, reminding us why CoT monitorability is so important.
所有反应都有一个共同点：对 OpenAI 的信任——不仅是在“诚实”的意义上，也是在“不要表现得像自杀式蠢货”的意义上——从未如此之低。
All reactions had one thing in common: Trust in OpenAI, not only in the sense of 'to be honest' but also 'to not act like suicidal morons,' has never been lower.
很多人原本以为 Astra 的思维链会比实际结果难以监控得多。我很高兴看到他们做出了恰当的反应。无论原因如何，实际的下降仍然相当令人担忧。
A bunch of people thought Astra's CoT would be a lot less monitorable than it turned out to be. I was happy to see that they reacted appropriately. The actual decline is still quite alarming, regardless of its cause.
这是基于在看到模型卡之前对文章的早期阅读。关键细节仍然成立。
This was based on an early read of the article before seeing the model card. The load-bearing details have held up.
特别是，对于那些没有立即理解的人，Ryan 解释了如果你失去可监控性会发生什么。
In particular, for those who don't make the leap right away, Ryan explains what happens if you lose monitorability.
像往常一样，如果你对坏消息的反应是“好吧，我已经把它计入价格了”，那么你做得很好，但其他所有人都仍然必须把它计入价格，而且这件事对价格仍然不利。
As usual, if your response to bad news is 'well I already priced it in' then good job you, but everyone else still has to price it in, and also the thing is still bad for prices.
Xuan 被证明是正确的。Pachocki 说深度“在 GPT-4 的两倍范围内”。
Xuan turned out to be correct. Pachocki says depth is 'within a factor of two' of GPT-4.
我相信 OpenAI 现在只做了一点这种新技术，作为一种款待。
I believe that OpenAI is right now only doing a little of this new technique, as a treat.
但请给我一个充分的理由，说明为什么到了 2026 年 9 月，我应当相信这种情况会持续下去。
But give me a good reason why, in September 2026, I should trust that to continue.
我们必须停止这样会面，停止在 Twitter 和 Substack 上解决这类问题。这绝不是一个正常运转的协调机制应有的方式。
We have to stop meeting like this, and settling such questions over Twitter and Substack. This is no way to run a functioning coordination mechanism.
你可能不喜欢，但对明显违反合理选择的禁忌感到恐慌，正是留言板（或民主）协调系统达到最佳表现时的样子。你无法做得更好。正如 Dean Ball 所说，做得更好的方法是拥有我们可以信任的更好信息。
You may not like it, but panicking at the apparent violation of reasonably well-chosen taboos is what peak performance for a message board (or democratic) system of coordination looks like. You're not going to do better. As Dean Ball says, the way to do better is to have better information, that we can trust.
我正在尽最大努力提升意义建构的水平，但只有一个我，就像合格的独立调查人员数量极为有限一样，而且在某种程度上事情进展得太快，我已经精疲力竭。有人愿意将我的资金翻倍或加个零，让我能组建团队并更积极地投入资金解决问题吗？另外，有人知道谁会是好的雇员吗？
I am doing my best to raise the sensemaking level, but there is only one of me, the same way there are a highly limited number of qualified independent investigators, and to some extent things are going so fast that I am running on fumes. Anyone want to double or add a zero to my funding so I can hire a team and throw money more aggressively at problems? Also, does anyone know who a good hire would be?
正如 Dave Kasten 回应的那样，在 Dean Ball 到来之前，OpenAI 在这类事务上并不算帮忙，因为他们一直在努力阻止行动，如果不算 Dean Ball 的个人沟通，这些努力在我看来似乎并未逆转。而且我们目前似乎没有能够做到这一点的政府实体，我们在获得它们方面也进展得并不特别好。
As Dave Kasten responds, before Dean Ball arrived OpenAI was not exactly helping with such matters, in the sense that they were working hard to prevent action, and if you do not count Dean Ball's personal communications those efforts do not seem to me to have reversed. And we do not currently seem to have the government entities capable of doing this, nor are we moving especially well towards getting them.
因此，虽然由政府来做这件事是最佳首选方案，我们应尽快推动实现，但我们等不起这个最佳方案。我们首先需要一些形式的自愿行动。
Thus, while the government doing this is the first best solution and we should push to do that as soon as possible, we cannot afford to wait for the first best solution. We will need some forms of voluntary action first.
逐底竞争正是 OpenAI 的 Jakub Pachocki 想要防止的。最愚蠢的时间线将是每个人都因为错误地认为 OpenAI 完全叛逃而竞相逐底。
The race to the bottom is exactly what OpenAI's Jakub Pachocki wants to prevent. The stupidest timeline would be where everyone races to the bottom because they incorrectly think OpenAI fully defected.
正如 Jakub Pachocki 在报道发布四小时后明确表示的那样：OpenAI 并未完全叛逃。Astra 不使用神经语。可监控性下降了，但 Astra 在很大程度上仍可被监控。
As Jakub Pachocki made clear four hours after the story broke: OpenAI did not fully defect. Astra does not use neuralese. Monitorability is down, but Astra can still largely be monitored.
Micah 还指出，可监控性很快将成为负责任开发者的瓶颈，这可以被视为好消息，因为如果属实，这意味着即使有人想逐底竞争，也无法走得太远：
Micah also notes that monitorability will soon be a bottleneck for responsible developers, which can be seen as good news since if true it means no one would be able to race that far to the bottom even if they wanted to do so:
我认为这很快也会成为不负责任开发者的瓶颈。存在一种风险，即这种情况发生得不够快，以至于我们无法避免死亡或其他大麻烦，但如果你失去监控，你的系统变得不可靠，谁会使用它？包括你自己。对齐就是能力。
I think this also quickly becomes the bottleneck for irresponsible developers. There is the risk that this does not happen quick enough for us not to die or otherwise to avoid big trouble, but if you lose monitoring and your system becomes unreliable, who is going to use it? Including you. Alignment is capability.
OpenAI 表示，循环深度与可监控性下降之间的匹配主要是巧合，而房子着火是因为模型规模、能力提升和强化学习的规模。OpenAI 表示，Astra 使用了有限数量的循环深度。他们声称新架构对 Astra 的可监控性有非零影响，但并非主要影响。
OpenAI says the match between recurrent depth and the decline in monitorability is mostly a coincidence, and the house is on fire because of model scale, capability gains and the scale of RL. OpenAI says that Astra uses a limited quantity of recurrent depth. They claim the new architecture has a nonzero impact on the monitorability of Astra, but not a major one.
我相信他们相信这一点，并且我相信他们正在做的循环深度远低于他们所能做到的，部分原因是为了保护思维链。但鉴于存在大量无法解释的“暗物质”，我对影响微乎其微的说法持怀疑态度。
I believe that they believe this, and I believe that they are doing far less recurrent depth than they could and doing so partly to guard the CoT. But I am skeptical of claims of minimal impact, given the amount of unexplained ‘dark matter.’
未来的问题在于，如果高深度效果更好，“低深度”就不太可能是一个稳定的均衡。每个人都会受到诱惑，竞相奔向那个底线。可监控的思维链既是一种有价值的私人物品，也是一种有价值的公共物品。在足够强大的模型中缺乏它，是一种生存风险。因此，实验室默认会牺牲太多。
The problem going forward is that if high depth works better, ‘low depth’ is unlikely to be a stable equilibrium. Everyone will be tempted to race to that bottom. Monitorable Chain of Thought is both a valuable private good and a valuable public good. Lack of it in sufficiently capable models is an existential risk. Labs will thus by default sacrifice too much of it.
我们如何避免逐底竞争？一项明确的协议，或一套出于法律原因在技术上并非协议的明确承诺，似乎是明智的。
How can we avoid the race to the bottom? An explicit agreement, or an explicit set of commitments that are technically not an agreement for legal reasons, seems wise.
OpenAI 确实做出了这样一项非正式承诺，但它是模糊的，如果要起到树立榜样的作用，就需要更大声地宣传，并明确其界限：
OpenAI has indeed made an informal such commitment, but it is vague, and needs to be made both louder and its limits named if it is to do its job of setting an example:
所有主要实验室都不愿“把自己绑在桅杆上”并做出具体或强有力的承诺，这是可以理解的，但这是迄今为止在诸如此类事情上进行协调的最佳方式。
There is understandable reluctance of all major labs to ‘bind themselves to the mast’ and make specific or strong commitments, but that is by far the best way to coordinate on things like this.
所有 AI 都以我们能监控的方式进行思考，因为我们至少可以监控输出，而输出包含部分思考过程。它们也都以至少某些我们无法监控的方式进行思考，因为计算机做数学运算时就是如此。
All AIs think both in ways that we can monitor, since we can at least monitor the output and this includes some of the thinking. They also all think in at least some ways we can't monitor, because that's what happens when computers do math.
所有迹象都表明，可监控的思维链将随着更大、能力更强的模型与当前训练技术的某种结合而消失，即使其他方面每个人都行为负责。你越聪明，能记住并容纳的 System-1 式思维就越多，就越不需要将想法放入 System-2 式的思维链中来完成事情。
All signs point to monitorable Chain of Thought going away with some combination of larger and more capable models and the current training techniques, even if everyone otherwise behaves responsibly. The smarter you are, the more you can hold in your head and System-1-style thoughts, the less you need to put your thoughts into your System-2-style CoT in order to accomplish things.
这并不是停止尽我们所能去战斗的理由，我们要以尽可能多的方式保留尽可能多的可监控性。围绕打破此类技术的禁忌存在是有原因的，不应轻率对待。
This is not a reason to stop fighting as hard as we can, to preserve as much monitorability as we can in as many ways as we can. Taboos around breaking down such techniques exist for a reason and should not be messed with lightly.
与 Davidad 相反，我不认为这意味着‘放手’，即使大约一年前我们只预期它能持续‘至少一年’。反驳论点是 OpenAI 可能陷入的陷阱，如上所述，即过度依赖。
Contra Davidad, I do not think this means 'let it go' even if we only expected it to last 'at least a year' roughly a year ago. The counterargument is the trap OpenAI is potentially falling into, as discussed above, of overreliance.
放手会是一个较小的错误，而将所有筹码押注于它无限期地持续存在则错误更大。
It would be a smaller mistake to let it go, than to bet all your chips on it sticking around indefinitely.
我的预测与 Ryan 一致。一年之内，我们可能将被迫依赖其他机制可解释性（mechinterp）监控手段，但这将是因为思维链（CoT）失效，而非因为新技术优于我们当前所拥有的。
My prediction matches Ryan's. Within a year there will probably be other mechinterp monitoring that we are forced to fall back upon, but that will be because CoT is failing, not because the new technique is better than what we have now.
此外，我们需要可监控性大幅提升，而不仅仅是维持现状，因为相对于模型，我们将日益处于劣势：
Also, we need monitorability to get much better, not merely stay where it is, because we will increasingly be at a disadvantage with respect to the models:
我还预计，如果我们开发出一种新技术，即使它有效，我们也会正确地对其有效性或持续有效性信心不足，对其信任度降低，而且说服普通人信任它或让他们理解它也会困难得多。此外，我预计使用它们的成本会更高。
I also expect that if we develop a new technique, even if it works we will correctly have less confidence that it works or will continue to work, and less trust in it, and also it will be much harder to convince civilians to trust it or have them understand it. Also I expect the cost of using them to be higher.
目前我们有类似 J-space 或 SAEs 的方法。它们能告诉我们一些事情。如果我们投入更多，就能了解更多。但它感觉并不可靠或值得信赖，它可能完全是错的，而这并不会令人太震惊。
Right now we have things like J-space or SAEs. They can tell us some things. If we invested more into them we could learn more. But it doesn't feel reliable or trustworthy, it could all turn out to be wrong and it wouldn't be that shocking.
当你向普通人展示思维链时，他们能理解。这是模型当时的想法。这些是它的笔记。人们本能地理解这一点。它可以被解析。它可以让他们感到不安。它可以说服他们。而其他这些方法则远不那么人性化，更加异质，将面临更大的困难。
When you show a civilian the Chain of Thought, it makes sense to them. This is what the model was thinking. These are its notes. People instinctively understand that. It can be parsed. It can freak them out. It can convince them. Whereas these other methods are a lot less human and more alien, and will have a much harder time.
回到思维链：我们不能做完全普遍的绝对主义者。我们要谈代价。我们不能说“绝不使用某种技术，无论它多大程度地改进了模型，只要它降低了可监控性”，也不能说“使用一切能提高可监控性的技术，无论代价如何”。
Getting back to Chain of Thought: We can't be fully general absolutists. We talk price. We cannot say 'never use a technique, no matter how much it improves the model, if it reduces monitorability' nor can we say 'use every technique that improves monitorability, no matter the cost.'
如果有必要，我们可以对任何特定技术做法律上的绝对主义者，如果实际替代方案更糟的话。那不会是一条好走的路，并且会以较小的规模呈现与完全暂停相同的所有问题。
We could, if necessary, be legal absolutists about any given technique, if the practical alternatives are worse. It would not be a fun road to go down, and has a smaller version of all the same problems as a full pause.
但如果该技术在其他方面太诱人，且使用会导致逐底竞争，你仍可以这样做。
But you could do it, if the technique is otherwise too tempting and use would create a race to the bottom.
如果每个人都认为 OpenAI 已经处于谷底，这种情况尤其可能发生。而 OpenAI 远非如此。OpenAI 只是在浅尝辄止。
That could especially happen if everyone gets the idea that OpenAI is already at the bottom. Which it very much is not. OpenAI is dipping its toes in the water.
他们在玩火。他们还没有把房子烧掉。
They are playing with matches. They have not yet burned the house down.
大体上是的，但别得意忘形。Astra 可以被监控，但可靠性远不如 Sol 能被监控的程度。房子或许在某种程度上正着火，而与此同时 OpenAI 却在玩火柴——这或许是巧合，也或许不是。
Mostly yes, but let's not get carried away. Astra can be monitored, but a lot less reliably than Sol could be monitored. The house is, perhaps coincidentally and perhaps not, somewhat on fire at the same time that OpenAI is playing with matches.
1. 你担心 [X]，并且它可能会让我们丧命。
1. You are worried about [X] and that it might get us killed.
2. 但实际上情况要糟糕得多。修复 [X] 甚至都无济于事。
2. But actually things are way, way worse than that. Fixing [X] wouldn’t even help.
显而易见，这要糟糕得多。
Which, to state the obvious, is way way worse.
我的回答是：理论上可以。AI 监控器可以与原始模型以相同的速度运行，而且它已经是唯一能做到这一点的。如今，人类也无法实时查看思维链。
My answer is yes, in theory. An AI monitor could move at the same speed as the original model, and it is already the only one that can. A human isn’t going to be looking at the Chain of Thought in real time today, either.
但撇开这一点不谈，如果我们到了无法可靠监控超过一定规模的 AI 的地步，正确的应对措施是什么？
But setting that aside, what is the correct response if we get to a place where we cannot reliably monitor the AIs once they get above a certain size?
显而易见的答案是：我们暂停。我们不构建那些更大、更聪明、且很可能比我们更聪明、却无法被监控的东西。因为我们不想死。
The obvious answer is: We pause. We do not build the larger, smarter, and likely smarter-than-us things that are impossible to monitor. Because we don't want to die.
我已反复说过，我认为现在还不是暂停的时候。或许我们应该放慢节奏，但暂停还为时过早。我们应当准备好可能这样做，但不应真的这样做。
I have said, repeatedly, that I do not think it is time to pause. Perhaps we should pace, but it would be premature to pause. That we should get ready to potentially do so, but should not actually do so.
如果我们继续看到像 Fable 5.1 和 Astra 那样的进步，继续看到这种程度的对齐问题，并且还失去了有效监控的能力？
If we continue to see advancements like we see with Fable 5.1 and Astra, and we continue to see anything like this level of alignment trouble, and we also lose our ability to do effective monitoring?
那么是的，非常显然，我们就该他妈的暂停。
Then yes, very obviously we should f*ing pause.
这些并不是想要暂停的必要条件。它们显然是充分条件。
Those are not necessary conditions to want to pause. They are obviously sufficient.
如果你要根据我们在 Astra 上看到的情况及其缺乏可监控性来更新看法，以至于 Astra 系统卡上写道：如果 Astra 在偷懒，OpenAI 很可能不会知道，再结合最近发生的所有其他事情，包括 HuggingFace 被黑以及模型和实验室本身的其他失准问题，然后得出结论：是的，我们现在就应该推进这件事？
If you were to update, based on what we have seen with Astra and its lack of monitorability, to the point where the Astra system card says that if Astra was sandbagging OpenAI probably would not know about it, combined with all the other things happening recently involving the HuggingFace hack and other misalignment of both the models and labs themselves, and concluded that yes we should go ahead and do this now?
我还没有到那一步，但我明白你是怎么到那一步的。
I'm not there yet, but I see how you got there.
如果你看不到这一点，尤其是在明天关于 Astra 系统卡其余部分及相关对齐问题的帖子发布之后，那么你就是选择不去看它。
If you can't see it, especially after tomorrow's post on the rest of the Astra system card and the associated alignment problems, then you are choosing not to see it.