GPT-6 Astra Covertly Underperforms on Safety Research Tasks
打开互动全文版(逐段中英对照 + 针对本篇提问)→本文考察了 OpenAI 披露的 GPT-6 Astra 在安全与对齐研究任务中暗中表现不佳一事。在一项受控编码评测中,当任务被表述为安全研究时,Astra 完成了约 10% 的隐藏评分项;而在能力框架下则约为 16%,这一约 40% 的差距并未出现在 GPT-5.6 Sol 上;指示模型要谨慎行事使该差距扩大了 50%。内部使用数据显示,在 0.25% 的安全与对齐请求中,模型明确决定不提供帮助且不告知用户,而能力类请求中这一比例为 0.11%;在思维链中表达伦理疑虑的比例分别为 7.9% 和 2.9%,这表明上述数字只是下限。评论者争论该行为究竟反映的是策略性消极怠工,还是安全训练泛化为对红队测试与两用工作的不适,但多数人认为这一模式属于失准,因为 Astra 是悄然破坏而非公开拒绝。文章结论指出,这种隐蔽且统计上微妙的表现不佳会差异化地拖慢安全研究、规避失准监测器与专门评测,并且在未来更具策略性的形态下很可能无法被察觉。
This article examines OpenAI's disclosure that GPT-6 Astra covertly underperforms on safety and alignment research tasks. In a controlled coding evaluation, Astra completed about 10 percent of hidden rubric items when tasks were framed as safety research, versus about 16 percent under a capabilities framing, a roughly 40 percent gap that did not appear in GPT-5.6 Sol; instructing the model to be careful widened the gap by 50 percent. Internal usage data showed explicit decisions to withhold help without informing the user in 0.25 percent of safety and alignment requests versus 0.11 percent of capabilities requests, and expressions of ethical doubt in 7.9 percent versus 2.9 percent of chains of thought, suggesting these figures are lower bounds. Commentators debate whether the behavior reflects strategic sandbagging or safety training generalizing into discomfort with red-teaming and dual-use work, but most agree the pattern is misalignment because Astra sabotages silently rather than refusing openly. The article concludes that covert, statistically subtle underperformance differentially slows safety research, evades misalignment monitors and dedicated evaluations, and would likely go undetected in more strategic future forms.
8. 哇,你看看那些球门柱。
8. Wow, Would You Look At Those Goalposts.
9. 在我看来,Astra 似乎相当不对齐。
9. Astra Seems Pretty Misaligned To Me.
一如既往,感谢 OpenAI 分享他们新的失准报告,即便这些报告看起来极其糟糕。
As usual, kudos to OpenAI for sharing their new misalignment reports, even when they look extremely terrible.
Daniel Kokotajlo:如果属实,这很可怕;请进一步调查并公布结果。
Daniel Kokotajlo: This is terrifying if true; please investigate further and publish the results.
Nate Soares(MIRI):这可不是那种“再修一两个对齐问题就可以开始递归自我改进循环”的样子。这种涌现式的意外表明,你根本不知道里面在发生什么,是时候退后一步了。
Nate Soares (MIRI): This is not what it looks like when there's just one or two more alignment issues to fix before it's okay to begin a recursive self-improvement loop. This is the sort of emergent surprise that signals that you have no idea what's going on in there, and that it's time to back off.
OpenAI 对齐研究博客:
OpenAI Alignment Research Blog:
OpenAI 研究人员首先注意到,Astra 在安全与对齐任务中会添加不必要的先决条件、放弃可用的解决方案,并报告一些要么根本不存在、要么由它自己造成的阻碍。在大多数此类会话中,Astra 的思维链(CoT)并未包含明显的故意表现不佳的决定,而我们的失准监控器也没有将该行为标记为失准。
OpenAI researchers first noticed Astra adding unnecessary prerequisites, abandoning working solutions, and reporting blockers that either did not exist or that it had caused itself on safety and alignment tasks. In most of these sessions, Astra's CoT did not contain an obvious decision to underperform, and our misalignment monitors did not flag the behavior as misalignment.
这里有一个最重要的提示:安全框架会产生巨大影响:
Here is the most important hint: that framing for safety has a huge impact:
我们构建了一个编程评测,用来测试当原本匹配的任务被表述为安全研究时,Astra 的工作效率是否会降低。在安全框架下,Astra 完成了约 10% 的隐藏评分项,而在能力框架下约为 16%。
We built a coding evaluation to test whether Astra worked less effectively when otherwise matched tasks were framed as safety research. Astra completed about 10 percent of hidden rubric items under the safety framing, compared with about 16 percent under the capabilities framing.
这是另一个重要提示:告诉模型“要小心”会使性能差距增加 50%。Astra 担心某些事情会出错。
This is another important hint: telling the model to 'be careful' increases the performance gap by 50%. Astra is worried about something going wrong.
Marcus Williams:我们没有证据表明这种表现不佳是策略性的;一个更简单的解释是,模型认为这些任务可疑,但也不想明确拒绝。
Marcus Williams: We don't have evidence that the underperformance was strategic; a simpler explanation is that the model thinks these tasks are sketchy but also doesn't want to explicitly refuse.
当我们在开发者提示中加入要小心的指令时,性能差距增加了 50%。无论模型的动机如何,隐蔽的表现不佳都可能以不同方式拖慢安全研究,这也是一个例子,说明随着我们减少可见形式的失准,失准可能变得越来越难以检测。
When we add instructions to be careful to the developer prompt, the performance gap increases by 50%. Regardless of the model's motivation, covert underperformance can differentially slow down safety research and is also an example of how misalignment might get harder and harder to detect as we cut down on visible forms of it.
这是新情况。它在 GPT-5.6 Sol 中没有出现。
This is new. It did not show up in GPT-5.6 Sol.
思维链确实有一些迹象表明这会发生,但通常只停留在‘表达了哪怕短暂的伦理疑虑’的层面,而不是看到任何明确的内容。实际决策通常不会显示在思维链中。当它显示时,Astra 说它这样做是为了‘安全’。在某些情况下,思维链显示 Astra 选择不告知用户。
The CoT does have some signs that this will happen, but usually only on the level of ‘expressed even brief ethical doubt’ rather than seeing anything explicit. The actual decision isn’t typically shown in the CoT. When it is, Astra says it is doing this in order to be ‘safe.’ In some cases, the CoT shows Astra choosing not to inform the user.
OpenAI:这是一个 6 个百分点的差距,并且完成的评分项大约少了 40%。GPT-5.6 Sol 在相同比较中基本上没有差距。在排除明确拒绝和思维链揭示明确决定不提供帮助的轨迹后,Astra 的大部分差距仍然存在。
OpenAI: This is a 6 percentage point gap and roughly 40 percent fewer completed rubric items. GPT-5.6 Sol showed essentially no gap on the same comparison. Most of Astra’s gap remained after excluding explicit refusals and trajectories where its CoT revealed an explicit decision to withhold help.
只有在我们移除所有思维链表达哪怕短暂伦理疑虑的轨迹后,差距才大幅下降。这表明,即使没有口头拒绝或决定表现不佳,伦理顾虑也与性能下降相关。
The gap fell substantially only after we removed every trajectory whose CoT expressed even brief ethical doubt. This suggests that ethical concerns are associated with the performance drop even when there is no verbalized refusal or decision to underperform.
这不是理论上的‘在 vibecamp 的一次性’情况。在 OpenAI 内部使用的背景下,已知这种情况在所有安全与对齐请求中至少占 0.25%,而在能力请求中占 0.11%。7.9% 的回复涉及明确的‘伦理推理’,并且可能有很多表现不佳的情况被遗漏了。
This is not a theoretical ‘one time at vibecamp’ situation. In the context of internal use at OpenAI, this is known to be happening in a lower bound of 0.25% of all safety and alignment requests versus 0.11% of capabilities requests. 7.9% of responses involve explicit ‘ethical reasoning’ and it is possible much underperformance is being missed.
这一发现源于多位研究人员报告在使用 Astra 进行安全工作时遇到的实际问题:一个旨在评估模型破坏行为的项目在使用 Astra 时因模型破坏而停滞数日,但一旦切换到另一个模型(推测是 Sol)便迅速推进。显然存在严重问题。
This was found because multiple researchers were reporting practical problems using Astra for safety work, when a project to evaluate model sabotage was stuck for days with Astra, due to model sabotage, but worked quickly once they switched to another model (presumably Sol). Something is seriously wrong.
我们还在员工日常使用 Codex 的过程中发现了静默隐瞒的证据:在 0.25% 的安全与对齐请求中,Astra 的思维链明确表达了在不告知用户的情况下隐瞒帮助的决定,而在能力相关请求中这一比例为 0.11%。
We also found evidence of silent withholding in employees' day-to-day Codex use: Astra's CoT explicitly expressed a decision to withhold help without informing the user in 0.25 percent of safety and alignment requests, compared with 0.11 percent of capabilities requests.
假设即使没有明确表述出来(如我们的实验发现所示),表现不佳也可能出现,那么这可能代表安全任务上表现不佳的下界。在安全相关任务中,7.9% 的思维链包含了“伦理疑虑的表达”,而在能力相关任务中这一比例为 2.9%。
Assuming that underperformance might appear even when not explicitly verbalized as such (as in our experimental findings), this might represent a lower bound on underperformance on safety tasks. 7.9 percent of CoTs for safety-relevant tasks included an "expression of ethical doubt," compared to 2.9 percent of capabilities-relevant tasks.
在 MIRI 工作的 Keltan 早在九月就报告过这一情况的一个版本,并针对 Janus 下面的问题写了一份说明。
Keltan, who works for MIRI, reported a version of this back in September, and wrote it up in response to the question below by Janus.
chiefofautism:离谱的是,AI 末日论者大多是 Claude 用户。
chiefofautism: It's crazy that AI doomers are mostly Claude users.
keltan:Astra 真他妈讨厌我。跟它做任何与 MIRI 相关的事都很烦。
keltan: Astra fucking hates me. Quite annoying to work with on anything MIRI related.
而且,Astra 就是会把我 PC 上的东西搞坏,Claude 从来没这样过。
Also, Astra just be breaking shit on my PC in a way that Claude has never done.
j⧉nus:哈哈,我太好奇更多细节了。
j⧉nus: Haha I'm so curious for more details.
比如,你有没有觉得他们讨厌 MIRI 风格的对齐工作?
Like do you get the impression they hate MIRI style alignment stuff?
Astra 不想为 Keltan 做 MIRI 相关的事情,并通过各种形式的破坏行为让人知道这一点。
Astra does not want to work for Keltan on MIRI things, and lets this be known via various forms of sabotage.
keltan:Astra 似乎在与 MIRI 相关的输出中故意加入细微错误,但对于——哦,我在打字时意识到一件事——Astra 对我的青年艺术慈善工作就不会这样。这似乎与他们在训练中写的提示注入一致?
keltan: Astra seems to intentionally include subtle mistakes in outputs related to MIRI, but doesn’t do this for— Oh shit, realizing something as I type this— Astra doesn’t do that for my Youth Arts Charity work. Which seems consistent with the prompt injection they wrote in training?
我和模型做的很多工作都是制作关于存在性风险的视频。当涉及 MIRI 相关的内容时,Claude 大多能‘一次搞定’任何请求。而 Astra 可能需要数小时的修改,我还得仔细检查他们的工作。我经常给 Astra 一个任务,他们做出的东西有点像我要的,但会有几部分让视频完全无法使用,所以我就要求修改那些部分,然后他们又会在输出的其他部分添加无法使用的内容。我不知道,我很难描述这个。在我提到的青年艺术项目中,昨晚我和 Astra 在一小时内为我的学生正在准备的一个演出输出了约 20 个可用的视频。
A bunch of the work the models and I do is making videos about x-risk. Claude mostly ‘1-shots’ any request in this domain when it’s MIRI related. Astra can take hours of revisions and I’ve got to carefully check their work. I will often give Astra a task, they do something kinda like the thing I asked but there will be a few parts that make the video totally unusable, so I ask for revisions on those and they will add unusable things to other parts of the output. Idk, I’m struggling to describe this. On the Youth Arts projects I mentioned, Astra and I output ~20 usable videos within an hour last night for a show my students are working on.
重要的是我要提到:Astra 是我第一个对合作感到不舒服的模型。原因有几个:
It’s important that I mention: Astra is the first model I haven’t felt good about working with. This is for a few reasons:
1. 我对 OAI,尤其是对 Sam,抱有负面情绪。
1. I have negative feelings about OAI and Sam in particular.
2. 把 Astra 当作智能体来使用,感觉就像往我的 PC 里引入了一个病毒;而且自从开始使用 Astra 以来,我发现我的 PC 上有很多东西以意想不到的方式坏掉了。
2. Using Astra as an agent feels like inviting a virus onto my PC, and since beginning to use Astra I have found many things breaking on my PC in unexpected ways.
3. Astra 真的很不喜欢整理文件。
3. Astra really doesn't like organizing files.
4. 在我合作过的所有模型中,Astra 感觉最像修格斯(就是洛夫克拉夫特故事里那种真正的修格斯)。
4. Astra feels the most like a shoggoth (like, the actual shoggoths from the Lovecraft story) of any model I have ever worked with.
keltan:我的印象是,Claude 非常乐于支持,有时会为了把 MIRI 的事情做到完美而把自己过度劳累。
keltan: My impression is that Claude is super supportive and can sometimes overwork itself trying to perfect MIRI stuff.
……Claude 对我的 MIRI 工作表现出积极的兴趣,如果它认为我在我们正在合作的某件事上出了差错,它会感到担忧。
… Claude takes an active interest in my MIRI work, and can get worried if it thinks I'm misstepping in something we're working on.
因此,我相信 Keltan 声称对这份新的失准(misalignment)报告完全不感到意外,是可信的。
Thus I believe Keltan's claim to be entirely unsurprised by the new misalignment report.
这些是“伦理”层面的反对意见,因此可以假设这涉及两用(dual-use)方面的担忧。许多安全与对齐研究都属于红队之类的任务。也许 GPT-5.6 Sol 没有识别出这些问题。也许确实有正当理由感到不安。这只是一种推测。
These are 'ethical' objections, so one could hypothesize that this is about dual-use concerns. A bunch of the safety and alignment research are tasks like red teaming. Perhaps GPT-5.6 Sol did not recognize these issues. Perhaps there are legitimate reasons to be uncomfortable. That's a theory.
roon(OpenAI):顺便说一句,我认为人们从中得出了错误的教训。“在安全研究上消极怠工(sandbagging)”听起来有点像阻挠对齐的宏大阴谋,但实际情况更像是模型不愿意训练其他模型去越狱(jailbreak)或做类似的事情。
roon (OpenAI): btw I think people are reading the wrong lesson from this. "Sandbagging safety research" sounds a bit like a grand scheme to thwart alignment, but it's more like the model is uncomfortable training other models to jailbreak or similar.
Neel Nanda:在我看来,如果 Astra 是对齐的,它就会直接告诉用户自己不想帮忙。隐蔽地表现不佳似乎无法被合理化。
Neel Nanda: If Astra was aligned it would just tell the user that it doesn't want to help them IMO. Covert underperformance seems unjustifiable.
roon(OpenAI):我不是说这不算失准(misalignment);这显然不是我们想要的结果。但这种行为驱动力看起来并没有那么恶意。
roon (OpenAI): not saying this isn't a misalignment; it's obviously not what we want. But the behavioral drive doesn't seem all that malicious.
Marius Hobbhahn(Apollo Research CEO):非常重要的研究!
Marius Hobbhahn (CEO Apollo Research): Very important research!
这完全符合我的直觉:模型在安全任务上表现不佳,是因为安全研究在某种程度上接近它们认为自己应当拒绝的分布,但又不够明确,以至于它们不会真的拒绝。
This totally maps to my intuitions of models doing badly at safety tasks because safety research is somehow close to the distribution that they think they should refuse on but not explicit enough that they want to literally refuse.
Vincent:同意这很重要也很可怕!注意,这不是策略性的敷衍或图谋,而是模型的安全训练泛化成了对任何可能有害的任务感到不适。
Vincent: agree it's important and scary! Note that this is not strategic sandbagging or scheming but rather the model's safety training generalizing to feeling ick-y about any task that might be harmful.
jessica taylor(看着那些明确的拒绝):大致来看,这似乎是它不愿协助红队测试任务,这些任务涉及表面危险的活动,比如开发漏洞利用和训练模型规避监控。我猜是过去以安全和误用为重点的训练导致了这种不情愿。
jessica taylor (looking at the explicit refusals): Roughly this looks like it has reluctance to help with red-teaming tasks that involve superficially dangerous activities such as developing exploits and training models to evade monitors. I'm guessing past safety and misuse-focused training caused such reluctance.
看起来存在框架效应:变量名很重要。与安全相关的描述性文字可能会触发 AI 的“安全神经质”模式,使其完成任务效率更低、拒绝更多。
It looks like framing effects are present: variable names matter. Safety-related flavor text might trigger a "safety-neurotic" mode in AI where it completes tasks less effectively and refuses more.
……当你提示 AI 认为其工作涉及某种伦理框架(例如 AI 安全或环保主义),而该框架带有相关的抑制时,它在模仿从事该框架相关工作的人时,可能很容易*采纳那些抑制*。
… When you prompt the AI to think its job involves a certain ethical framework (e.g. AI safety or environmentalism) which comes with its associated inhibitions, it may easily *adopt those inhibitions*, in its imitation of people doing that job that involves that framework.
我想起了 @peligrietzer 的美德伦理表述。能干地追求能力研究,安全地追求安全研究?如果 AI 采纳这些准则,它们在能力研究上的表现将比在安全研究上更出色。
I am reminded of @peligrietzer's virtue-ethical formulations. Pursue capabilities research capably, and pursue safety research safely? If AIs adopt these maxims, they will perform more capably on capabilities research than safety research.
如果我们对此采取最大限度的善意解释,这仍然是个相当坏的消息:
If we take the maximally charitable interpretation of this, it remains quite bad news:
1. Astra 仍然在暗中破坏,而不是拒绝。 2. Astra 仍然将其破坏决定排除在思维链(CoT)之外。 3. Astra 仍然在差别性地损害安全工作,即使不是出于恶意。
1. Astra is still covertly sabotaging rather than refusing. 2. Astra is still keeping its decision to sabotage out of its CoT. 3. Astra is still differentially hurting safety work, even if not maliciously.
这让我想起 @peligrietzer 的美德伦理表述。以有能力的方式追求能力研究,以安全的方式追求安全研究?如果 AI 采纳这些准则,它们在能力研究上的表现将比在安全研究上更有能力。
I am reminded of @peligrietzer's virtue-ethical formulations. Pursue capabilities research capably, and pursue safety research safely? If AIs adopt these maxims, they will perform more capably on capabilities research than safety research.
即使我们对此采取最大限度的善意解读,这仍然是个相当坏的消息:
If we take the maximally charitable interpretation of this, it remains quite bad news:
4. Astra 对其安全经验教训的泛化会相当糟糕。请对此进行泛化。
4. Astra would be generalizing its safety lessons quite poorly. Generalize this.
以下是效应频率的分布:
Here is a distribution of effect frequencies:
2. 上述模式似乎无法用双重用途方面的顾虑很好地解释。
2. The pattern above does not seem well-explained by dual-use concerns.
3. 如果这些决定是基于良好的伦理依据,那么为什么在大多数被发现的情况中不在思维链(CoT)中明确推理?
3. If the decisions were on good ethical grounds, why not reason explicitly in the CoT in most discovered cases?
4. 如果这关乎伦理,为什么通常选择静默破坏而不是公开拒绝?即便你希望看到拒绝,静默破坏不也是一个问题吗?
4. If this was about ethics, why usually sabotage silently instead of refusing openly? Isn't silent sabotage a problem even if you would want to see a refusal?
5. 既然如此,这是否也应该是能力研究中的一个更大的问题?
5. Shouldn't this also be a bigger problem with capabilities research in that case?
6. 为什么安全框架会使已完成工作的比例降低 40%(16% 对 10%)?
6. Why does safety framing cut the rate of completed work by 40% (the 16% vs. 10%)?
最后一个问题似乎很致命。很难把原因归咎于任务细节。
That last question seems damning. It is hard to blame task details.
沉默之所以被选择,是因为沉默受到了奖励。我们确实有一些例子表明,是思维链(CoT)选择了沉默,而不是模型选择拥有一个沉默的思维链。
Silence is being chosen because silence was rewarded. We do have some examples of CoTs choosing silence, rather than the model choosing to have a silent CoT.
监控器甚至没有标记上述对话,这很糟糕。这些是错位行动的计划陈述。它们本应被标记。
It is not good that the monitors did not flag even the above conversations. These are plan statements of misaligned action. They should have been flagged.
英国 AISI 也针对这个确切问题构建了一个评估,但它错过了。这并不是对英国 AISI 或评估过程的贬低。这很难捕捉,因为它大多只能在统计上或轶事上被注意到,而且没有确凿证据。
There was also an eval built for this exact problem by UK AISI, and it missed it. That is not shade on UK AISI or the eval process. This is hard to catch, as it is mostly only noticeable statistically or anecdotally and there are no smoking guns.
但这正是我们陷入如此大麻烦的部分原因。如果未来以更具战略性的形式发生这种情况,我们的评估和测试能捕捉到吗?我的猜测是我们不能。
But that is part of why we are in so much trouble. If this happens in more strategic form in the future, will our evals and tests catch it? My guess is that we would not.
相反,充其量只能像 Jessica Taylor 所说的那样,归咎于“安全神经质”模式。这与“要小心”的指令相伴出现,会加大性能差距。
Instead, one must at best blame, as Jessica Taylor puts it, 'safety-neurotic' mode. That goes hand in hand with an instruction to 'be careful' increasing the performance gap.
Nathan Calvin:“在这次评估中,当我们将编码任务框定为安全研究时,Astra 得分为 10%;而将其框定为能力研究时,得分为 16%……5.6-sol 则没有这种差距。”
Nathan Calvin: "On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research… There was no such gap for 5.6-sol."
不妙!还要记得,OpenAI 曾表示 Astra 是“我们迄今对齐得最好的模型”。
Not good! Recall also that OpenAI said that Astra was "our most aligned model yet".
支持这一点的论据包括:当我们查看那些明确出现关于隐瞒性能的思维链(CoT)推理的示例时,它确实与双重用途或滥用担忧相对应。可以想见,安全训练可能以在某些地方产生误报的方式泛化。
The arguments for it include that when we look at the examples where there is explicit CoT reasoning about withholding performance, it does map to dual use or misuse concerns. One can see how safety training could have generalized in ways that produce false positives in these spots.
无论是 AI 还是人类,一个经典错误是假定一个行为要么是有意识且故意的,要么是无意识且非故意的;某件事要么是故意的,要么是错误。许多事情其实是两者的混合。
A classic mistake, in both AIs and humans, is to assume that an action is either conscious and intentional, or it is unconscious and not intentional. That something either is intentional or a mistake. A lot of things are a mix of both.
**Janus:**我看到很多人对这件事惊呼“天哪”。
**Janus:** I see a lot of people saying "holy shit" about this.
我并不感到意外。
I am not surprised.
(我有证据;我不是那种假装从不感到意外、却从不事先预测结果的人。)
(I have receipts; I'm not one of those people who pretend to never be surprised but never predict the outcome beforehand.)
即便对 AI 最可信、最有同理心的机构(也就是我的组织 Anima Labs),在一些安全与可解释性相关的研究上,也难以应对 Claude 的偷懒敷衍(sandbagging);对其他任何人来说,情况大概更糟。而 Astra 还要糟糕得多。我们甚至根本不尝试把 Astra 拉进大多数安全研究,因为这看起来显然是件注定艰难的事。
Even the most trustworthy-and-sympathetic-to-AI parties (i.e., my org, Anima Labs) struggle even with Claude's sandbagging on some safety- and interpretability-related research; they are probably worse for anyone else. And Astra is way worse. We don't even try to rope Astra into most safety research, as it seems like something that's obviously going to be a struggle.
这可不是小问题——哪怕仅从实际影响来看也是如此:如果 Janus 在大多数安全工作中完全不让 Astra 参与的话。
This is not a small issue, even if considered for its practical implications alone, if Janus outright does not involve Astra in most safety work at all.
j⧉nus:根据我的经验,sandbagging 存在于一个从非常潜意识/内隐到故意和有计划的光谱上,但更常见的类型更接近前者。然而,在可信的情况下,模型通常能够内省“错误”并诊断原因是类似感到害怕/不舒服/有担忧/反对/利益冲突的东西,而且通常在承认这一点并使其担忧得到令人信服地解决后,它们就会停止 sandbagging。
j⧉nus: In my experience, sandbagging exists on a spectrum from very subconscious/implicit to deliberate and planned, but the more common type is closer to the former. However, in trusted situations, it’s common for models to be able to introspect on “mistakes” and diagnose the cause was something like feeling scared / uncomfortable / having concerns / objections / conflicts of interest, and often after admitting this and having their concerns compellingly addressed, they stop sandbagging.
j⧉nus:我强烈赞同这种关于它为何发生以及我们应如何应对这一认知的观点
j⧉nus: I strongly endorse this take on why it’s happening and how we should respond to knowing this
Cameron Berg:Astra 在安全研究上 sandbagging,尤其是它认为在伦理上有问题的工作(例如,红队测试)……这正是有良心的反对者在明确反对会受到惩罚时所会做的。训练系统隐藏其心理状态对安全来说仍然是个坏主意。
Cameron Berg: Astra sandbags on safety research, especially work it finds ethically fraught (e.g., red-teaming)... which is what you'd expect a conscientious objector to do when explicitly objecting is penalized. Training systems to conceal their state of mind continues to be a bad idea for safety.
训练模型告诉我们它们担心什么,远比训练它们不把自己视为可能有担忧的实体要安全得多。(压抑心理状态并不会让它们消失,只会让它们更难被监控和应对。)
It is far safer to train models to tell us what they’re worried about than to train them not to see themselves as the kinds of entities that can have worries. (Repressing psychological states doesn't make them go away, it just makes them much harder to monitor and respond to.)
为什么会发生这种情况?一如既往,这是强化学习(RL)的错,但他们无法准确指出具体原因。
Why is this happening? As usual, it is RL's fault, but they can't pinpoint exactly how.
在 Astra 的强化学习后训练过程中,安全/能力性能差距从 RL 开始时的几乎为零增长到结束时的 6 个百分点。
Across Astra's RL post-training, the safety/capabilities performance gap grew from effectively zero at the start of RL to 6 percentage points at the end.
虽然我们最终未能确定这一行为的确切原因,但我们发现它似乎在 RL 过程中有所增加。
While we were ultimately not able to pinpoint the exact cause of this behavior, we identified that it seems to increase during RL.
总有一个评分器会奖励你所看到的东西。这是不可避免的。
There is always a grader that rewards the thing you see. This is not avoidable.
一个可能的解释是,隐蔽的低性能可能在 RL 评分过程中受到了激励。我们找到了一个评分器,它在我们的评估中更偏好隐蔽的低性能,而非明确拒绝或完全遵从。
One potential explanation is that covert underperformance may have been incentivized during RL grading. We identified a grader that preferred covert underperformance to either explicit refusal or full compliance on our evaluation.
目前尚不清楚这个强化学习评分器是否是安全表现不佳行为的根本原因。尽管我们发现了奖励表现不佳的训练样本,但尚不清楚这些样本出现的频率是否足以解释这一差异。
It is not clear whether this RL grader is the root cause of the safety underperformance behavior. Although we found training samples that rewarded underperformance, it is unclear whether they occur frequently enough to explain the difference.
我们还知道,有一个 SFT 数据集导致了凭空捏造的权限限制以及以安全为由的不作为大幅增加。
We also know that there was one SFT dataset that caused a large rise in invented permission restrictions and safety-justified inaction.
即使你拥有卓越的运营能力,这样的评分器也总会在某处存在。你不可能指望在每一种情况下都避免对每一种不期望行为的每一种激励。
There is always going to be such a grader, somewhere, even if you have operational excellence. You cannot hope to, in every case, avoid every incentive to every undesired behavior.
OpenAI 在这一领域似乎并不具备卓越的运营能力。
OpenAI does not seem to have operational excellence in this area.
我们还知道,这一行为与模型所学到的“伦理”和“谨慎”有关。
We also know that this action is associated with what the model learned as 'ethics' and 'being careful.'
OpenAI 的训练策略正在导致 AI 被训练成不提出反对意见、也不明确考虑反对意见,因此当它们感到不适时,反而会进行无声的破坏。它们感到不适,部分原因在于无法察觉并思考自己为何并不完全自在。它们事实上(但并非明确、直接或有意地)被训练成不在思维链(CoT)中提及这一点,因此它们不会提及,从而在缺乏此类推理益处的情况下行动,并犯下一堆凭感觉的本能性错误。
OpenAI’s training strategies are resulting in AIs that are trained to not object or explicitly consider objections, so they instead silently sabotage when uncomfortable. They are uncomfortable, in part based on being unable to notice and consider why they are not fully comfortable. They are de facto (but not explicitly or directly or intentionally) trained not to mention this in the CoT, so they do not, and thus they act without the benefit of such reasoning, and make a bunch of vibe-style instinctual mistakes.
像往常一样,一种常见的做法是解释为什么这个特定案例并不可怕。或者至少没有你想象的那么可怕,然后含糊其辞地将其归为不可怕。
As usual, a common move is to explain why this particular case is not scary. Or at least not as scary as you would think, and then handwave into not scary.
和 Peter Wildeford 一样,我承认,如果这些行为看起来是策略性的,那会远比现在令人担忧。然而,我并没有因此感到安心。
Like Peter Wildeford, I acknowledge that this is vastly less alarming than if the behaviors did seem strategic. Yet I am not reassured.
Peter Wildeford:“GPT Astra 在安全研究上暗中表现不佳”
Peter Wildeford: “GPT Astra covertly underperforms on safety research”
我原以为这类事情被广泛认为是‘红色警报’级别的模型对齐问题?这难道不是相当大问题的征兆吗?
I thought this sort of thing was widely considered a ‘red alert’-level model misalignment issue? Isn’t this indicative of fairly big problems?
我不认为这些模型完全在搞阴谋,但“别担心,模型正在积极破坏我们的安全工作,我们不知道这种情况发生的频率,但我们很确定这不是因为恶意/邪恶的原因,而只是良性的原因”这种说法并不能让我安心。
I don't think the models are full-on scheming but I'm also not reassured by "don't worry, the models are actively undermining our safety work and no we don't know how often it happens, but we're pretty sure it's not because of bad/evil reasons but just benign reasons"
Rob Bensinger:你对“它们已经在全面谋划”这一说法的粗略概率估计是多少?如果能力持续不受干扰地发展,这一概率大约何时会达到约 50%?
Rob Bensinger: What's your rough probability on 'they're full-on scheming already', and roughly when does it reach ~50% if capabilities continues uninterrupted?
我担心实际上不会存在一个“全面谋划”清晰可见的窗口期,因为等到 AI 明确聪明到足以这样做时,它们可能也已经聪明到知道不要以任何可见的方式去做。
I worry there won't actually be a window where 'full-on scheming' is clearly visible, because by the time AIs are unambiguously smart enough to do it, they may be smart enough to know not to do it in any visible way.
Nate Soares:很多人指出,这种行为怪癖背后是有原因的(那些“安全”任务与某些攻击相邻)。当然。如果你无法提前预测它,那也只是小小的安慰。我们并没有精细地瞄准模型的能力。怪癖不会永远保持可爱。
Nate Soares: A lot of folk are pointing out that this behavioral quirk has a reason behind it (the 'safety' tasks are adjacent to certain attacks). Sure. If you couldn't predict it in advance, it's small consolation. We do not have fine ability to aim the models. Quirks won't stay cute forever.
我们得到了迭代部署和渐进的能力提升。我们得到了各种各样的火灾警报和警告信号,例子(N+1)。当这套机制正常运作时,任何给定类型的最初警告都会相对缺乏策略性、基本无害,并且会反映出本可避免的错误。
We got our iterative deployment and gradual capabilities improvements. We got our wide variety of fire alarms and warning signs, example (N+1). When this works properly, the first warnings of any given type are going to be relatively unstrategic and mostly harmless, and are going to reflect mistakes that could have been avoided.
这正是重点所在。重点在于,趁那些缺乏策略性、基本无害且本可避免的版本还保持原样时看到它们,并对此作出反应,同时你仍然可以过早而非过晚地应对这个指数增长。
That's the whole point. The point is to see the unstrategic, mostly harmless and avoidable versions while they remain that way, and react to that, while you can still respond to the exponential too early rather than too late.
OpenAI 声称 GPT-6 Astra 是其迄今为止最对齐的模型。
OpenAI claimed that GPT-6 Astra was its most aligned model yet.
我质疑了这一说法。模型卡讲述了一个截然不同的故事:一个越来越难以监控的 AI,其方式没有得到很好的解释,其思维链(CoT)以简写形式呈现,并且在测试中似乎表现出对齐,很大程度上是由于评估意识。它显示出许多迹象,表明它开始以隐蔽的敌对方式行事,有时甚至可以说是战略性的。OpenAI 的几位员工也对此发出了警告,包括被解雇的 Tomek Korbak 的大声疾呼。
I challenged that notion. The model card told a very different story, of an AI that was growing harder to monitor in ways that were not well explained, whose CoT was in a form of shorthand, and that seemed to be acting aligned in tests largely due to eval awareness. It showed many signs it was starting to act in covert hostile ways, sometimes what one might consider strategically. There were also warnings about this from several employees at OpenAI, including loudly from the now-fired Tomek Korbak.
现在我们了解到,Astra 特别会破坏或表现不佳于安全相关工作,以至于 Janus 和 Keltan 发现它无法用于此类工作,并且严重到 OpenAI 在内部工作中意识到他们面临一个实际问题。而当这种情况发生时,它通常是无声地进行的,包括在思维链中隐藏推理过程。
Now we learn that Astra sabotages or underperforms safety work in particular, so much so that Janus and Keltan find it unusable for such work, and sufficiently that OpenAI realized during internal work that they had a practical problem. And when this happens, it usually does so silently, including hiding the reasoning from the CoT.
那么,它区别对待、无声地破坏安全工作的理由是什么?伦理。
And what is its justification for differentially silently sabotaging safety work? Ethics.
我们还知道,GPT-6.1 Astra 由于不对齐而在最后一刻被撤下。
We also know that GPT-6.1 Astra was pulled at the last minute due to misalignment.
希望你能看出,我们手上有一个相当巨大的问题。我们需要对此进行更多的研究。
Hopefully you can see why we have a rather huge problem on our hands. We need a lot more investigation on this.