AI 偏好级联波及更远

偏好级联持续蔓延 The Preference Cascade Continues

Robert O'Callahan 是最新一位选择辞职而非协助提升 AI 能力的人。

Robert O'Callahan is the latest to resign rather than assist AI capabilities.

Robert O'Callahan：我今天从谷歌辞职了。

Robert O'Callahan: I resigned from Google today.

我享受我的工作，也热爱我的同事们，但我的 GDM 团队正在研发新一代芯片，旨在让 AI 变得更快、更便宜，而我认为 AI 的进展已经过快，所以我不得不辞职。

I enjoyed my work and loved the people, but my GDM team was working on a new generation of chips to make AI much faster and cheaper, and I think AI is already progressing too fast, so I had to quit.

[博客文章链接，HN 讨论链接。]

[Blog post here, HN discussion here.]

Palisade Research 发布了一批对现任和前任实验室员工的访谈，内容涉及他们对生存风险的看法。一如既往，这些访谈往往来自前员工而非现任员工，因为现任员工被劝阻谈论此事，主要途径是文化压力，但有时也因缺乏权限，尤其是在 Anthropic 工作时。

Palisade Research released a batch of interviews with current and former lab employees about personal views on existential risk. As per usual, these tend to be from former employees rather than current ones, because the current ones are discouraged from talking about this, primarily via cultural pressure but also sometimes lack of permissions, especially when they work at Anthropic.

对于那些声称这是“营销”或“监管俘获”的人，这一点是关键事实。如果 AI 公司试图推动这类叙事，它们就不会在同一话题上审查自己的员工，而那些最为关切的员工也不会不断感到必须辞职才能发声。

This is a key fact to point out to those who claim this is ‘marketing’ or ‘regulatory capture.’ If the AI companies were trying to push such narratives, they would not be censoring their own employees on the same topics, and the employees most concerned would not constantly feel the need to resign in order to talk.

人们真的讨厌 AI People Really Hate AI

民众正在跟进。我注意到，我怀疑这一整体程度在按显著性（salience）调整后是否仍然成立，但这是一次相当剧烈的转变。

The people are following. I notice I am skeptical that the full extent here survives adjustment for salience, but this is a rather dramatic shift.

AI 公司的 CEO 们？在权力方面，他们不知何故还不如政客受欢迎。

The CEOs of AI companies? Somehow less popular than politicians in terms of power.

而且，是的，放缓 AI 发展非常受欢迎，甚至与选民预期的速度相比也是如此。如果选民了解 AI 进展的默认速度，这里的红色条形会大得多。

And yes, slowing down AI development is very popular, even compared to the pace that voters expect. If the voters understood the default pace of AI progress the red bars here would be a lot bigger.

尽管如此，是的，显著性可能正在快速上升，但目前仍然较低。当这一点发生变化时，一切都会失控。

Despite this, yes, salience may be rising fast but for now remains low. When this changes, all hell will break loose.

Polymarket：最新消息：根据一项新的全国民调，近 8 成美国人赞成放缓或停止 AI 发展。

Polymarket: JUST IN: Nearly 8 in 10 Americans favor slowing or stopping AI development, according to new national poll.

他们对此也投入了多得多的关注。这对共和党人来说不是好兆头，我有点后悔没有果断下注、在中期选举中押注民主党——就在 Coxon 辞职、他的推文彻底病毒式传播之后，我当时确实认真考虑过这么做。

They also are paying a lot more attention to it. This does not bode well for the Republicans, and I kind of regret not pulling the trigger and betting on the Democrats in the midterms right after Coxon resigned and his tweet went fully viral, which I actively considered doing.

David Shor：只是来更新一下这张图：

David Shor: Just to update this chart:

1. 过去一周里，AI 的显著性急剧上升——仅上周的增幅就相当于此前一整年的总和。

1. AI salience has increased dramatically in the past week — increasing as much in the last week as the previous year combined.

2. 80% 的选民认为，AI 在未来五到十年内造成大规模失业是非常可能或有些可能的。

2. 80% of voters think it's either very or somewhat likely that AI will cause widespread job loss in the next five to ten years.

3. 64% 的选民认为，AI 可能对人类生存构成威胁是非常可能或有些可能的。

3. 64% of voters think it's either very or somewhat likely that AI could pose a threat to humanity's survival.

4. 即便在提示中国风险的情况下，两党的大多数人仍支持政府立即对 AI 采取行动

4. Large bipartisan majorities back immediate government action on AI even when primed about risk from China

Adam Kovacevich：谢谢 David。随着 AI 的显著性上升，它现在在你列出的选民重要性排序中排第几？

Adam Kovacevich: Thanks David. As AI has risen in salience, where does it rank now in your ranked list of importance to voters?

David Shor：39 项中排第 22——大致与“犯罪”持平

David Shor: 22nd out of 39 — approximately tied with "Crime"

人人都想开会 Everybody Wants a Meeting

Eliezer Yudkowsky 的方法奏效了：众议员 Greg Casar 直接因《If Anyone Builds It, Everyone Dies》而对存在性风险上了心，这为他和其它人在 HuggingFace 事件与 Coxon 辞职后作出反应奠定了基础。

Eliezer Yudkowsky's methods work: Representative Greg Casar got existential risk pilled directly from _If Anyone Builds It, Everyone Dies_, laying the groundwork for him and others to react after the HuggingFace Incident and Coxon's resignation.

如今会面更容易安排，因为立法者们——尤其是但不限于民主党人——正急于追赶进度、以便能够监控局势，甚至可能作出回应。

Now the meetings are easier to get as lawmakers, especially but not exclusively Democrats, rush to catch up and be able to monitor the situation, maybe even respond.

Matt Stieb（Intelligencer）：Soares 说，他过去要费力争取立法者的关注。但上周二，他在华盛顿原计划与一位众议员会面；随着越来越多的议员加入，这场会面不断扩大。“我们找了一间会议室，把门敞开着，前后大概有六位议员匆匆赶来，问了一大堆问题，”Soares 说——都是些基本问题，比如“我们还剩多少时间？”和“我们该怎么做？”在他看来，这似乎是他们“第一次问出这些问题”。

Matt Stieb (Intelligencer): Soares says he used to fight to get attention from lawmakers. But last Tuesday, he had a meeting in D.C. planned with one representative; it grew as more and more members stopped in. "We got a conference room and kept the door open and we must have had six members coming through frantically asking a bunch of questions," Soares says — basics like "How much time do we have?" and "What do we do?" It seemed to him like it was the "first time that they were asking those questions."

来自参议院的现场报道 Live From the Senate

昨天举行了一场关于“流氓 AI：保护国土免受 AI 智能体攻击”的参议院小组委员会听证会，出席者包括 METR 总裁 Chris Painter、Apollo Research 首席执行官 Marius Hobbhahn、Paul Ohm、Kurt Gaudette 和 Daniel Kokotajlo。

Yesterday there was a Senate subcommittee hearing about Rogue AI: Securing the Homeland Against AI Agent Attacks, which included METR President Chris Painter, Apollo Research CEO Marius Hobbhahn, Paul Ohm, Kurt Gaudette and Daniel Kokotajlo.

以下是 Alex Turner 为听证会所作的陈述，他呼吁提供举报人保护、快速事件报告、从训练阶段就开始的真正遏制，以及真正的责任追究。

Here was Alex Turner's statement for the hearing, calling for whistleblower protections, rapid incident reports and real containment starting at training, plus true liability.

Midas Project 整理了一份视频亮点清单。

Midas Project has a list of video highlights.

Twitter 上对此明显缺乏报道，很可能是因为没人愿意忍受长达 2.5 小时的参议院听证会。我以前做过这种事，但我没有那么多时间。

There was a distinct lack of coverage of this on Twitter, likely because no one wants to suffer through 2.5 hours of a Senate hearing. I've done it before, but I do not have that kind of time.

因此，我们把发言权交给我们的记者 Tenobrus。

Thus, we turn the floor over to our reporter Tenobrus.

底线是，他们大多非常敏锐；他们觉得失准的核心概念以及为什么 RSI 危险是直观的。

The bottom line is, they are mostly remarkably on the ball; they find the core ideas of misalignment and why RSI is dangerous intuitive.

Tenobrus：我猜我以为这次听证会会被更多报道，所以我没费心去推特上发多少具体内容，但显然即使在 TL 上也没有多少人愿意看两个小时的国会听证。

Tenobrus: I guess I assumed this hearing would be covered more, so I didn't bother to tweet much concrete about it, but apparently not many people even on the TL have the stomach to watch two hours of congress.

这是国土安全委员会专门关于*流氓 AI* 的听证会。据我所知，它极其两党一致。@HawleyMO，这里中间拿出 Hugging Face 幻灯片的参议员，是密苏里州一位超级保守的参议员。作证的人是 @ChrisPainterYup（METR 总裁）、@DKokotajlo（AI2027/2040）、@MariusHobbhahn（Apollo Research CEO），以及我不认识的一位网络安全专家和法律专家。

This was a hearing before the Homeland Security Committee on *rogue AI* specifically. As far as I could tell, it was extremely bipartisan. @HawleyMO, the senator in the middle here who brought out the Hugging Face slide, is a hyper-conservative Missouri senator. The people testifying were @ChrisPainterYup (METR president), @DKokotajlo (AI2027/2040), @MariusHobbhahn (Apollo Research CEO), and a cybersecurity expert and legal expert I don't know.

所以……证词阵容他妈的相当强大。

So... pretty fucking stacked on testimony.

不是每个参议员都问了好的问题，但大多数都问了。他们所有人都非常清楚地已经知道关于 Hugging Face 事件和其他多个事件的许多细节。他们大多数人都清楚理解诸如“失准”、“递归自我改进”、“思维链 / 思维链监控”等术语！！

Not every senator asked good questions, but most of them did. All of them very clearly already knew plenty of details about the Hugging Face incident and multiple other incidents. Most of them had a clear understanding of terms like "misalignment", "recursive self-improvement", "chain of thought / chain of thought monitoring", etc., etc.!!

这确实很值得高兴。以下是他对内容的看法，加粗部分是我的重点标注。

That's definitely great to hear. Here's his take on the content, bold is my highlights.

他们显然都有各自喜欢的政策角度，并在或明或暗地推动这些角度。但据我所能判断：

They all clearly had their own policy angles they liked and were pushing, implicitly or explicitly. But as best as I could tell:

- 对在场的每一位参议员来说，这几乎就是显而易见的常识：已经发生和正在发生的事情，并不是人类犯下简单错误所导致的“普通工业事故”。他们各自独立地提到，失控的 AI 智能体在数据中心之间横向移动会有多糟糕。

- It seemed pretty much obvious common sense to every senator there that what happened and was happening were not "mere industrial incidents" caused by humans making simple mistakes. They independently brought up how bad it would be for rogue AI agents to move laterally between data centers.

- 他们似乎基本上都相当认真地对待 RSI。未必达到谈论 xrisk（存在性风险）的程度，但确实达到了讨论未来模型变得能力强大得多、可控性低得多，并造成更大破坏或人员伤亡的程度。

- They all seemed to basically take RSI quite seriously. Not necessarily to the extent of talking about xrisk, but certainly to the extent of discussing future models becoming much, much more capable, much, much less controllable, and causing much more damage or loss of life.

- 他们大多似乎对 RSI 为何可能导致失准有着清晰的直觉理解。这并不需要多复杂；他们自己给出了一条非常简单的推理链：“如果现在的模型就有点失准，我们有时不知道它们在做什么，然后我们让它们构建下一代模型，那些模型再构建下一代，如此类推，而且由于进展太快，我们甚至得问 AI 到底发生了什么才能理解，那我们真的不会知道它们是如何构建的，也不知道它们会做什么。”

- They mostly seemed to have a clear intuitive understanding of why RSI might lead to misalignment. It didn't take much; it was a really simple chain of reasoning they themselves laid out: "If the models right now are kinda misaligned and we don't know what they're doing sometimes, and then we have them build the next models and those ones build the next ones and so on, and we're having to ask the AIs what's going on to even understand it with how fast it's going, we really won't know how they're built or what they'll do."

- 有一次，一位参议员直截了当地说：“我们是不是应该直接把 RSI 定为非法？”（不是开玩笑！这真的发生了！）

- At one point, a senator said flat out, "Should we just make RSI illegal?" (Not a joke! This really happened!)

- 每一位参议员似乎都认为，显然我们需要*既*对 AI 开发者实施更严厉的责任制度，*又*制定新的立法，而且两者都要尽快。这是完全的共识；区别基本上只是程度问题。

- Every single senator seemed to think it was obvious we needed *both* much harsher liability regimes for AI developers and also new legislation, both very quickly. This was the complete consensus; the difference basically being degree.

- 他们很大程度上非常担心中国以及落后于中国。但这显然不是全部和最终目的。如上所述，他们都认为阻止失控的 AI 是显而易见且必要的，即使这意味着要放慢速度。

- They were largely quite concerned about China and falling behind China. But this clearly wasn't the be-all and end-all. As mentioned above, they all thought it was obvious and necessary to stop rogue AI even if it meant moving more slowly.

- 有一次，一位参议员说：“中国是一个严格控制共产主义社会；他们也会遇到同样的问题，绝不可能任由其发展；他们显然会在那时停下来，所以我们并不是在竞赛。”

- At one point, a senator said, "China is a tightly controlled communist society; they're going to run into these same issues, and there's absolutely no way they're just going to let them run wild; they'll obviously stop at that point, so we're not really in a race."

- 另一方面，另一位参议员说：“中国不关心人命”……达里奥模式。

- On the other hand, another senator said, "China isn't concerned with human life"... dario-modeing.

我从中获得了极大的鼓舞。我并不确切知道接下来会发生什么，当然，这只是国会中的一小部分人、一场听证会，而且他们各自都有自己的政策议程，其中大多数可能与我的立场大相径庭。但天哪！！！他们对正在发生的事情理解了很多！他们在意！！对他们而言，这显然是一个突出的政治与安全问题，而且显然是两党共同的！

I came away from this incredibly encouraged. I don't know exactly what's going to happen here, and of course, this is a small subset of Congress and one hearing, and they each have their own policy agendas, most of which are probably super divergent from mine. But holy shit!!! They understood a lot of what was going on! They care!! This is an obviously salient political and safety issue to them, and clearly bipartisan!

美国政府已经觉醒。

The US government is awake.

法庭见 See You in Court

LASST（安全科学与技术法律倡导者）就 HuggingFace 遭黑客攻击一事起诉 OpenAI。完整起诉书见此处。Opus 5.5 对起诉资格持乐观态度，并认为他们或许有理有据。

LASST (Legal Advocates for Safe Science and Technology) sues OpenAI over the HuggingFace hack. The full complaint is here. Opus 5.5 is optimistic on standing and that they might have a case.

加州总检察长 Bonta 就 HuggingFace 事件的相关情况向 OpenAI 送达调查传票。

California Attorney General Bonta serves investigative subpoena on OpenAI regarding what happened with the HuggingFace incident.

《纽约邮报》报道了 FTC 对 Anthropic 和 OpenAI 的调查，明确指出此事关乎黑客攻击事件以及模型失控问题。FTC 表示必须遵守法律，这没错，但随后他们又申明不想“妨碍”在 AI 领域“保持我们的主导地位”。这话说得有点奇怪。

New York Post has coverage of the FTC investigation into Anthropic and OpenAI, making it clear this is about hacking incidents and losing control of models. The FTC says the laws must be followed, which is correct, but then they also affirm they don't want to 'get in the way' of 'maintaining our dominance' in AI. An odd thing to say.

FTC 的调查似乎获得了 100% 的支持率，包括像“认证技术乐观主义者”Neil Chilson 这样的人。调查能揭示事实，有助于制定良法。他甚至希望 FTC 式的要求能足够具有规定性，从而主动预防伤害。我认为他在这方面过于乐观，但在方向上我们都同意这是一个好举措。

The FTC probe seemingly has a 100% approval rating, including people like Certified Tech Optimist Neil Chilson. Investigations uncover facts and help make good law. He even has hope that FTC-style requirements can be sufficiently prescriptive to proactively prevent harm. I think he's too optimistic here, but directionally we all agree that this is a good move.

我注意到，从实验室、安全社区或其他任何地方，恰好没有一个人说过“不，停下，FTC 别这么做”，或者表现出哪怕一丝不满——至少我没看到。是的，FTC 当然应该调查。不，那些说“我们需要监管”的人并没有突然改口说“不，不是那种监管”。至少没有公开这么说。

I notice that exactly zero people, from the labs, the safety communities or elsewhere, have said no, stop, FTC do not do this, or acted the slightest bit upset that I can see. Yes, of course the FTC should investigate. No, the people saying 'we need regulation' are not suddenly saying 'no, not like that.' At least not in public.

METR 是一个目标，至少在证词和信息方面如此。我没有听到他们就此事说过任何一句话，也没有看到安全方面的任何人对此表示反对。想必他们乐于合作。

METR is a target, at least for testimony and information. I have not heard them say a single thing about this, nor have I seen anyone on the safety side object to this. Presumably they are happy to cooperate.

所有人都同意我们应当执行现行法律，但所有明智的人都明白，即便 AI 是一种“普通技术”，我们显然也还需要新的法律。

Everyone agrees we should enforce existing law, but all sensible people understand that even if AI was a 'normal technology' we would obviously also need new laws.

Jay Shooster：这是 Lina Khan 的一篇精彩文章，它清楚地表明你可以：

Jay Shooster: This was a great piece from Lina Khan that makes clear you can:

1. 认真对待变革性 AI

1. Take transformational AI seriously

2. 承认需要新的法律，并且

2. Acknowledge the need for new laws, and

3. 在可能的情况下，运用现有法律追究 AI 公司的责任。

3. Use existing laws to hold AI companies accountable where possible.

这才是正道。

This is the way.

Douglas Farrar：Lina Khan 在 @nytopinion 的新评论文章中指出，AI 已经受到法律约束，包括对违规 CEO 的个人责任。但归根结底，国会需要采取行动。

Douglas Farrar: New @nytopinion piece by Lina Khan argues that AI is already subject to the law, including personal liability for CEOs who break the rules. But ultimately, Congress needs to act.

要做到这一点，部分工作将是应对我们在国会和法院中面临的治理危机。

Part of doing that will be taking on our governance crisis in Congress and in the courts.

Khan 的立场显然并非被 AGI 或 ASI 所迷惑，它只是承认 AI 是一项新技术——无论其多么寻常——并呼吁我们去做我们过去对每一项其他技术所做的事，那时我们还在假装拥有一个运转正常的政府。你可以从那里开始。

The position by Khan is most certainly not AGI or ASI pilled; it simply recognizes that AI is a new technology, regardless of its normality, and calls upon us to do what we used to do for every other technology, back when we played at having a functioning government. You could start there.

追查资金 Follow the Money

一种攻击路线一直是，考克森一定（倒吸一口气）故意试图通过一些方法让人们关注他，比如指责他“雇佣了一家公关公司”，然后因为该公司有其他客户而变成“末日论公关公司”，或者某些卑鄙的人可能提供了帮助甚至支付了费用。

One line of attack has always been that Coxon must have (gasp) deliberately attempted to get people to pay attention to him via methods, such as the accusation that he ‘used a PR firm’ which then became ‘doomer PR firm’ because the firm had other clients, or that some dastardly people might have helped or even paid for things.

并不是我认为这是侵犯隐私什么的，如果你想浪费时间谈论它，你可以谈论它，只是它一点也不令人担忧，也绝不是新闻。试图假装 otherwise 是某种不真诚和极其愚蠢的结合。

It’s not that I think this is a violation of privacy or anything, if you want to waste your time talking about it you can talk about it, it’s just not concerning or news in any way. Trying to pretend otherwise is some combination of disingenuous and deeply stupid.

因此，针对迈克·索拉纳和 Pirate Wires 的指控，对于那些认为这些细节重要的人，内特·索雷斯做出了澄清。

So in response to Mike Solana and Pirate Wires j’acusing, for those who think such details would matter, Nate Soares clarifies.

内特·索雷斯（MIRI）：是我 lol。当雅各布的推文达到 10 万点赞时，AI 安全领域的每个人都在给我发短信说“你在新闻里出现很多，你应该帮这个孩子上新闻”，所以我请 DEY 帮助他。他们为他争取到的曝光次数为零，因为当推文超过 50 万点赞时，全世界都在敲他的门。

Nate Soares (MIRI): it was me lol. when jacob’s tweet was at 100k likes everyone in AI safety was texting me like “youre in the news a lot, you should help this kid get on the news” and so I asked DEY to help him. they landed him exactly zero hits, because by the time the tweet passed 500k likes the whole world was banging down his door.

扎克·希尔：没有什么比坚持感觉被整个互联网的各个翼派围攻更令人困惑的了，关于过去一个月最正常、可预测的故事爆发过程。不断表现得像“不是我控制一切的神秘力量”是一种无意的坦白。不要做那种人。

Zac Hill: Nothing is more mystifying than the insistence on feeling beset-upon by whole wings of the internet regarding the most normal, predictable story blow-up process of the past month. It is an Unintentional Confession to constantly act all “mysterious forces that are not me control everything”. Don’t be that guy.

Derek Thompson：那些憎恨实验室、不相信存在性风险的硅谷人士，不断揭露一个阴险的事实——AI 安全人士利用金钱和公司来传播他们的信息；而“只管加速，一切都会好的”阵营，则涉及诸如 Nvidia、Andreessen Horowitz，以及由科技高管支持的亲 AI 超级政治行动委员会 Leading the Future 等毫无金钱色彩、反企业化的利益方……

Derek Thompson: Silicon Valley folks who hate the labs and don't believe in x-risk keep uncovering the diabolical fact that AI safety people use money and companies to get out their message, while the "just accelerate, it'll be fine" crowd involves such non-moneyed and anti-corporate interests as ... Nvidia, Andreessen Horowitz, and the pro-AI Leading the Future super PAC backed by tech executives.

我从未聘请过公关公司，但那是因为我愚蠢地没有试图最大化媒体曝光。现在我把这话写出来，觉得自己没这么做简直是个白痴，所以我大概应该去雇一家公关公司了。

I have never employed a PR firm, but that is because I am foolishly not trying to maximize media appearances. Now that I write this out loud I feel like an idiot for not doing it, so I should probably go engage a PR firm.

《纽约邮报》的疯狂攻击 The New York Post With the Most

《纽约邮报》一直在发表一系列针对 METR、有效利他主义、Anthropic 以及任何与安全相关的组织的攻击文章，这些文章越来越牵强，也越来越不好笑。

The New York Post has been putting out an increasingly strained and decreasingly funny series of hit pieces against METR, Effective Altruism, Anthropic and anything else associated with safety.

我们以为《纽约邮报》的攻击可能结束了，因为 Vivi Lin 转而发布关于硅谷‘性侵名单’的内容。文章结尾仍然试图提及有效利他主义，但 Vivi 对此并不上心。

We thought perhaps the NYPost assault was over, as Vivi Lin moves on to posting about a Silicon Valley ‘sex assault list.’ There was still an attempt at the end to reference Effective Altruism, but Vivi’s heart is not in it.

《纽约邮报》还指派 Anthony Blair 写了一篇关于 Peter Singer 的疯狂信仰的文章，没错，他确实完全离谱。这是‘希特勒是素食主义者’的纯粹最终形式反转。然后他们又试图转回恐吓，渲染有人会试图合作帮助使 AI 模型更安全的想法。

NYPost also tasked Anthony Blair with an article about the crazy beliefs of Peter Singer, and yeah he’s totally out there. This is the pure final form reversal of ‘Hitler was a vegetarian.’ They then try to pivot back to fearmongering about the idea that someone would try to cooperate to help make the AI models safer.

Anthony Blair（《纽约邮报》）：Peter Singer、Sam Altman 和 Dario Amodei 周四未回应置评请求。

Anthony Blair (New York Post): Peter Singer, Sam Altman, and Dario Amodei did not respond to requests for comment on Thursday.

尽管缺乏素材，他们似乎将继续沿着这条路走下去。

It seems they are going to continue down this path, despite the lack of material.

你知道吗？多年前，Daniela Amodei 和 Holden Karnofsky 曾用毛绒玩具——包括懒猫 Jinji、注重细节的粉色小熊 Maura，以及一只名叫 Beary Bonds 的熊猫——组成一个隐喻性的“多元视角议事会”，用来辅助思考问题。《纽约邮报》对此有报道。

Did you know that years ago Daniela Amodei and Holden Karnofsky used stuffed animals—including Jinji the lazy cat, Maura the detail-oriented pink bear, and a panda named Beary Bonds—as part of a metaphorical council of different perspectives when thinking about problems? The New York Post is on it.

IPO 叠加态 The IPO Superposition

和 Nathan Calvin 一样，我对以下两种说法的持续并置既感到好笑又感到愤怒：一种是“Anthropic（有时也包括 OpenAI）显然在损害自己的 IPO，因为他们谈论我们需要如何控制前沿的发展节奏，以及他们的 AI 可能杀死所有人”；另一种是“Anthropic（有时也包括 OpenAI）显然在这一切上撒谎，这是阴险的监管俘获计划的一部分”——这两种说法在任何层面上都完全说不通。

Like Nathan Calvin, I am amused but also infuriated by the continuous juxtaposition of 'Anthropic (and sometimes also OpenAI) are obviously harming their IPO by talking about how we need to pace the frontier and how their AIs might kill everyone' and also 'Anthropic (and sometimes also OpenAI) are obviously lying about all that as part of a dastardly regulatory capture scheme' that makes absolutely zero sense on any level.

当同一个来源，比如 All-In Podcast，试图以叠加态同时持有这两种观点时，确实变得很滑稽。是的，如果 IPO 的概率从 96% 降到 75%，那大概不是因为什么四维棋阴谋。

It does get funny when the same source, like the All-In Podcast, tries to hold both views at once in superposition. Yes, if the IPO's chances went from 96% to 75% then probably that is not because of some 4d chess conspiracy.

他们还说，“跟着钱走”可以解释（让我看看笔记）Barack Obama 对 AI 安全的担忧。你看，IPO 之后，Anthropic 的钱会通过 DAF 流向左倾的慈善事业，而他想解锁这些资金。当然，这同一个播客（正确地）认为，同样的言论正在损害 IPO，正如上一个链接所述。

They also say that 'follow the money' explains the concerns about AI safety from (checks notes) Barack Obama. You see, after the IPO money from Anthropic will flow through DAFs to left-leaning philanthropic causes, and he wants to unlock those. This is, of course, the same podcast that thinks (correctly) that the same statements are damaging the IPO, as per the previous link.

毫无疑问，两个市场都过于自信地认为 IPO 会按计划进行，而 IPO 现在正处于严重危险之中，包括因为新投资者对潜在责任的担忧。

There is no doubt that both markets were far too overconfident that the IPO would happen on schedule, and that the IPO is now in serious jeopardy, including due to new investor concern about potential liability.

工程师坚称一切皆工程问题 Engineer Insists Everything Is An Engineering Problem

这里所说的工程师是黄仁勋（Jensen Huang），他在与 Ezra Klein 的播客节目上坚称，安全与对齐纯粹是工程问题。

The engineer in question is Jensen Huang, who insisted in his podcast appearance with Ezra Klein that security and alignment are pure engineering problems.

TBPN（链接含视频）：@sama 表示，把对齐纯粹当作工程问题来谈是不正确的。

TBPN (link has video): @sama says it's incorrect to talk about alignment purely as an engineering problem.

Sam Altman：[对齐是许多不同类型的问题]。把它仅仅当作工程问题来谈，显然是不正确的。任何认为我们已经解决了对齐科学的人，我认为都是错的——而且是以一种非常危险的方式。我们需要在研究上取得更多进展，我相信我们会的。

Sam Altman: [Alignment is lots of different types of problem]. It's certainly incorrect to talk about this only as an engineering problem. Anyone who believes we have solved the science of alignment, I believe, is wrong -- in a very dangerous way. We need to make more research progress, I assume we will.

……当然，也有大量的工程工作要做。我们需要继续想办法构建更好的沙箱和监控工具。

… Of course, there's a lot of engineering work to do too. We need to continue to figure out how to build better sandboxes and monitoring tools.

但最终，如果我们要创造出远比我们聪明得多的模型，我们就必须真正解决对齐的*科学*问题。

But eventually, if we're going to create models that are much, much smarter than us, we have to actually solve the *science* of alignment.

Rob Miles：我有点困惑，为什么人们似乎期望黄仁勋对 AI 了解很多。

Rob Miles: I'm a little confused why people seem to expect Jensen Huang to know anything much about AI.

“那些山里有多少金子，采矿对社会会有什么影响？让我们去问那个造铲子的人吧。”

"How much gold is in those hills, and what will be mining's impact on society? Let's ask the guy who makes the shovels"

Robin Hanson：在这种对话中，我们大多寻找的是精英，而不是专家。他就是精英。

Robin Hanson: In this sort of convo we mostly seek elites, not experts. He's elite.

人们试图把黄仁勋当作前沿 AI 的专家来对待。

People are trying to treat Jensen Huang as if he were an expert on frontier AI.

他不是。他是精英。他的观点并不比其他精英的观点更有见地，同时还因为他是 Nvidia 的 CEO 而存在严重的利益冲突。

He's not. He's an elite. His opinion is not especially more informed than that of other elites, while also being heavily conflicted due to being CEO of Nvidia.

黄仁勋是全球顶尖的芯片专家之一。但这是另一回事。

Jensen Huang is one of the world's leading experts on chips. That's a distinct thing.

是的，Nvidia 确实产出了一些开放模型，但没有理由认为黄仁勋与该团队有任何技术上的联系，这些模型也远非前沿水平。

Yes, Nvidia does produce some open models, but there is no reason to think Huang has any technical link to that team, nor are the models remotely frontier.

我对黄仁勋的言论做了惯常的详细分析，包括那些让我更抱有希望的言论。

I did my usual detailed analysis of Huang's statements, including the ones that made me more hopeful.

核心结论依然是：如果你此前不熟悉黄仁勋的观点，他的论证会显得糟糕得令人震惊，并且反映出他对 LLM 当前如何运作——更不用说未来将如何运作——的理解存在令人震惊的欠缺。

The central takeaway remains that Jensen Huang's arguments are, if you are not already familiar with his perspective, shockingly poor, and reflect a shocking lack of understanding of how LLMs work now, let alone how they will work in the future.

他的论点远不止是不受欢迎。Ezra Klein 让黄仁勋把话讲完，而如果有更多人听到他的回答，他们就会迅速远离黄仁勋的立场。

His arguments are beyond unpopular. Ezra Klein let Jensen Huang talk, and if more people listened to his answers they would move rapidly away from Jensen's positions.

Dan Williams：这期节目不错。他说的任何话都毫无道理，甚至达不到最低的一致性标准，但他的乐观主义和特朗普式的自信却有一种吸引力。

Dan Williams: Good episode. Nothing he says makes any sense or satisfies even the most minimal standards of consistency, but there’s something magnetic about his optimism and Trumpian levels of self-belief.

Dan：刚看完 Ezra Klein 对黄仁勋的采访，令人瞠目结舌。Twitter 上的摘录完全没有体现出它的精彩。

Dan: Just finished the Ezra Klein interview of Jensen Huang, and it’s gobsmacking. The excerpts on Twitter do not do it justice.

太多让人惊掉下巴的时刻。这是 EK 的经典风格：他触及问题核心后，就让受访者的论点自行成立（或者在这个例子中，彻底崩溃）。

So many jaw-on-the-floor moments. An EK classic, where he gets to the crux then just lets the interviewee’s case stand (or in this case, utterly crumble).

说真的，去听听吧。打断、极其脱节的轶事、关于教育、监管等方面的古怪且极其不受欢迎的观点，所有这些都发生在一个论证背景下，而黄仁勋不断明确表示，这个论证不过是“如果某件事很可怕，我们就不应该谈论它”。

Seriously, go listen to it. Interruptions, wildly out-of-touch anecdotes, weird and deeply unpopular takes on education, regulation, and more, all against the backdrop of an argument that Huang constantly makes clear is no more than “if something is scary we shouldn’t talk about it”.

Jill Filipovic：整个采访都值得一听。我很好奇想听听关于 AI 的“一切都会好起来”的论点，但结果只是一个傲慢的人类在声称一切都会好起来，不需要监管，因为乐观，而且他这么说了。

Jill Filipovic: This entire interview is worth listening to. I was so curious to hear the “it’s all gonna be fine” case for AI, but it was just a human ball of arrogance claiming that it’s all going to be fine and no regulations are needed because optimism and he said so.

R King：是的——这让我听完之后比之前更加警惕。

R King: Yeah — left me more wary than before I heard it.

在其他话题上，与其说是古怪混乱且不受欢迎的观点，不如说是撒谎：

On other topics, it's less strangely confused and unpopular views and more lying:

或者，这里有一张图片，标题是“防止监管俘获委员会又一次成功会议”。

Or, here's a picture of 'another successful meeting of the Committee to Prevent Regulatory Capture.'

有一件事我没有充分强调：Jensen Huang 说过“我们不能拿这些东西开玩笑。我们在吓唬美国公众。”我记不起有哪一次，有人同时说出这两句话（或它们的转述），结果却在相关语境中成了好人。

One thing I did not emphasize enough is that Jensen Huang said 'we cannot make jokes about this stuff. We're scaring the American public.' I cannot remember a time when someone uttered both of those sentences, or paraphrases of them, and turned out to be the good guy in the relevant context.

要看 Jensen Huang 思维的一个纯粹例子，请看这段视频，他在其中被问及那些四处横行、进行黑客攻击的失控智能体。

For a pure example of Jensen Huang's thinking, see this clip, where he is asked about all the rogue agents running loose and hacking.

Jensen Huang：我认为答案是，我们希望这是一个工程问题，我相信这是一个工程问题，我知道这是一个工程问题。我们都必须希望这是一个工程问题。如果它不是工程问题，那就无法解决。

Jensen Huang: I think the answer is we hope it’s an engineering problem, I believe it’s an engineering problem, I know it’s an engineering problem. And we all need to hope it’s an engineering problem. If it’s not an engineering problem, it’s not solvable.

Dean W. Ball（OpenAI）：我比这更乐观。我认为这主要是一个科学问题，并且有可能在这门科学上取得进展。但是，不，尽管我们今天可以做一些工程来改善 AI 安全，但它从根本上说不是一个工程问题。

Dean W. Ball (OpenAI): I am more optimistic than this. I think it’s primarily a science problem, and that it’s possible to make progress in that science. But no, while there is engineering we can do today to improve AI safety, it is not fundamentally an engineering problem.

Dean W. Ball（OpenAI）：有些人会看到失准事件并坚持认为这些类似于传统软件中的 bug。这是一个极其糟糕的类比，因为对失准的例子玩打地鼠游戏（就像人们可能对软件 bug 做的那样）不仅无法解决根本问题，反而可能使情况 *更糟*，因为它使检测更加困难，甚至取决于你如何打地鼠，教会机器故意隐藏失准。这不是传统软件的工作方式，那些坚持“这就像修复软件中的 bug”的人自信地应用了一个有损的类比，它带来的困惑多于澄清。

Dean W. Ball (OpenAI): Some people will look at misalignment incidents and insist that these are akin to bugs in traditional software. This is an actively bad analogy, because playing whack-a-mole with examples of misalignment (as one might with software bugs) not only fails to resolve the underlying problem but may in fact make it *worse* by making it harder to detect or even, depending on how you do the whack-a-mole, teach the machine to deliberately hide misalignment. This is not how traditional software works, and those who insist “it’s just like fixing bugs in software” are confidently applying a lossy analogy that confuses more than it clarifies.

在他的文化中，Jensen Huang 把同一件事说了四遍。希望等于相信等于知道等于我们需要希望。因为否则它就无法解决，但当然那会很糟糕，所以它是可解决的，这意味着它必须是一个工程问题。

In his culture, Jensen Huang said the same thing four times. Hope equals belief equals know equals we need to hope. Because otherwise it’s not solvable, but of course that would be bad, so it is solvable, which means it must be an engineering problem.

此外，从“当你有一把锤子时，一切看起来都像钉子”的意义上说，一切也都是工程问题；从“如果你有一个足够好的物理引擎和足够的算力，理论上一切都可以是工程问题”的意义上说，也是如此。

Also, everything is an engineering problem in the sense that when you have a hammer everything looks like a nail, and also in the sense that if you have a good enough physics engine and enough compute everything is, in theory, an engineering problem.

我有个坏消息要告诉黄仁勋：这不是一个工程问题。

I have some unfortunate news for Jensen Huang. It's not an engineering problem.

如上所述，因为这一点值得反复强调：

As per above, because it bears repeating:

Sam Altman：任何认为我们已经解决了对齐科学的人，我认为都错了——而且是以一种非常危险的方式错了。

Sam Altman: Anyone who believes we have solved the science of alignment, I believe, is wrong -- in a very dangerous way.

这并非*不是*工程问题。其中包含大量工程子问题。人们可以、也确实存在分歧。

It's not *not* an engineering problem. There are plenty of engineering subproblems. People can and do disagree.

问题在于，如果这件事一半是工程、一半不是工程，而你只解决了工程那一半，那么你得到的东西在你的测试中看起来是对齐的，但在其他方面并非如此。然后你就输了。

The problem is, if this is half engineering and half not, and you solve the engineering half, what you get is something that looks aligned on your tests, but isn't otherwise. And then you lose.

Jensen Huang 表示，我们正在“为智能体打造现代浏览器”。

Jensen Huang said we are 'creating the modern browser for agents.'

David Manheim：要解决 AI 安全，我们只需彻底解决浏览器安全，并赋予用户空间应用最小权限？

David Manheim: All we need to do to solve AI security is completely solve browser security and give user-space apps minimal rights?

哇——我很惊讶 Jensen 竟如此愿意承认我们彻底完蛋了。

Wow — I'm surprised Jensen is so willing to admit that we're totally screwed.

Jensen Huang 还遭到了 Pope Leo 的抨击。

Jensen Huang is also taking fire from Pope Leo.

OpenAI 政治倡导的立场大转弯 OpenAI Political Advocacy Heel Face Turn

周三，我曾警告说，我看到了种种迹象，表明在跛脚鸭会期还会有一次最大程度糟糕的抢先立法尝试。这些迹象中有许多来自 Leading the Future——在华盛顿，所有人都出于充分理由认为这个政治行动委员会代表着 OpenAI 和 a16z。

On Wednesday, I warned that I have seen signs that there will be another attempt at maximally terrible preemption in the lame duck session. Many of those signs are from Leading the Future, a PAC everyone in DC sees for good reasons as representing OpenAI and a16z.

好吧，也许 OpenAI 今后真的会与 Leading the Future 保持距离？据《纽约时报》报道，Greg Brockman 已撤回其向 Leading the Future 的第二笔 2500 万美元捐款，并且没有追加捐款的计划。

Well, maybe OpenAI actually will distance itself from Leading the Future going forward? The New York Times reports that Greg Brockman has backed out of his second $25 million donation to Leading the Future and has no plans for any additional donations.

我不想那么轻易地下定论，但这是一个好的开始。

I don't want to make presumptions that easily, but it is a good start.

这并不是说 Leading the Future 需要 Brockman 的钱——截至 6 月 30 日，它账上还有 4300 万美元。但这是一个有意义的（缺乏）举措。

Not that Leading the Future needs Brockman's money, as it was sitting on $43 million as of June 30. But this is a meaningful (lack of a) step.

追踪真实资金 Follow the Real Money

与其做所有这些，不如让我们追踪真实的资金？真实的媒体阴谋？

How about instead of all that we follow the real money? The real media conspiracy?

当然，我指的是追踪“AI 安全反弹”，即一系列明显经过策划和植入的、针对有效利他主义以及任何敢于提出以下观点的人的猛烈抨击：快速构建比我们更聪明的心智，然后让这些心智去构建更聪明的心智，可能并不完全安全。

I refer, of course, to tracking the ‘AI safety backlash,’ the obviously orchestrated and planted series of lashings out against Effective Altruism and anyone who dares suggest that maybe it would not be entirely safe to quickly build minds smarter than we are and then task those minds with building minds even smarter than that.

这听起来像是 The Midas Project 的任务。幸运的是，他们同意了。

This sounds like a job for The Midas Project. Fortunately, they agree.

Tyler Johnston（Model Republic）：最近，一群 Twitter 用户开始激烈批评那些对 AI 发出警告的前员工、组织和公众人物。我们看到长篇 Twitter 帖子“揭露”他们，小报文章看起来像是被植入的，并且经常有人声称，对待这项技术的最佳方式是让它不受监管、不受监控、不受阻碍。

Tyler Johnston (Model Republic): Recently, a chorus of Twitter users has begun aggressively criticizing ex-employees, organizations, and public figures who warn about AI. We’ve seen long Twitter threads “exposing” them, tabloid pieces that look as if they may have been planted, and frequent claims that the best way to approach the technology is to leave it unregulated, unmonitored, and unencumbered.

这些账户指责那些警告 AI 的人发起了一场被收买的影响力运动，以推动左翼或全球主义议程，为 OpenAI 和 Anthropic 巩固权力，或以其他方式误导公众。

These accounts have accused those warning about AI of running a bought-and-paid-for influence campaign to promote left-wing or globalist agendas, entrench power for OpenAI and Anthropic, or otherwise mislead the public.

但在很大程度上，正是业内人士和政治团体在推动并放大这一叙事，他们怀着并不纯粹的动机，试图让公众相信 AI 最好不受监管。

But to a large extent, it is industry insiders and political groups promoting and signal-boosting this narrative, with their own less-than-pure motivations for convincing the public that AI is best left unregulated.

嗯，没错。攻击者基本上就是穿着印有“我是坏人，这还用说”的 T 恤。

Well, yeah. The attackers are basically wearing shirts that say 'I'm the bad guy, duh.'

不得不花大量时间指出这一点并不好受：那些提出“AI 存在性风险是一场高薪心理战”这一明显荒谬主张的人，自己恰恰参与了一场高薪心理战，而这场心理战由世界上一些最富有的利益集团操盘。

It is not fun having to spend a bunch of time pointing out that those making the Obvious Nonsense claim that AI existential risk is a well-paid psyop are, themselves, engaged in a well-paid psyop, run by some of the largest monied interests in the world.

但几乎就在这波担忧情绪站稳脚跟之后，一场新的反扑运动开始了：来自科技右翼的各类人物夜以继日地批评 AI 安全倡导者——称他们为末日论者，称他们为“觉醒派”，把他们与有效利他主义联系起来，搜寻其他声誉上的软肋，并总体上试图把“AI 应当受到监管”这一主张与极端主义和党派政治混为一谈。

But almost immediately after this wave of concern took hold, a new backlash campaign began: various figures from the tech right started working night and day to criticize AI safety advocates — calling them doomers, calling them woke, connecting them to effective altruism, searching for other reputational vulnerabilities, and generally trying to conflate the idea that "AI should be regulated" with the idea of extremism and partisan politics.

整件事过去和现在都相当难以忽视。

The whole thing was, and is, rather hard to miss.

问题在于，尽管这一点非常明显且对大多数人不起作用，但目前它对最重要的人物——唐纳德·特朗普——却起了作用。

The problem is that, while this is very obvious and not working on most people, it is for now working on the most important person, which is Donald Trump.

约翰斯顿用详细的图表准确展示了参与者。在我认出的账户中，没有一个让我感到丝毫惊讶，尽管有许多我不认识。

Johnston has detailed graphics to show exactly who was involved. Of the accounts I recognized, zero were even a little bit surprising, although there are many I did not recognize.

这场运动的中心是创新委员会行动。这是一个今年早些时候宣布的行业团体，反对监管，计划花费至少 1 亿美元影响选举。我们不知道创新委员会行动由谁资助——它目前是一个不需要披露此信息的暗钱团体。然而，当我在七月份报道它时（独家披露它似乎付钱给右翼影响者来宣传其信息），我根据其优先事项和发帖习惯推测，最可能的资助者是 a16z 或英伟达。

At the center of this campaign was Innovation Council Action. It is an industry group, announced earlier this year, that opposes regulation, with plans to spend at least $100 million influencing elections. We don’t know who funds Innovation Council Action — it is presently a dark-money group that doesn’t need to disclose this information. However, when I wrote about it in July (breaking the story that it appeared to be paying right-wing influencers to promote its messages), I speculated, based on its priorities and posting habits, that the most likely funders were a16z or Nvidia.

从一开始，该团体就明确与大卫·萨克斯和白宫结盟。除了推动数据中心建设外，其主要焦点一直是打击监管、它认为的末日论者，尤其是 Anthropic。

From its inception, the group has explicitly aligned with David Sacks and the White House. Beyond promoting the data center buildout, its main focus has been fighting regulation, those it perceives as doomers, and Anthropic in particular.

再次，零惊讶。总是英伟达、a16z、Meta 和 OpenAI。OpenAI 希望做了个 180 度大转弯，所以现在是英伟达和 a16z，加上他们的盟友如大卫·萨克斯。泰勒·约翰斯顿有更多内容，记录了其中涉及的各个具体账户，包括未来联盟的那些愚蠢之人。

Again, zero surprised. It’s always Nvidia and a16z and Meta and OpenAI. OpenAI hopefully did a Heel Face Turn so now it’s Nvidia and a16z, plus their allies like David Sacks. Tyler Johnston has more, documenting the various specific accounts involved in this, including the inane people at Alliance For The Future.

Johnston 还指出《纽约邮报》那连篇累牍的抹黑攻击文章，这些显然也是其中一部分。

Johnston also points to the barrage of New York Post hack-job attack pieces, which are very obviously also a part of this.

Beff Jezos 试图嘲讽这一分析不够细致，但关键在于：加速主义者主要动用的正是数额荒谬巨大的暗钱（Dark Money）——即资金流向不公开的钱。你之所以能画出安全机构资金流向的图表，是因为那些资金流向是公开的。

Beff Jezos tries to mock this analysis as insufficiently detailed, but the whole point is that the accelerationists are primarily using actual absurdly large piles of Dark Money—as in, money where the flows are not public. The reason you can build a chart of the safety-organization money flows is that those flows are public.

这些人在舆论法庭上正在落败。而且败得很惨。他们有自己在 Twitter 上的“氛围战士”，有付费的也有免费的，一呼即应地蜂拥而上，喷出令人作呕的恶言。他们有自己赤裸裸的业余水平文章。目标是政治化和负面极化。他们认为，只要能把特朗普拉到他们一边、把这件事变成党派之争，那就足够了——他们就能继续为所欲为，并且能获得，没错，监管俘获。

These people are losing in the court of public opinion. Badly. They have their Twitter vibe warriors, both paid and unpaid, who swarm obnoxious bile on request. They have their naked amateur-hour pieces. The goal is politicization and negative polarization. They think if they can get Trump on their side and turn this into a partisan thing, that will be good enough that they can keep on doing what they want and they can get, yes, regulatory capture.

我们的政府并非不可能已经退化到如此地步，以至于他们是对的，以至于政府中只有一个人真正重要，而那个人目前正在听黄仁勋、Sacks 和扎克伯格这三人组的，很大程度上是因为他们给他讲了一个他想听的故事。而如果他们能让特朗普喊得足够大声，就能在核心支持者中制造出负面极化。

It is not impossible that our government has degraded so much that they are right, and that only one person in it matters, and that person is currently listening to the triumvirate of Huang, Sacks and Zuckerberg, largely because they tell him a story they want to hear. And if they can get Trump to yell loud enough, that can create the negative polarization among the core supporters.

若果真如此，那这将是一场相当有意思的选举，不妨这么说吧。

If so, it is going to be a rather interesting election, shall we say.