三粒 AI 药丸

三颗药丸 Three Pills

这三颗药丸大致是指认真对待以下三件事：

The three pills are, roughly, taking each of the following three things seriously:

1. AI 药丸：AI 存在，并且能完成它已经能做的事情。

1. AI pilled. AI exists and can do the things it can already do.

2. AGI 药丸：AI 将能够完成更多的事情。

2. AGI pilled. AI will be able to do a lot more of the things.

3. ASI 药丸：AI 将能够在我们的自然寿命内，几乎在所有事情上都比你做得更好。

3. ASI pilled. AI will be able to do approximately all the things better than you, within our natural lifetimes.

我服下了 ASI 药丸。前沿实验室的很大一部分员工也服下了 ASI 药丸。实验室本身也是如此。

I am ASI pilled. A large percentage of employees of the frontier labs are ASI pilled. The labs themselves are ASI pilled.

未服药之人 The Unpill People

我在哪里见到他们？无处不在。大多数人还没有服下第一颗药丸。

Where do I see them? Everywhere. The majority of people have not taken the first pill.

大多数人不知道前沿 AI 能为他们做什么。他们不了解编码智能体。他们只使用过 ChatGPT，用于无害的小事，并且对它的失败记忆停留在多年前。他们嘲笑任何地方的任何失败，称之为“AI 能做的事”。

Most people have no idea what frontier AIs can do for them. They are unaware of coding agents. They have used only ChatGPT, for harmless trifles, and they hold years old memories of its failings. They mock any failure anywhere as 'what AI can do.'

他们因为 AI 指路到一家关门的商店或推荐了错误数量的披萨而认为 AI 毫无价值。他们引用那些在发表之前就已过时、且使用了糟糕提示技术的旧研究。

They dismiss AI as worthless because it pointed them to a closed store or recommended the wrong number of pizzas. They cite old studies that were obsolete before they were published and used terrible prompting techniques.

他们常常还在谈论“随机鹦鹉”或 AI 永远不可能思考，以及一切必定是从训练数据中窃取的。如此等等。

They often still talk about 'stochastic parrots' or how AI can never possibly think and everything must be stolen from the training data. And so on.

其中有些说法是错误的，有些则“连错误都算不上”。没有一种是合理的。

Some versions of this are wrong. Some are Not Even Wrong. None are reasonable.

当你与完全未入坑的人讨论 AI 时，你的目标通常是先给他们服下 AI 之药。向他们展示 AI 已经能做到的事情。

When you discuss AI with people who are fully unpilled, your goal is usually to first give them the AI pill. Show them that AI can do the things it can already do.

AI 药丸 The AI Pill

即使完全吞下第一颗 AI 药丸，也是一件大事。

Even fully taking the first AI pill is a big deal.

现有的 AI 在今天实际上解锁了大量酷炫的功能。在许多方面，它已经比你更聪明、更有能力。

Existing AI unlocks, today, in practice, tons of cool things. It is, in many ways, already smarter and more capable than you.

许多你过去需要手工或以其他方式完成的事情，现在通过向文本框输入快速请求就能做得更好。

So many things that you used to do by hand, or some other way, are now better done by typing a quick request into a text box.

许多以前不值得做的事情，现在值得做了。

So many things that previously were not worth doing are now worth doing.

许多以前不值得问的问题，现在值得问了。

So many questions previously not worth asking are now worth asking.

了解 AI 能为你做什么的边际成本往往接近于零。

The marginal cost of seeing what the AI can do for you is often very close to zero.

AI 也能做各种有害的事情，或者做一些对你有用但使他人受损、或扰乱或破坏规范与体系的事情。人们不喜欢那样。

AI can also do a variety of harmful things, or do things that are useful for you but make others worse off or disrupt or invalidate norms or systems. People don’t like that.

大多数经济学家以及大多数从事政策和政府工作的人，最多只服用了这第一颗药丸，甚至低估了仅第一颗药丸的影响。

Most economists, and most people who work in policy and government, have taken at most this first pill, and underestimate even the impacts of the first pill alone.

他们常说“AI 用在 [X] 上太贵了”，因为他们没有意识到同等智能水平的成本很快会降低几个数量级。或者他们指出 AI 在某些细节上表现不佳，并认为这不会得到修复。他们认为 AI “没有竞争力”，却没有意识到这种情况是暂时的。

Often they say things like ‘AI will be too expensive to use on [X]’ because they don’t realize it will soon be orders of magnitude cheaper for the same level of intelligence. Or they point to particular details where AI does poorly, and presume this will not be fixed. They see AI as ‘uncompetitive’ without realizing the situation is temporary.

当你与只服用了第一颗药丸的人讨论 AI 时，你通常有三个基本选择。

When you discuss AI with someone who has taken only the first pill, you typically have three basic options.

1. 你可以尝试给他们吃下“完全 AI 药丸”，解释 AI 已经能做的事情以及这意味着什么，即使事情就此止步。

1. You can try to 'fully AI pill' them and explain the things AI can already do and the implications of what that means, even if things stop here.

2. 你可以尝试解释，我们会更好地利用现有的 AI，而且即使事情就此止步，我们还有很多“解绑”工作要做。

2. You can try to explain that we will get better at using what AI we have, and that there is a lot of 'unhobbling' left for us to do, even if things stop here.

3. 你可以解释事情不会就此止步，你需要思考未来的 AI 将能做什么。至少让他们吃下 AGI（通用人工智能）药丸。

3. You can explain that things will not stop here, and you need to be thinking about what future AIs will be able to do. Get them to take at least the AGI pill.

即使 AI 永远只能做它目前能做的事情，那也将是互联网级别的重大事件，并将彻底改变世界，而且大多是向好的方向。

Even if AI could permanently only do the things it can currently do, that would be Internet big, and radically change the world, mostly for the better.

AI 的能力不会永远止步于此。不吃第二颗药丸是错误的。

AI capabilities will not permanently stop here. It is wrong to not take the second pill.

卡在第一颗药丸 Stuck At The First Pill

我们关于 AI 的讨论在很大程度上仍停留在已解决的问题上，因为许多人甚至无法吞下第一颗药丸。

Our debates about AI remain largely stuck on settled questions, because so many people cannot even take the first pill.

为了进行良好的讨论，我们至少需要吞下第二颗药丸。

To have good discussions, we need to at least take the second pill.

然而，确实有许多人，甚至包括许多从事 AI 工作的人，真的认为当前的 AI“已经足够好”，无法想象更好的 AI 能做什么。比如，有人在推特上对 Sam Altman 说：“Sol 能完成我想做的所有事情，这就是我所需要的一切”，而 Altman 转发并说他们错了。显然他们确实错了。

Whereas, yes, many people, even many who work with AI, really do say current AI is ‘good enough’ and can’t imagine what a better one can do. As in, someone tweeting at Sam Altman saying ‘Sol does everything I want it to do, this is all I ever need’ and Altman retweeting saying they were wrong. Which they obviously are.

AGI 药丸 The AGI Pill

AGI 药丸比 AI 药丸要重要得多。

The AGI pill is a much bigger deal than the AI pill.

如果你服下 AGI 药丸，你就会明白 AI 的能力正在迅速提升。

If you take the AGI pill, you understand that AI is advancing its capabilities rapidly.

即使你认为这样的 AGI 将完全处于人类控制之下，仍然是“纯粹的工具”，并且你预期大多数人的日常生活体验不会发生如此剧烈的变化，你也会明白它们的能力将“改变一切”。

Even if you think that such AGIs will remain fully under human control, and remain 'mere tools,' and you expect the lived experience of most people's everyday lives to not change so radically, you understand that their capabilities will 'change everything.'

你会看到，我们将面临巨大的转型和不确定性的时期，这些时期有可能变得极其糟糕，而那些在 AI 上取得成功的人将把未成功者远远甩在身后。

You see that we will face times of great transition and uncertainty, that have the potential to go extremely badly, and that those who succeed at AI will leave those who do not behind in the dust.

未来的世界将与我们现在截然不同。AI 将能够完成大部分数字工作，在大多数情况下，还有不可避免的机器人和自动驾驶汽车等等。许多现有的工作岗位将消失，无论它们是否被新的、可能更好的岗位所取代。经济增长和生产率将加速。

The world in the future will be very different from our own. AIs will be able to do most digital work, most of the time, along with the inevitable robots and self-driving cars and so on. Lots of current jobs will go away, whether or not they are replaced by new and potentially better ones. Economic growth and productivity will accelerate.

你会看到，如果我们过早地允许滥用此类先进 AI 系统，尤其是在网络和生物风险等领域，将会带来哪些危险。

You see some of the dangers of what would happen if we empowered misuse of such advanced AI systems before we were ready, especially in places like cyber and bio risk.

你会看到权力集中或不平等的可能性，以及某些形式的失控性渐进式剥夺权力。

You see the potential for centralization of power, or inequality, and also for some forms of runaway gradual disempowerment.

你会看到大规模失业的可能性，无论是过渡性的还是永久性的。

You see the potential for mass unemployment, either transitional or permanent.

你明白我们的法律和监管体系尚未准备好，既无法防范和减轻风险与危害，也无法把握机遇并消除阻碍技术普及和日常应用的瓶颈。

You understand that our legal and regulatory regimes are not ready, either to protect against and mitigate the risks and harms, or to allow for the opportunities and remove the bottlenecks to diffusion and mundane utility.

做好准备的必要性 The Need To Be Prepared

那些期望 AI 迅速变得足够先进、从而对物理世界产生巨大影响的人，通常看到巨大的危险。他们注意到，结果可能是所有人都很快死亡。通常他们认为这实际上是坏事。

Those who expect AI to quickly become sufficiently advanced to greatly impact the physical world usually see great danger. They notice that as a result everyone may soon die. Usually they think this is bad, actually.

因此，这些人呼吁采取协调行动，以减轻此类影响的下行风险，使我们免于死亡，并理想情况下帮助获取上行收益。

Thus such folks call to take coordinated action to mitigate the downside risks of such impacts, keep us all from dying, and ideally also to help capture the upside benefits.

那些期望 AI 变得重要地更先进，但对物理世界的影响较慢且较小，并认为更多智能的实际价值将封顶的人。

Those who expect AI to become importantly more advanced, but with a slower and smaller impact on the physical world, and who think the practical value of more intelligence will cap out.

对于“足够先进”的不同值，如 AGI 与 ASI，你会看到不同程度的危险。

For different values of ‘sufficiently advanced,’ as in AGI versus ASI, you would see different degrees of danger.

截至 2026 年 8 月，AGI 药丸对于 Overton 窗口内或正在认真考虑的大多数事情仍然足够。我们几乎完全在考虑那些结果确定、成本低、收益高的干预措施。

The AGI pill is still sufficient for most things in the Overton window or under serious consideration as of August 2026. We are almost entirely considering overdetermined, low cost, high benefit interventions.

没有充分的理由不去大幅增加在以下方面的投入：对齐、基础设施与监管、国家能力、透明度、责任追究、信息披露、安全测试（包括内部模型测试）、红队测试、审计、出口管制执行，以及为外交奠定基础。

There is no good case for not doing radically more investment in alignment, infrastructure and oversight, state capacity, transparency, liability, disclosures, safety testing including of internal models, red teaming, auditing, enforcement of export controls and laying the groundwork for diplomacy.

这包括为必要时“控制前沿”奠定基础。

This includes laying the groundwork to Pace the Frontier should that prove necessary.

如果你完全相信超级智能（ASI）即将到来，并且对当前对齐状态以及超级智能若很快到来可能如何发展持现实态度，那么你会希望更进一步。你会愿意去做那些有实际负面影响、需要真正权衡取舍的事情。

If you are fully ASI pilled, and realistic about the current state of alignment and how superintelligence likely plays out if it arrives soon, then you will want to go further. You will want to do things that have real downsides, and require real tradeoffs.

有些人希望全面暂停前沿 AI 开发。如果你完全相信超级智能，并且认同他们关于超级智能到来速度及其能力的看法，你很可能也会同意他们的观点。

Some such people want to do a full international pause of frontier AI development. If you took the full ASI pill and believed what they do about superintelligence, in terms of how fast it might arrive and what it can do, you might well agree with them.

ASI 药丸 The ASI Pill

ASI 药丸是指这样一种认识：AI 正以这样的速度发展，以至于它将能够比你更好地完成几乎所有的事情。

The ASI pill is the understanding that AI is on pace to be able to do approximately all of the things better than you.

我说的是“几乎”。正如我详细阐述的那样，这并不意味着字面意义上的所有事情。有些事情本质上需要人类身份或极大受益于人类身份。而且可能存在一些奇怪的边缘情况，AI 无法胜任。

I said 'approximately.' As I go over in detail, that does not mean literally all of the things. There are some things that inherently require or greatly benefit from being a human. And there may be weird corner cases where the AI won’t be good enough.

这并不意味着全能或全知，尽管人们应该预期，在未加辅助的人类看来，它很像全能或全知。

It does not mean omnipotence or omniscience, although one should expect it to look a lot like that to an unaided human.

它确实意味着 AI 会取代你的工作，然后取代你转而从事的新工作，除非你转向“需要真正人类”的领域。在几乎所有其他任务上，你都将失去竞争力。

It does mean the AI takes your job, and then takes the new job that you switch into, unless you pivot to 'requires literal human.' You will be uncompetitive at essentially any other task.

它确实意味着 AI 将利用这种能力，以惊人的速度弄清楚几乎所有的事情，直到触及物理极限。

It does mean that it will use this capability to figure out approximately all of the things, remarkably quickly, until you hit the physical limits.

这确实意味着，那些更依赖此类 AI 的人，将在包括资源在内的所有意义上，可靠地胜过那些较少依赖此类 AI 的人。

It does mean that those who rely more on such AIs will reliably outcompete, in all senses including for resources, those that rely on such AIs less.

这确实意味着，在“公平竞争”或足够开放的竞争中，AI 会获胜。

It does mean that, in a 'fair fight' or sufficiently open competition, the AI wins.

这也意味着 AI 常常会想出并做出你未曾想象或预料到的事情。

It also means the AIs often figuring out and doing things you did not imagine or anticipate.

这意味着要认识到，智能远不会止步于人类水平，它在因果空间中规划通往理想原子排列路径的能力同样如此。

It means realizing that intelligence does not stop anywhere near the human level, nor does its ability to chart paths through causal space towards preferred arrangements of atoms.

这也意味着不要假装你那微不足道的武器、纸上的文字、数据库中的条目，或你的监管俘获和寻租行为，能够匹敌其卓越的智能。

It also means not pretending that its superior intellect can be matched by your puny weapons, or your pieces of ink on paper, or your entries in a database, or your regulatory capture and rent seeking.

然后，你的生活不会有太大改变 And Then Nothing Much Changes For You

尽管如此，与 ASI（超级智能）药丸相对，AGI（通用人工智能）药丸的标志是相信日常生活将继续与现在相似，就像我们认为 1926 年的日常生活与 2026 年的生活没有太大不同一样。

Despite all that, the sign of the AGI pill, as opposed to the ASI pill, is the belief that day-to-day life will continue to look similar to how it looks now, in the sense that we see day-to-day life in 1926 as not that different from life in 2026.

有时这显然是虚伪的，因为它来自一个对 ASI 足够了解的人。我认为萨姆·奥尔特曼所说的各种形式的“奇点之后生活不会有太大改变”基本上是没有根据的明显废话，是为了安抚听众，而不是一个连贯的立场。有时这些人最想安抚的是他们自己。

Sometimes this is clearly disingenuous, as it is coming from someone ASI-pilled enough to know better. I read Sam Altman saying various forms of 'life will not much change after the singularity' as basically unjustified Obvious Nonsense to reassure his audiences, rather than a coherent position. Sometimes the person such folks want to reassure most is themselves.

其他人有时确实对未来以及为什么不会改变太多有一些更具体的想法，但连贯程度不一。

Others sometimes do have something more concrete in mind as their vision of the future, and why it will not change so much, with varying degrees of coherence.

只服用 AGI 药丸的人相信存在阻碍变革的瓶颈。

Those with only the AGI pill believe in bottlenecks that hold back change.

他们通常认为，我们指数级增长 AI 能力和算力的能力将遇到各种物理限制。芯片数量有限。行动需要时间。太遥远的事情常常被轻蔑地斥为“魔法”。

They often believe that our ability to exponentially grow AI's capacity and capabilities will hit various physical limits. There can only be so many chips. Actions take time. Things too far out there are often pejoratively dismissed as 'magic.'

他们常常认为，在物理发现方面，已经没有太多可探索的了，就像经典的‘关闭专利局’那种想法，甚至在理论上也是如此。你的牛排只能嫩到一定程度，龙虾只能黄油到一定程度，寿命只能长到一定程度，地位只能高到一定程度，所以这又有什么关系呢？我强烈不同意关于寿命和健康的观点，并期望我们在寻找价值方面还有很多路要走，尽管他们关于物理人脑瞬间最大享乐体验的观点可能有一定道理。

They often believe there is not that much left to physically discover, in the classic 'close the patent office' kind of way, even in theory. Your steak can only be so tender, your lobster so buttery, your lifespan so long, and your status so high, so why does it matter. I strongly disagree on lifespan and health, and expect we have a long way to go in so many other ways in terms of finding value, although they may have a point about moment-to-moment maximal hedonic experiences of a physical human brain.

他们常常认为，智能的上限受到重要限制。即没有任何心智，无论多么先进，能够如此有说服力，或如此具有经济价值，或能够如此创造物理世界的创新，或运行足够准确的模拟，或做出足够强的预测，甚至能够克服官僚主义和监管俘获等问题。

They often believe that the upside of intelligence is importantly limited. That no mind, however advanced, could be all that persuasive, or that economically valuable, or that capable of creating innovations in the physical world, or of running sufficiently accurate simulations, or making sufficiently strong predictions, or even able to do things like overcome red tape and regulatory capture.

智能否认主义 Intelligence Denialism

我有时称之为“智能否认主义”：认为变得更聪明并不那么重要，无论一个人变得多聪明。认为存在一种叫做“智能”的东西，你要么拥有它，要么没有，而且心智存在上限。

I sometimes call this Intelligence Denialism: the idea that being smarter is not all that, no matter how smart one gets. That there is this thing, intelligence, that you either have or don’t have, and that minds cap out.

这种观点常常进一步否认更聪明的人类能够做到他们显然做到的那些事情。有时，它认为智能的上限就是“聪明的人类”，而心智所能做的只是模仿那个聪明的人类。也许你可以做得更快、更便宜、大规模，并且拥有更好的记忆等等。

Often this extends to denying that more intelligent humans can do and accomplish the things they clearly do and accomplish. Other times, it is the idea that intelligence tops out at ‘smart human,’ and all a mind can do is imitate that smart human. Maybe you can do it faster and cheaper, and at scale, with better memory and so on.

但仅此而已。这些人未能理解，如果你将全人类的心智能力与知识获取途径结合起来，以大规模、并行、更快更便宜的方式运行，仅凭这一点就能在各个方面碾压任何地方的任何人。而如果这缺乏物理能力或访问途径，那也是轻而易举就能获得的。

But that’s it. And such folks fail to understand that if you took the union of all human mental capabilities, and all access to knowledge, at scale, in parallel, much faster and cheaper, that this alone would run circles around anyone and everyone, everywhere. And that if this lacked physical capabilities or access, this would be trivial to get.

这通常是那些“AGI 信徒”不服用“ASI 药丸”的核心原因。他们无法理解超级智能是真实存在的。

This is, usually, the central good reason people who are AGI pilled do not take the ASI pill. They are unable to understand that superintelligence is a thing.

这种困惑也常常是模型将商品化这一直觉背后的原因。存在一个理想的东西——“智能”，你通过一条渐近线接近它。更多是不可能的，或者更多对你没有帮助，取决于你如何表述。

This confusion is also often behind the instinct that models will commoditize. There is an ideal thing, ‘intelligence,’ and you approach it via an asymptote. More is impossible, or more won’t help you, depending on how you frame it.

超级智能与全知全能 Superintelligence Versus Omniscience and Omnipotence

一个常见的错误是认为任何超级智能都会是全知甚至全能的，因此超级智能是不可能的。

A common error is to think that anything superintelligent would be omniscient or even omnipotent, and therefore superintelligence is impossible.

然而，再次强调，在人类之上有大量的“空间”，而无需达到全知全能。

Whereas, again, there is a lot of 'space above humans' without becoming omni.

还有许多类似的混淆，即“对[X]存在某个上限”与“我们处于或接近[X]的上限”被混为一谈。

There are many other similar confusions, where 'there is some upper bound to [X]' is confused with 'we are at or near the upper bound to [X].'

在这里，Adi 为我们提供了一种解释正在发生的事情的途径，从我的角度来看，这相当于核心“城堡与壕沟”谬误的镜像。

Here Adi gives us a way into explaining what’s happening, by doing what from my perspective is a mirror image of the central motte-and-bailey.

仍会有一些事情需要耗费非平凡的现实世界时间才能实现。特定的行动、路径和方法会有其局限性。这些局限性很重要。完全忽略它们可能是一个大错误。

There will still be things that require non-trivial real world time to achieve. There will be particular actions and paths and methods that have their limitations. These limitations matter. Fully abstracting them away can be a large mistake.

我经常看到这样的说法：“一个至善的头脑仍然无法做到[在我看来它显然能做到的事情]”，甚至“[它已经能做到的事情]”。我认为这种连词形式在很大程度上解释了人们为何以及如何为自己辩护。当 AI 获得能力，更多事情变得可行时，他们移动了球门柱，但没有改变游戏规则。

I constantly see versions of 'the mind that is maximally good still could not do [thing it seems to me it could obviously do]' or even '[thing it can already do]' and I think this form of conjunction is a lot of why, and how people justify themselves. When AI gains capabilities, and more things fall, they move the goalposts but don’t change the game.

Adi 上面的评论是在讨论“A 计划”的背景下提出的，在 Adi 的术语中，这些 AI 显然不是“大写的超级智能”。这些 AI 有一些实际无法做到的事情，或者只能做到一定程度。

Adi’s comment above was in the context of discussing Plan A, where the AIs are very obviously not 'Big-S Superintelligence' in Adi’s lexicon. There are practical things that these AIs cannot do, or can only do up to a point.

这个错误并没有被犯下。它有时会被犯下，但这种情况很少见。

The mistake was not being made. It sometimes gets made, but this is rare.

说服（一个实例） Persuasion Persuasion (A Worked Example)

最多只能声称某种特定能力，尤其是非常有效的说服（通常称为“超级说服”），是不可能实现的。事实上，一个常见的论点是，即使是一个理想化的 AI 也无法超越最优秀的人类说服者，甚至无法超越一个典型的优秀人类说服者（尽管后者远不如历史上最顶尖的人类说服者），尽管 AI 拥有许多人类无法比拟的优势，例如更快的思考速度和信息获取能力，包括对目标每一个细微反应（包括肢体语言）的捕捉。

At most one can claim that some particular capability, typically very effective persuasion (often 'super-persuasion'), would not be possible. Indeed, a common argument states that even an idealized AI would be unable to be superior to the best human persuader, or even a typical good human persuader (who is much worse than the historical best human persuaders), despite having access to many advantages over humans, such as much faster thinking speed and access to information, including every little reaction of its target including body language.

这通常是通过引用“上下文对说服很重要”来论证的。许多人说得好像这种上下文会压倒一切，尽管对人类而言并非如此。他们还认为，高能力的 AI 无法构建有利的上下文。

Often this is done by citing that context matters for persuasion. Many talk as if this context would trump everything, despite this not being true for humans. They also believe that highly capable AIs could not engineer favorable contexts.

这通常是一种“我干脆选择不被说服”或“如果你不能冷启动、仅用文本一次性说服所有人，那就不算数”之类的说法。

Often it's a form of 'I would simply choose not to be persuaded' or 'if you can't do it cold and one shot everyone with only text it does not count' or something like that.

如果它需要人类在场或社会情境来进行说服，它可以通过多种方式轻松获得这些条件。

If it needs a human presence to do the persuasion, or a social context, it can acquire one easily enough in various ways.

我完全无法理解为什么有人会认为先进的 AI 将保持相对缺乏说服力，除非他们认为 AI 的能力不会比现在提升太多，而且前提是我们完全限制 AI 只能使用已知的标准说服技巧，并排除任何“魔法”般的创新。

I flat out cannot understand why someone would think advanced AIs will remain relatively unpersuasive, other than to think that AI will not get much more capable than it already is, and that's if we fully restrict the AI to using known standard persuasion techniques and rule out any wizardry.

然而，我并非超人般的说服者，因此许多人觉得我的论点缺乏说服力。

Yet I am not a superhuman persuader, so many find my arguments unconvincing.

不，说真的，就像 Adi 引用那段高亮文字，仿佛这并非显而易见的事实。我的天，在 AI 能力持续飞速发展的情景下，你怎么能认为这不会发生？如果说有什么的话，2035 年似乎都慢得离谱：

No, seriously, as in Adi is citing the highlighted passage as if this was not very obviously true, my lord, how can you think this is not going to happen in a scenario where AI capabilities continue to develop apace, if anything 2035 seems crazy slow:

[](https://substackcdn.com/image/fetch/$s_!Ot9i!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F204d1c86-7105-4aa5-b81c-2bec58ab0b1e_1200x607.jpeg)

[](https://substackcdn.com/image/fetch/$s_!Ot9i!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F204d1c86-7105-4aa5-b81c-2bec58ab0b1e_1200x607.jpeg)

这算不上什么高门槛。也不要求 AI 仅凭纯文本界面就能做到人类面对面才能做到的事。它只是说，在获得类似工具的情况下，AI 将能比任何人类做得更好。考虑到情景中的其他部分，你怎么能不这么想呢？即使它只是集各种人类之所长，再加上能更快地处理更多信息的能力。

This just isn’t much of a threshold. Nor does it require that the AI do this from pure text box versus what a human can do in person. It just says that, given access to similar tools, the AI will be able to do considerably better than any human. How could you not think this, given the rest of the scenario, even if all it does is have all the advantages and skills of the various different humans, plus the ability to process a lot more information a lot faster?

而且，最坏的情况是，AI 戴个耳机，告诉人类该做什么，天哪。

And like, worst case, the AI gets an earpiece and tells a human what to do, Jesus.

我的回答是，对于一个足够有说服力的 AI，本质上：是的，但不可能不附带接管至少加州政府。工具性趋同。

My answer, for a sufficiently persuasive AI, is essentially: Yes, but not without incidentally taking over at least the California government. Instrumental convergence.

并不是说你会因此选择专注于高铁，正如 Dean 所回应的。这不是一个“会”的问题，而是“能”的问题。关键在于理论上“能”。

Not that you would then choose to focus on High Speed Rail, as Dean replies. This is not a case of would, so much as could. The point is the could, in theory.

一个几乎完美诠释“你永远无法做到[你本已能做到的事]”的例子：

An example that is almost too perfect an example of 'you will never be able to do [thing that you can already do]':

我们在 2025 年已经成功将铅变成了黄金，只是用当前方法成本过高。人们确实以这样的方式建模世界：任何我们没有明确路径已经做到的事情，就永远不可能做到。每个人都想关闭专利局。

We already have successfully turned lead into gold in 2025, it's just cost prohibitive with current methods. People really do model the world as if anything we don't specifically have a clear path to already doing cannot ever be done. Everyone wants to close the patent office.

AI 可能做到但并非“被灌输”所必需的事情 Things AI Could Probably Do But Are Not Required For Being Pilled

这样的 AI 能否向生物实验室发出订单，最终得到可编程的自我复制金刚石纳米探针？它能否完全基于模拟，无需任何物理实验就做到这一点？它能否通过一系列相对快速的实验做到？或者，它能否做其他类似强大的事情，绕过一切障碍，让它为所欲为？时间尺度又是怎样的？

Could such an AI send off an order to a bio lab and end up with programmed self-replicating diamond nanoprobes? Could it do so without any physical experiments, based purely on simulations? Could it do so with a relatively fast series of experiments? Could it instead do other similarly powerful things that do an end run around everything and let it do whatever it wants? On what time frame?

没有人知道物理极限在哪里，无论是完全 RSI（递归自我改进）的终点，还是当前周期在实践中可能达到的上限。我完全预期会发现各种我们未曾预料到的、在许多情况下甚至未曾想象到的心理和物理能力以及新的可供性，而且按照定义，很难知道具体是哪些。

No one knows where the physical limits lie, either for the end of full RSI (recursive self-improvement) or for where the current cycles might top off in practice. I fully expect to discover various mental and physical capabilities, and new affordances, that we did not expect, and in many cases did not imagine, and by definition it is hard to know which.

我基本上不再谈论这些可能性，尽管在我看来它们相当可能，因为：

I mostly stopped talking about those possibilities, despite them seeming rather likely to me, because:

1. 当你提到这些事情时，人们会攻击这种说法，并以此为由否定你的整个论点、所有来自高级 AI 的风险，以及你这个人。2. 这种策略之所以有效，是因为这类说法更难证明，确定性更低。

1. When you mention such things, people attack that claim, and use it as a reason to dismiss your entire argument, all risks from advanced AI, and you as a person. 2. This strategy works, because such claims are much harder to justify, less certain.

Could such an AI send off an order to a bio lab and end up with programmed self-replicating diamond nanoprobes? Could it do so without any physical experiments, based purely on simulations? Could it do so with a relatively fast series of experiments? Could it instead do other similarly powerful things that do an end run around everything and let it do whatever it wants? On what time frame?

3. 你不需要这些主张也能得到同样的结果。足够先进的 AI 只需具备我们有信心这类 AI 会拥有的能力就能‘达到目标’，因为人类在足够的算力、参数和数据下已经拥有这些能力。

3. You don’t need such claims to get the same results. Sufficiently advanced AI can ‘get there’ with only capabilities we can be confident such AIs will have, because humans already have them modulo sufficient compute, parameters and data.

例如：也许詹姆斯·邦德并不总能赢得美人芳心，也许他能，也许 Q 的最新 gadget 是你能造出来的东西，也许不是，但我敢肯定邦德有枪而且非常擅长用枪，而在这件事上他确实只需要枪，但很多人不明白这一点。所以我选择假设他只有枪来进行论证。

E.g.: Maybe James Bond can’t always get the girl, maybe he can, and maybe Q’s latest gadget is a thing you can build and maybe it isn’t, but I am damn sure Bond has a gun and is very good with it, and the gun really is all he needs on this one, but a lot of people don’t understand this. So I choose to argue as if all he has is the gun.

另一种方法是这样的，它对某些人有效，但我预计它对 Timothy Lee 完全没有说服力：

Another approach is this, which works for some but I do not expect it to be at all persuasive to Timothy Lee:

生活来得越来越快 Life Comes At You Increasingly Fast

另一个让人们止步于 AGI 药丸的原因，是不理解或直接拒绝接受递归自我改进、奇点或事物可能急剧加速的想法，认为这些太过科幻、怪异、荒谬或诸如此类。

Another reason people stop at the AGI pill is not understanding, or rejecting out of hand as too sci-fi or weird or absurd or what not, the idea of recursive self-improvement, or a singularity, or that things might be radically accelerating.

他们可以接受事情在加速，但无法接受加速本身也在加速，如此等等，而这确实已经发生且难以忽视。

They can accept that things are speeding up, but not that the speeding up is itself speeding up, and so on, which indeed is already happening and hard to miss.

从某种意义上说，这种情况已经持续了很长时间。数学预测了奇点。这有时被用来表示“这并非新进展”，但指数增长正是如此运作的。没有核心的新事物发生，然而砰的一声。

This has in some sense been happening for a very long time. The math predicts a singularity. This is then sometimes used to say 'well this is not a new development' but that is how exponentials work. Nothing centrally new happens, and yet kaboom.

提醒一下（四舍五入，别找我）：

As a reminder (with rounding, don't @ me):

4. 300 年前：工业革命。

4. 300 years ago: Industrial Revolution.

你可能会被这样的说法误导：‘哦，存在一个阶跃变化，你突然进入奇点并实现递归自我改进，在那之前则不然’，或者‘这个新事物并无不同。’

You can be misled by 'oh there is this step change where you are suddenly in a singularity and having recursive self-improvement, and until then no' or 'this new thing is not different.'

不信仰 AGI 是否合理？ Is It Reasonable To Not Be AGI Pilled?

如果你不信仰 AGI，你对 AI 的反应就不会明智或审慎。遗憾的是，我们的许多政策和对话正由那些不信仰 AGI 的人所驱动。

If you are not AGI-pilled, your reactions to AI will not be wise or prudent. Alas, much of our policy and conversation is being driven by people who are not AGI-pilled.

当我们看到那些依赖于不信仰 AGI 的立场时，我们应该明确指出，并据此决定是否与之互动。

When we see positions that rely on not being AGI-pilled, we should say so, and engage or not engage with them accordingly.

只信奉 AGI 是否合理？ Is It Reasonable To Only Be AGI Pilled?

是的，如果你认真思考其含义，并知道自己的关键点所在。

Yes, if you seriously grapple with its implications, and know what your cruxes are.

我认为这是错误的。但这是一个连贯的立场，至少是错误的，认为我们当前的技术和资源不会完全‘达到目标’，AI 能力很可能在我们真正称之为超级智能之前达到顶峰。

I think it is wrong. But it is a coherent position, that is at least wrong, to think that our current techniques and resources will not fully ‘get there’ and AI capabilities are likely to top out before things we would properly call superintelligence.

或者至少，我认为把这种立场视为不合理不会有什么成效。

Or at least, I don’t think treating this as unreasonable would be productive.

如果我们‘撞墙’，即至少在一段持续时间内出现明显的收益递减，特别是如果我觉得使用去年的模型并不落后，事情真的商品化，我会转而增加对这一立场的重视。

I would switch to putting increasing weight on this position if we ‘hit a wall’ of at least clearly diminishing returns for a sustained period of time, and especially if I don’t feel so behind using last year’s model and things really do commoditize.

我们仍可能看到 AI 收入和算力需求的急剧增长。对 AI 的投资可能在供应链上下游都获得丰厚回报。这一切都与只信奉 AGI 的立场完全兼容。

We could still see dramatic growth in AI revenue and demand for compute. Investments in AI could pay off handsomely up and down the supply chain. That is all fully compatible with the AGI-only position.

如果你确实只相信 AGI 的立场，我有两个请求。

If you do believe the AGI-only position, I have two requests.

第一，认真思考你认为 AI 将能够做到的事情的影响，不要退缩，也不要试图假设一切都会神奇地顺利解决，并且人们的生活经历不会发生太大变化。这意味着现在就要支持我们为减轻所有这些风险所需的行动，并帮助事情朝着好的方向发展。

First, seriously tackle the implications of what you do think AI is going to be able to do, without flinching, and without trying to assume everything magically works out and somehow people’s life experiences do not much change. That means then supporting actions now that we need to mitigate the risks of all that, and help make things go well.

第二，写下阻止你接受 ASI 药丸的关键因素。特别是，写下（在这里的评论中写会很好）AI 永远无法做到的最不令人惊讶或最不令人印象深刻的事情，或者会改变你对未来走向看法的东西，以及哪些近期观察会让你要么确信自己是对的，要么意识到自己错了。

Second, write down what are your cruxes that are stopping you from taking the ASI pill. In particular, write down (in the comments here would be great) what is the least surprising or impressive thing an AI will never be able to do, or that would change your mind about where this is going, and what other near term observations would cause you to either be confident you are right, or realize you are wrong.

在那之前，有两个有效的选择：你可以停在 AGI 药丸，或者完全接受 ASI。

Until then, there are two valid choices: You can stop at the AGI pill, or go full ASI.

还有两个常见但无效的选择：认为 AI 到此为止，或者否认现实。

There are also two common but invalid choices: Think AI stops here, or deny reality.