三粒 AI 药丸

The Three AI Pills

兹维·莫绍维茨 Zvi Mowshowitz · Don't Worry About the Vase · 2026-08-05 · Don't Worry About the Vase ↗

打开互动全文版(逐段中英对照 + 针对本篇提问)→

摘要 · Abstract

本文介绍了一个由三粒“药丸”组成的框架,用以描述对 AI 的认知水平:AI 药丸(认识到当前 AI 能力)、AGI 药丸(预期快速进步至人类水平 AI)、以及 ASI 药丸(预期超级智能在我们有生之年出现)。作者自认是 ASI 派,并认为大多数人,包括政策制定者,甚至还没有服用第一粒药丸,导致讨论方向错误。核心论点是,AI 当前和未来的能力将彻底改变社会,需要紧急投资于对齐、安全和治理。文章区分了 AGI 和 ASI 视角,批评了低估超级智能潜力的“智能否认主义”。结论是,至少服用 AGI 药丸对于有意义的讨论至关重要,而 ASI 药丸则意味着更激进的行动,如可能暂停开发,以减轻生存风险。

The article introduces a framework of three 'pills' to describe levels of AI awareness: AI pilled (recognizing current AI capabilities), AGI pilled (anticipating rapid advancement to human-level AI), and ASI pilled (expecting superintelligence within our lifetimes). The author identifies as ASI pilled and argues that most people, including policymakers, have not even taken the first pill, leading to misguided debates. The core argument is that AI's current and future capabilities will radically transform society, requiring urgent investment in alignment, safety, and governance. The article distinguishes between AGI and ASI perspectives, critiquing 'intelligence denialism' that underestimates superintelligence's potential. It concludes that taking at least the AGI pill is essential for meaningful discussion, while the ASI pill implies more drastic actions, such as potential pauses on development, to mitigate existential risks.

核心贡献 · Key contributions

局限 · Limitations

章节 · Sections(共 15)

全文 · Full text(逐段中英对照)

三粒药丸 Three Pills

这三粒药丸大致是指认真对待以下三件事:

The three pills are, roughly, taking each of the following three things seriously:

1. **AI 药丸**。AI 已经存在,并且能够完成它目前已经能做的事情。

1. AI pilled. AI exists and can do the things it can already do.

2. **AGI 药丸**。AI 将能够完成更多的事情。

2. AGI pilled. AI will be able to do a lot more of the things.

3. **ASI 药丸**。在我们自然的寿命之内,AI 将能够把几乎所有事情都做得比你好。

3. ASI pilled. AI will be able to do approximately all the things better than you, within our natural lifetimes.

我是 ASI 药丸派。前沿模型实验室中很大一部分员工都是 ASI 药丸派。实验室本身也是 ASI 药丸派。

I am ASI pilled. A large percentage of employees of the frontier labs are ASI pilled. The labs themselves are ASI pilled.

未服下药丸的人们 The Unpill People

我在哪里看到他们?到处都是。大多数人还没有服下第一颗药丸。

Where do I see them? Everywhere. The majority of people have not taken the first pill.

大多数人对前沿 AI 能为他们做什么一无所知。他们不知道编码智能体。他们只用过 ChatGPT,处理些无害的琐事,并且对其失败之处保留着多年前的记忆。他们嘲笑任何地方的任何失败,称之为‘AI 能做的事’。

Most people have no idea what frontier AIs can do for them. They are unaware of coding agents. They have used only ChatGPT, for harmless trifles, and they hold years-old memories of its failings. They mock any failure anywhere as 'what AI can do.'

他们因为 AI 指给他们一家关门的商店或推荐了错误数量的披萨,就认为 AI 毫无价值。他们引用那些在发表之前就已过时的旧研究,并且使用了糟糕的提示技术。

They dismiss AI as worthless because it pointed them to a closed store or recommended the wrong number of pizzas. They cite old studies that were obsolete before they were published and used terrible prompting techniques.

他们常常还在谈论‘随机鹦鹉’,或者 AI 永远不可能思考,一切都必须是从训练数据中窃取的。等等。

They often still talk about 'stochastic parrots' or how AI can never possibly think and everything must be stolen from the training data. And so on.

其中一些说法是错误的。有些甚至错得离谱。没有一个是合理的。

Some versions of this are wrong. Some are Not Even Wrong. None are reasonable.

当你与完全未觉醒(unpilled)的人讨论 AI 时,你的目标通常是先让他们服下“AI 药丸”。向他们展示 AI 已经能够做到的事情。

When you discuss AI with people who are fully unpilled, your goal is usually to first give them the AI pill. Show them that AI can do the things it can already do.

AI 药丸 The AI Pill

即便只是完全服下第一颗 AI 药丸,也已是件大事。

Even fully taking the first AI pill is a big deal.

现有的 AI 在今天、在实践中,已经解锁了大量很酷的东西。在许多方面,它已经比你更聪明、更有能力。

Existing AI unlocks, today, in practice, tons of cool things. It is, in many ways, already smarter and more capable than you.

许多你过去靠手工或以其他方式完成的事情,如今只需在文本框里快速输入一条请求,就能做得更好。

So many things that you used to do by hand, or some other way, are now better done by typing a quick request into a text box.

许多过去不值得做的事情,如今变得值得做了。

So many things that previously were not worth doing are now worth doing.

许多过去不值得问的问题,如今变得值得问了。

So many questions previously not worth asking are now worth asking.

看清 AI 能为你做什么的边际成本往往非常接近于零。

The marginal cost of seeing what the AI can do for you is often very close to zero.

AI 也可能做出各种有害的事情,或者做出对你有利、却让他人境况变差,或扰乱、破坏规范或系统的事情。人们不喜欢这样。

AI can also do a variety of harmful things, or do things that are useful for you but make others worse off or disrupt or invalidate norms or systems. People don't like that.

大多数经济学家,以及大多数在政策和政府部门工作的人,最多只服下了这第一颗药丸,而且他们甚至低估了仅第一颗药丸本身的影响。

Most economists, and most people who work in policy and government, have taken at most this first pill, and underestimate even the impacts of the first pill alone.

他们常说诸如“在 [X] 上使用 AI 会太贵”之类的话,因为他们没有意识到,在同等智能水平下,成本很快就会便宜几个数量级。或者他们指出 AI 表现不佳的某些具体细节,并假定这些问题不会被解决。他们认为 AI“缺乏竞争力”,却没有意识到这种局面是暂时的。

Often they say things like 'AI will be too expensive to use on [X]' because they don't realize it will soon be orders of magnitude cheaper for the same level of intelligence. Or they point to particular details where AI does poorly, and presume this will not be fixed. They see AI as 'uncompetitive' without realizing the situation is temporary.

当你与一个只服下第一颗药丸的人讨论 AI 时,你通常有三种基本选择。

When you discuss AI with someone who has taken only the first pill, you typically have three basic options.

1. 你可以尝试让他们“完全接受 AI 药丸”,并解释 AI 已经能做到的事情以及这意味着什么,即使事情到此为止。

1. You can try to 'fully AI pill' them and explain the things AI can already do and the implications of what that means, even if things stop here.

2. 你可以尝试解释,我们将更善于利用现有的 AI,而且即使事情到此为止,我们还有很多“解除束缚”的工作要做。

2. You can try to explain that we will get better at using what AI we have, and that there is a lot of 'unhobbling' left for us to do, even if things stop here.

3. 你可以解释事情不会到此为止,你需要思考未来的 AI 能做什么。至少让他们接受 AGI 药丸。

3. You can explain that things will not stop here, and you need to be thinking about what future AIs will be able to do. Get them to take at least the AGI pill.

即使 AI 永久只能做它目前能做的事情,那也将是互联网级别的重大事件,并从根本上改变世界,而且大多是往好的方向改变。

Even if AI could permanently only do the things it can currently do, that would be Internet big, and radically change the world, mostly for the better.

AI 能力不会永久止步于此。不接受第二颗药丸是错误的。

AI capabilities will not permanently stop here. It is wrong to not take the second pill.

卡在第一颗药丸 Stuck At The First Pill

我们关于 AI 的辩论在很大程度上仍停留在已解决的问题上,因为许多人甚至无法迈出第一步。

Our debates about AI remain largely stuck on settled questions, because so many people cannot even take the first pill.

Dean W. Ball:几年前,AI 辩论理所当然地集中在一些听起来疯狂的事情上,比如“AI 自主取得数学突破”和“AI 突破沙盒并在互联网上黑客攻击”在短期内是否会成为现实。有时感觉我们还在进行那样的辩论。这令人沮丧,因为在我看来,那场辩论已经解决,而且很久以前就解决了。

Dean W. Ball: A couple years ago, the AI debate was centered, rightfully, on whether crazy-sounding things like “AIs autonomously making math breakthroughs” and “AIs breaking from their sandbox and hacking on the internet” would be real things in the near term. Sometimes it feels like that’s still the debate we’re having. This can be frustrating, because in my view, that debate is settled and was settled quite a while ago.

要进行良好的讨论,我们至少需要迈出第二步。

To have good discussions, we need to at least take the second pill.

然而,是的,许多人,甚至许多从事 AI 工作的人,确实说当前的 AI 已经‘足够好’,无法想象更好的 AI 能做什么。比如,有人在推特上对 Sam Altman 说‘Sol 做了所有我想让它做的事,这就是我需要的全部’,而 Altman 转发说他们错了。他们显然错了。

Whereas, yes, many people, even many who work with AI, really do say current AI is ‘good enough’ and can’t imagine what a better one can do. As in, someone tweeting at Sam Altman saying ‘Sol does everything I want it to do, this is all I ever need’ and Altman retweeting saying they were wrong. Which they obviously are.

AGI 药丸 The AGI Pill

AGI 药丸远比 AI 药丸重要得多。

The AGI pill is a much bigger deal than the AI pill.

如果你服下 AGI 药丸,你就会明白 AI 正在快速提升其能力。

If you take the AGI pill, you understand that AI is advancing its capabilities rapidly.

即便你认为这类 AGI 将完全处于人类控制之下、仍只是“纯粹的工具”,并且你预期大多数人的日常生活体验不会发生如此剧烈的变化,你也会明白它们的能力将“改变一切”。

Even if you think that such AGIs will remain fully under human control, and remain 'mere tools,' and you expect the lived experience of most people's everyday lives to not change so radically, you understand that their capabilities will 'change everything.'

你会看到,我们将面临剧烈转型与不确定性的时代,这些时代有可能走向极其糟糕的结局,而在 AI 上取得成功的人将把未能成功者远远甩在身后。

You see that we will face times of great transition and uncertainty, that have the potential to go extremely badly, and that those who succeed at AI will leave those who do not behind in the dust.

未来的世界将与我们的世界截然不同。AI 将能够在大多数时间完成大多数数字工作,还有不可避免的机器人、自动驾驶汽车等等。许多现有工作将消失,无论它们是否会被新的、可能更好的工作所取代。经济增长与生产率将加速。

The world in the future will be very different from our own. AIs will be able to do most digital work, most of the time, along with the inevitable robots and self-driving cars and so on. Lots of current jobs will go away, whether or not they are replaced by new and potentially better ones. Economic growth and productivity will accelerate.

你会看到,如果我们尚未做好准备就赋予对这类先进 AI 系统的滥用能力,将会带来哪些危险,尤其是在网络与生物风险等领域。

You see some of the dangers of what would happen if we empowered misuse of such advanced AI systems before we were ready, especially in places like cyber and bio risk.

你会看到权力集中或不平等的潜在可能,也会看到某些形式的失控式渐进性去权化(disempowerment)。

You see the potential for centralization of power, or inequality, and also for some forms of runaway gradual disempowerment.

你会看到大规模失业的潜在可能,无论是过渡性的还是永久性的。

You see the potential for mass unemployment, either transitional or permanent.

你会认识到,我们的法律与监管制度尚未就绪:既不足以防范和缓解风险与危害,也不足以把握机遇、消除阻碍扩散与日常效用的瓶颈。

You understand that our legal and regulatory regimes are not ready, either to protect against and mitigate the risks and harms, or to allow for the opportunities and remove the bottlenecks to diffusion and mundane utility.

有备无患的必要性 The Need To Be Prepared

那些预期 AI 会迅速变得足够先进、从而对物理世界产生巨大影响的人,通常看到了巨大的危险。他们注意到,结果可能是所有人很快都会死亡。通常,他们实际上认为这是坏事。

Those who expect AI to quickly become sufficiently advanced to greatly impact the physical world usually see great danger. They notice that as a result everyone may soon die. Usually they think this is bad, actually.

因此,这些人呼吁采取协调行动,以减轻此类影响的下行风险,防止我们所有人死亡,并且理想情况下也帮助获取上行收益。

Thus such folks call to take coordinated action to mitigate the downside risks of such impacts, keep us all from dying, and ideally also to help capture the upside benefits.

那些预期 AI 会变得重要地更加先进,但对物理世界的影响更慢且更小,并且认为更多智能的实际价值将会封顶的人。

Those who expect AI to become importantly more advanced, but with a slower and smaller impact on the physical world, and who think the practical value of more intelligence will cap out.

对于‘足够先进’的不同取值,例如 AGI 与 ASI,你会看到不同程度的危险。

For different values of 'sufficiently advanced,' as in AGI versus ASI, you would see different degrees of danger.

截至 2026 年 8 月,AGI 药丸对于奥弗顿窗口内或正在被认真考虑的大多数事情仍然足够。我们几乎完全在考虑过度确定的、低成本、高收益的干预措施。

The AGI pill is still sufficient for most things in the Overton window or under serious consideration as of August 2026. We are almost entirely considering overdetermined, low cost, high benefit interventions.

没有任何正当理由不去大幅加大对以下方面的投入:对齐、基础设施与监督、国家能力、透明度、责任、信息披露、安全测试(包括对内部模型的测试)、红队测试、审计、出口管制的执行,以及为外交奠定基础。

There is no good case for not doing radically more investment in alignment, infrastructure and oversight, state capacity, transparency, liability, disclosures, safety testing including of internal models, red teaming, auditing, enforcement of export controls and laying the groundwork for diplomacy.

这包括在必要时为“Pace the Frontier”(控制前沿发展节奏)奠定基础。

This includes laying the groundwork to Pace the Frontier should that prove necessary.

如果你完全信奉 ASI(超级智能),并且对当前对齐的现状以及超级智能若很快到来可能如何演变持现实态度,那么你会希望走得更远。你会希望去做那些确实有负面影响、需要真正权衡取舍的事情。

If you are fully ASI pilled, and realistic about the current state of alignment and how superintelligence likely plays out if it arrives soon, then you will want to go further. You will want to do things that have real downsides, and require real tradeoffs.

一些持此类观点的人希望对前沿 AI 发展实施全面的国际暂停。如果你完全接受了 ASI 这一信念,并且像他们一样相信超级智能——就其可能多快到来以及它能做什么而言——那么你很可能会赞同他们。

Some such people want to do a full international pause of frontier AI development. If you took the full ASI pill and believed what they do about superintelligence, in terms of how fast it might arrive and what it can do, you might well agree with them.

ASI 药丸 The ASI Pill

ASI 药丸是一种认识:AI 正朝着能够在几乎所有事情上比你做得更好而发展。

The ASI pill is the understanding that AI is on pace to be able to do approximately all of the things better than you.

我说的是‘几乎所有’。正如我详细讨论的,这并不意味着字面上的所有事情。有些事情本质上需要人类或极大地受益于人类。而且可能存在一些奇怪的边缘情况,AI 不够好。

I said ‘approximately.’ As I go over in detail, that does not mean literally all of the things. There are some things that inherently require or greatly benefit from being a human. And there may be weird corner cases where the AI won’t be good enough.

这并不意味着全能或全知,尽管人们应该预期,对于未增强的人类来说,它看起来会非常像那样。

It does not mean omnipotence or omniscience, although one should expect it to look a lot like that to an unaided human.

这确实意味着 AI 会取代你的工作,然后取代你转向的新工作,除非你转向‘需要真正人类’的领域。你在几乎任何其他任务上都将缺乏竞争力。

It does mean the AI takes your job, and then takes the new job that you switch into, unless you pivot to ‘requires literal human.’ You will be uncompetitive at essentially any other task.

这确实意味着它将利用这种能力以惊人的速度弄清楚几乎所有事情,直到你触及物理极限。

It does mean that it will use this capability to figure out approximately all of the things, remarkably quickly, until you hit the physical limits.

这确实意味着,那些更依赖此类 AI 的人将在所有意义上(包括资源方面)可靠地胜过那些较少依赖此类 AI 的人。

It does mean that those who rely more on such AIs will reliably outcompete, in all senses including for resources, those that rely on such AIs less.

这确实意味着,在“公平竞争”或充分开放的竞争中,AI 会获胜。

It does mean that, in a ‘fair fight’ or sufficiently open competition, the AI wins.

这也意味着 AI 常常会想出并做出你未曾想象或预料的事情。

It also means the AIs often figuring out and doing things you did not imagine or anticipate.

这意味着要认识到,智能远不会在人类水平附近止步,其通过因果空间规划通往偏好原子排列的路径的能力也是如此。

It means realizing that intelligence does not stop anywhere near the human level, nor does its ability to chart paths through causal space towards preferred arrangements of atoms.

这也意味着不要假装其卓越的智力可以被你微不足道的武器、纸上的墨迹、数据库中的条目,或你的监管俘获和寻租所匹敌。

It also means not pretending that its superior intellect can be matched by your puny weapons, or your pieces of ink on paper, or your entries in a database, or your regulatory capture and rent seeking.

然后对你来说没什么大变化 And Then Nothing Much Changes For You

尽管如此,AGI 药丸(而非 ASI 药丸)的标志是相信日常生活将继续看起来与现在相似,就像我们认为 1926 年的日常生活与 2026 年的生活没有太大不同一样。

Despite all that, the sign of the AGI pill, as opposed to the ASI pill, is the belief that day-to-day life will continue to look similar to how it looks now, in the sense that we see day-to-day life in 1926 as not that different from life in 2026.

有时这显然是虚伪的,因为说这话的人对 ASI 足够着迷,本应更明白。我把 Sam Altman 说的各种形式的‘奇点之后生活不会有太大变化’解读为基本上是没有根据的明显胡说,是为了安抚听众,而不是一个连贯的立场。有时这类人最想安抚的人是他们自己。

Sometimes this is clearly disingenuous, as it is coming from someone ASI-pilled enough to know better. I read Sam Altman saying various forms of ‘life will not much change after the singularity’ as basically unjustified Obvious Nonsense to reassure his audiences, rather than a coherent position. Sometimes the person such folks want to reassure most is themselves.

其他人有时对未来以及为什么未来不会有太大变化有更具体的想法,其连贯程度各不相同。

Others sometimes do have something more concrete in mind as their vision of the future, and why it will not change so much, with varying degrees of coherence.

只有 AGI 药丸的人相信存在阻碍变化的瓶颈。

Those with only the AGI pill believe in bottlenecks that hold back change.

他们通常认为,我们指数级增长 AI 容量和能力的能力将遇到各种物理限制。芯片数量有限。行动需要时间。太遥远的东西常被贬斥为‘魔法’。

They often believe that our ability to exponentially grow AI’s capacity and capabilities will hit various physical limits. There can only be so many chips. Actions take time. Things too far out there are often pejoratively dismissed as ‘magic.’

他们常常认为,从经典的“关闭专利局”式的角度来看,即便在理论上,物理上可供发现的东西也已经不多了。你的牛排只能嫩到某个程度,你的龙虾只能黄油味浓到某个程度,你的寿命只能长到某个程度,你的地位只能高到某个程度,那又有什么要紧呢。我在寿命和健康方面强烈反对这一看法,并预期在寻找价值方面,我们在许多其他方面还有很长的路要走,尽管他们关于物理人类大脑每时每刻的最大享乐体验或许有一定道理。

They often believe there is not that much left to physically discover, in the classic 'close the patent office' kind of way, even in theory. Your steak can only be so tender, your lobster so buttery, your lifespan so long, and your status so high, so why does it matter. I strongly disagree on lifespan and health, and expect we have a long way to go in so many other ways in terms of finding value, although they may have a point about moment-to-moment maximal hedonic experiences of a physical human brain.

他们常常认为,智能带来的上行空间是相当有限的。即任何心智,无论多么先进,都不可能那么有说服力,或那么有经济价值,或那么有能力在物理世界中创造创新,或运行足够精确的模拟,或做出足够强的预测,甚至无法做到像克服繁文缛节和监管俘获这样的事情。

They often believe that the upside of intelligence is importantly limited. That no mind, however advanced, could be all that persuasive, or that economically valuable, or that capable of creating innovations in the physical world, or of running sufficiently accurate simulations, or making sufficiently strong predictions, or even able to do things like overcome red tape and regulatory capture.

智能否认主义 Intelligence Denialism

我有时称之为智能否认主义:认为无论变得多聪明,更聪明也并非那么重要。认为智能这种东西,你要么有要么没有,而且心智是有上限的。

I sometimes call this Intelligence Denialism: The idea that being smarter is not all that, no matter how smart one gets. That there is this thing, intelligence, that you either have or don’t have, and that minds cap out.

这常常延伸到否认更聪明的人类能够做到并完成他们显然做到并完成的事情。其他时候,这种观点认为智能的上限是‘聪明的人类’,而所有心智所能做的只是模仿那个聪明的人类。也许你可以做得更快、更便宜、更大规模,拥有更好的记忆等等。

Often this extends to denying that more intelligent humans can do and accomplish the things they clearly do and accomplish. Other times, it is the idea that intelligence tops out at ‘smart human,’ and all a mind can do is imitate that smart human. Maybe you can do it faster and cheaper, and at scale, with better memory and so on.

但仅此而已。这些人未能理解,如果你将全人类所有心智能力以及所有获取知识的途径,大规模地、并行地、更快更便宜地结合起来,仅此一点就会让任何地方的任何人相形见绌。而且,如果这缺乏物理能力或访问权限,获得这些将是轻而易举的。

But that’s it. And such folks fail to understand that if you took the union of all human mental capabilities, and all access to knowledge, at scale, in parallel, much faster and cheaper, that this alone would run circles around anyone and everyone, everywhere. And that if this lacked physical capabilities or access, this would be trivial to get.

这通常是 AGI(通用人工智能)信仰者不服用 ASI 药丸的核心正当理由。他们无法理解超级智能是真实存在的。

This is, usually, the central good reason people who are AGI pilled do not take the ASI pill. They are unable to understand that superintelligence is a thing.

这种混淆也常常是模型将商品化的直觉背后的原因。存在一种理想的东西,‘智能’,你通过渐近线接近它。更多是不可能的,或者更多不会帮助你,取决于你如何表述。

This confusion is also often behind the instinct that models will commoditize. There is an ideal thing, ‘intelligence,’ and you approach it via an asymptote. More is impossible, or more won’t help you, depending on how you frame it.

超级智能与全知全能之辨 Superintelligence Versus Omniscience and Omnipotence

一个常见的错误是认为任何超级智能的存在都会是全知的、甚至是全能的,因此超级智能是不可能的。

A common error is to think that anything superintelligent would be omniscient or even omnipotent, and therefore superintelligence is impossible.

然而,同样地,在人类之上存在大量“空间”,而并不需要变成全知全能。

Whereas, again, there is a lot of 'space above humans' without becoming omni.

还有许多其他类似的混淆,即把“对[X]存在某个上界”与“我们正处于或接近[X]的上界”混为一谈。

There are many other similar confusions, where 'there is some upper bound to [X]' is confused with 'we are at or near the upper bound to [X].'

在这里,Adi 为我们提供了一种解释正在发生之事的途径,他所做的在我看来正是核心的 motte-and-bailey(堡垒与外围)论证的镜像。

Here Adi gives us a way into explaining what's happening, by doing what from my perspective is a mirror image of the central motte-and-bailey.

Timothy B. Lee:在那些认为超级智能意味着近乎全能的人与那些(像我一样)不这么认为的人之间,存在一道认知鸿沟。

Timothy B. Lee: There's an epistemic chasm between those who think superintelligence implies near-omnipotence and those (like me) who don't.

Adi:在“超级智能”的两种定义之间,存在一种**莫特与贝利(motte-and-bailey)**式的偷换手法。

Adi: There is a motte-and-bailey trick being played between two definitions of "superintelligence".

其一是“大写 S 的超级智能(Big-S Superintelligence)”,按定义即全知全能。

"Big-S Superintelligence", which is definitionally omnipotent.

其二是“小写 s 的超级智能(small-s superintelligence)”,即当前 AI 研究脉络所假设的终局状态。

And "small-s superintelligence", which is the hypothesized end state of the current thread of AI research.

令人惊讶的是,AI 安全社区对这些可疑的等同关系处理得如此不加批判,尤其是考虑到对莫特与贝利谬误最著名的论述之一,恰恰出自该社区中相当知名的 Scott Alexander 之手。

It is surprising how uncritically the AI safety community treats these dubious equivalences, especially since one of the most famous treatments of the motte-and-bailey fallacy comes from none other than Scott Alexander, who is quite prominent in the community.

Zvi Mowshowitz:这里的莫特与贝利手法确实存在,但更常见的情形是反向的,即表现为这样一种形式:

Zvi Mowshowitz: The motte-and-bailey here is real, but more commonly it is in reverse, as in a form of:

1. 你声称超级智能 [S] 很快就能远超人类,做出荒谬的事情。

1. You claim superintelligence [S] will soon run circles around humans and do absurd stuff.

2. 但 [S] 并不会是全知全能的。

2. But [S] would not be fully omniscient/omnipotent.

3. 因此 [S] 将是一种~正常的技术。

3. Therefore [S] would be a ~normal technology.

Adi:我也见过这种说法,并且我同意它同样是错误的。

Adi: I've seen this too, and I agree that it's equally wrong.

仍然会有一些事情需要花费不可忽略的现实世界时间才能实现。会有特定的行动、路径和方法,它们各有其局限性。这些局限性很重要。将它们完全抽象掉可能是一个巨大的错误。

There will still be things that require non-trivial real-world time to achieve. There will be particular actions and paths and methods that have their limitations. These limitations matter. Fully abstracting them away can be a large mistake.

我不断看到这样的说法:“一个至善的心灵仍然无法做到[在我看来它显然能做到的事]”,甚至[它已经能做到的事]。我认为这种连词形式在很大程度上解释了人们为何以及如何为自己辩护。当 AI 获得能力、更多事情被攻克时,他们移动球门,却不改变游戏规则。

I constantly see versions of 'the mind that is maximally good still could not do [thing it seems to me it could obviously do]' or even [thing it can already do] and I think this form of conjunction is a lot of why, and how people justify themselves. When AI gains capabilities, and more things fall, they move the goalposts but don’t change the game.

Adi 的上述评论是在讨论 Plan A 的背景下提出的,在 Adi 的术语中,这些 AI 显然不是“大写 S 的超级智能”。这些 AI 有一些实际做不到的事,或者只能做到某种程度。

Adi’s comment above was in the context of discussing Plan A, where the AIs are very obviously not ‘Big-S Superintelligence’ in Adi’s lexicon. There are practical things that these AIs cannot do, or can only do up to a point.

这个错误并没有被犯下。它有时会被犯,但很罕见。

The mistake was not being made. It sometimes gets made, but this is rare.

说服:一个实例分析 Persuasion Persuasion (A Worked Example)

充其量只能主张某种特定能力——通常是极为有效的说服(常被称为“超级说服”)——不可能实现。事实上,一种常见的论点是:即便是理想化的 AI,也无法超越最优秀的人类说服者,甚至无法超越一个典型优秀的人类说服者(后者远逊于历史上最优秀的人类说服者),尽管它相对于人类拥有诸多优势,例如快得多的思考速度以及获取信息的能力,包括目标对象的每一个细微反应,乃至肢体语言。

At most one can claim that some particular capability—typically very effective persuasion (often 'super-persuasion')—would not be possible. Indeed, a common argument states that even an idealized AI would be unable to be superior to the best human persuader, or even a typical good human persuader (who is much worse than the historical best human persuaders), despite it having access to many advantages over humans, such as much faster thinking speed and access to information, including every little reaction of its target including body language.

人们常常通过援引“情境对说服很重要”来做到这一点。许多人谈论起来仿佛这种情境会压倒一切,尽管这对人类而言并不成立。他们还认为,能力极强的 AI 无法设计出有利的情境。

Often this is done by citing that context matters for persuasion. Many talk as if this context would trump everything, despite this not being true for humans. They also believe that highly capable AIs could not engineer favorable contexts.

这往往表现为一种“我干脆选择不被说服”的说法,或者“如果你不能冷启动、仅凭文本一次性说服所有人,那就不算数”之类的话。

Often it's a form of 'I would simply choose not to be persuaded' or 'if you can't do it cold and one shot everyone with only text it does not count' or something like that.

如果说服需要人的在场,或者需要社会情境,它可以通过各种方式轻而易举地获得。

If it needs a human presence to do the persuasion, or a social context, it can acquire one easily enough in various ways.

我完全无法理解为什么会有人认为先进的 AI 会保持相对缺乏说服力,除非是认为 AI 不会比现在强多少——而且这还得是在我们完全限制 AI 只能使用已知的标准说服技巧、并排除任何奇技淫巧的前提下。

I flat out cannot understand why someone would think advanced AIs will remain relatively unpersuasive, other than to think that AI will not get much more capable than it already is, and that's if we fully restrict the AI to using known standard persuasion techniques and rule out any wizardry.

然而我并不是一个超级说服者,所以许多人觉得我的论证缺乏说服力。

Yet I am not a superhuman persuader, so many find my arguments unconvincing.

不,说真的——就像 Adi 引用那段高亮文字时,仿佛这并非显而易见地成立。天哪,在一个 AI 能力持续快速发展的情景中,你怎么会认为这不会发生?要说的话,2035 年都显得慢得离谱:

No, seriously—as in Adi is citing the highlighted passage as if this were not very obviously true. My lord, how can you think this is not going to happen in a scenario where AI capabilities continue to develop apace? If anything, 2035 seems crazy slow:

而且,最坏的情况,AI 戴个耳机告诉人类该怎么做,天哪。

And like, worst case, the AI gets an earpiece and tells a human what to do. Jesus.

Adrien Ecoffet:一个超级有说服力的 AI,能否利用现有建筑技术,以低于 500 亿美元的成本、在 10 年内建成旧金山与洛杉矶之间的高铁线路?

Adrien Ecoffet: Could a superpersuasive AI get a high-speed rail line between SF and LA constructed for under $50B and under 10 years with existing construction technology?

Dean W. Ball:不

Dean W. Ball: No

Nat McAleese:@deanwball

Nat McAleese: @deanwball

我认为这是一个重要的问题,不应过度自信!如果它感觉显而易见,那你肯定是过度自信了。

I think this is an important question not to be overconfident on! And if it feels obvious, you are surely overconfident.

Dean W. Ball:显然没有保证,但“前沿 AI 系统改革 CEQA,然后干净利落地通过加州政府输送数百亿美元”似乎不现实,而且与许多其他事情相比,也不像是一件值得优先考虑的事

Dean W. Ball: obviously there are no guarantees, but "frontier AI systems reforming CEQA and then cleanly funneling tens of billions of dollars through California's government" doesn't seem realistic + like a thing to prioritize compared to many other things

对于足够有说服力的 AI,我的答案基本上是:是,但不会不顺便接管至少加州政府。工具性趋同。

My answer, for a sufficiently persuasive AI, is essentially: Yes, but not without incidentally taking over at least the California government. Instrumental convergence.

正如 Dean 所回复的,这并不是说你会因此选择专注于高速铁路。这不是一个“会”的问题,而是一个“能”的问题。关键在于理论上的“能”。

Not that you would then choose to focus on High Speed Rail, as Dean replies. This is not a case of would, so much as could. The point is the could, in theory.

一个几乎过于完美的例子,展示了“你将永远无法做到[你已经能做到的事情]”:

An example that is almost too perfect an example of ‘you will never be able to do [thing that you can already do]’:

Ramez Naam:“无论你的 token 质量多高,它们都不能把铅变成金。”令人愉快地失控,但总体正确。

Ramez Naam: “No matter how high quality your tokens are, they cannot turn lead into gold.” Pleasantly unhinged and overall correct.

《奇点更近》:在《Prime Intellect 的变形记》中,硬起飞之所以奏效,是因为 AI 发现了关联效应,某种操纵物质的量子技巧。在现实中,并不存在关联效应。无论你的 token 质量多高,它们都不能把铅变成金。

The Singularity is Nearer: In _The Metamorphosis of Prime Intellect_, the hard takeoff works because AI discovers the correlation effect, some quantum trick to manipulate matter. In reality, there is no correlation effect. No matter how high quality your tokens are, they cannot turn lead into gold.

Ryan Greenblatt:我明白这个想法,但这是一个有趣的例子,因为把铅变成金是完全可行的。极其强大的 AI 系统(在技术研发和基础设施建设之后)将有能力非常迅速地建立把铅变成金的基础设施。

Ryan Greenblatt: I get the idea, but this is a funny example because it is totally feasible to turn lead into gold. Extremely capable AI systems (after tech R&D and infrastructure build out) will have the capacity to very quickly build infrastructure for turning lead to gold.

我们其实在 2025 年就已经成功把铅变成了金,只是用现有方法成本高得令人却步。人们确实在以一种方式对世界建模:凡是目前还没有明确路径做到的事,就永远不可能做到。人人都想关掉专利局。

We already have successfully turned lead into gold in 2025; it is just cost-prohibitive with current methods. People really do model the world as if anything we don't specifically have a clear path to already doing cannot ever be done. Everyone wants to close the patent office.

AI 可能做到但并非“觉醒”所必需的事 Things AI Could Probably Do But Are Not Required For Being Pilled

这样的 AI 能否向生物实验室发送订单,并最终得到可编程的自我复制金刚石纳米探针?它能否在没有任何物理实验的情况下、仅凭模拟就做到这一点?它能否通过一系列相对快速的实验做到这一点?它能否转而做其他同样强大、能绕过一切限制并让它为所欲为的事情?时间尺度如何?

Could such an AI send off an order to a bio lab and end up with programmed self-replicating diamond nanoprobes? Could it do so without any physical experiments, based purely on simulations? Could it do so with a relatively fast series of experiments? Could it instead do other similarly powerful things that do an end run around everything and let it do whatever it wants? On what time frame?

无论是完全 RSI(递归自我改进)的终点,还是当前循环在实践中可能达到的上限,都没有人知道物理极限在哪里。我完全预期会发现各种我们未曾预料、且在多数情况下未曾想象过的心理与物理能力以及新的可供性(affordances),而按定义,很难知道具体是哪些。

No one knows where the physical limits lie, either for the end of full RSI (recursive self-improvement) or for where the current cycles might top off in practice. I fully expect to discover various mental and physical capabilities, and new affordances, that we did not expect, and in many cases did not imagine, and by definition it is hard to know which.

我基本上不再谈论这些可能性,尽管它们在我看来相当可能,因为:

I mostly stopped talking about those possibilities, despite them seeming rather likely to me, because:

1. 当你提到这类事情时,人们会攻击该主张,并以此为由否定你的整个论证、先进 AI 的所有风险,以及你这个人。

1. When you mention such things, people attack that claim, and use it as a reason to dismiss your entire argument, all risks from advanced AI, and you as a person.

2. 这种策略奏效,因为这类主张要证明起来困难得多,确定性也更低。

2. This strategy works, because such claims are much harder to justify, less certain.

3. 你不需要这些主张就能得到相同的结果。足够先进的 AI 仅凭我们确信这类 AI 会具备的能力就能“达到目标”,因为人类已经拥有这些能力,只需足够的算力、参数和数据。

3. You don’t need such claims to get the same results. Sufficiently advanced AI can ‘get there’ with only capabilities we can be confident such AIs will have, because humans already have them modulo sufficient compute, parameters and data.

例如:也许詹姆斯·邦德不能总是赢得女孩,也许他能,也许 Q 的最新小工具是你能造出来的,也许不是,但我非常确定邦德有枪而且很擅长用枪,而在这一件事上枪确实是他所需要的全部,但很多人不明白这一点。所以我选择这样论证,就好像他只有枪一样。

E.g.: Maybe James Bond can’t always get the girl, maybe he can, and maybe Q’s latest gadget is a thing you can build and maybe it isn’t, but I am damn sure Bond has a gun and is very good with it, and the gun really is all he needs on this one, but a lot of people don’t understand this. So I choose to argue as if all he has is the gun.

另一种方法是这个,它对某些人有效,但我预计它对 Timothy Lee 完全没有说服力:

Another approach is this, which works for some but I do not expect it to be at all persuasive to Timothy Lee:

Jack Gallagher:让我豁然开朗的直觉是,不要从“智能”的角度来思考,而要从 metr 时间跨度之类的角度来思考。很多工作都受限于智能 × 繁琐程度,而随着你在这些维度上的提升,几乎一切最终都会迎刃而解。也许你仍然缺少一些伟大的创造性火花,但纳米技术从这里开始并不需要创造性火花,它需要的是在小型化上不懈地钻研。

Jack Gallagher: the intuition that made it click for me was thinking of it less in terms of “intelligence” and more in terms of things like metr time horizons. a lot of work is intelligence x schlep gated and as you improve on those axes just about everything eventually falls. maybe you still miss some grand creative spark but nanotech doesn’t require a creative spark from here it requires relentless grinding on miniaturization.

许多事情仍将受限于串行实验,但随着任何给定智能水平的成本趋近于零,我们最终会并行运行更多可用的实验。

many things will still be bound on serial experimentation, but as the cost of any given level of intelligence goes to zero we’ll end up running way more of the available experiments in parallel.

生活节奏日益加快 Life Comes At You Increasingly Fast

人们止步于 AGI 这颗药丸的另一个原因,是不理解递归式自我改进、奇点,或事物可能正在急剧加速这些想法,要么就是直接把它们当作太科幻、太怪异、太荒谬之类的东西而一口回绝。

Another reason people stop at the AGI pill is not understanding, or rejecting out of hand as too sci-fi or weird or absurd or what not, the idea of recursive self-improvement, or a singularity, or that things might be radically accelerating.

他们能接受事物正在加速,但不能接受加速本身也在加速,如此类推——而这确实已经在发生,且难以忽视。

They can accept that things are speeding up, but not that the speeding up is itself speeding up, and so on, which indeed is already happening and hard to miss.

从某种意义上说,这种情况已经持续了很长时间。数学预测了一个奇点。于是这有时被用来表示“这并非新进展”,但指数增长就是这样运作的。核心上并没有新东西发生,然而轰的一声。

This has in some sense been happening for a very long time. The math predicts a singularity. This is then sometimes used to say 'well this is not a new development' but that is how exponentials work. Nothing centrally new happens, and yet kaboom.

提醒一下(取整处理,别 @ 我):

As a reminder (with rounding, don't @ me):

4. 300 年前:工业革命。

4. 300 years ago: Industrial Revolution.

你可能会被这样的说法误导:“哦,存在这样一个阶跃变化,你会突然进入奇点并实现递归自我改进,而在那之前则不会”;或者被“这个新事物并没有什么不同”所误导。

You can be misled by 'oh, there is this step change where you are suddenly in a singularity and having recursive self-improvement, and until then no' or 'this new thing is not different.'

不押注 AGI 是否合理? Is It Reasonable To Not Be AGI Pilled?

如果你没有对 AGI 抱有坚定信念(即不是“AGI pilled”),那么你对 AI 的反应就不会明智或审慎。遗憾的是,我们的许多政策和讨论正由那些没有对 AGI 抱有坚定信念的人所主导。

If you are not AGI pilled, your reactions to AI will not be wise or prudent. Alas, much of our policy and conversation is being driven by people who are not AGI pilled.

当我们看到依赖“未对 AGI 抱有坚定信念”这一前提的立场时,我们应当明确指出这一点,并据此决定是否与之互动。

When we see positions that rely on not being AGI pilled, we should say so, and engage or not engage with them accordingly.

只押注 AGI 是否合理? Is It Reasonable To Only Be AGI Pilled?

是的,前提是你认真对待其影响,并清楚自己的关键症结所在。

Yes, if you seriously grapple with its implications, and know what your cruxes are.

我认为这是错的。但它是一个自洽的立场——至少是错的——即认为我们当前的技术和资源不会完全“达到目标”,并且 AI 能力很可能在我们真正称之为超级智能的东西之前就达到上限。

I think it is wrong. But it is a coherent position, that is at least wrong, to think that our current techniques and resources will not fully 'get there' and AI capabilities are likely to top out before things we would properly call superintelligence.

或者至少,我不认为把这种立场视为不合理会有成效。

Or at least, I don't think treating this as unreasonable would be productive.

如果我们“撞墙”,即至少在一段持续时间内回报明显递减,尤其是如果我不觉得使用去年的模型落后太多,而且事物确实商品化了,我会转而越来越重视这一立场。

I would switch to putting increasing weight on this position if we 'hit a wall' of at least clearly diminishing returns for a sustained period of time, and especially if I don't feel so behind using last year's model and things really do commoditize.

我们仍可能看到 AI 收入和算力需求的急剧增长。AI 投资可能在供应链上下游带来丰厚回报。这些都与只押注 AGI 的立场完全兼容。

We could still see dramatic growth in AI revenue and demand for compute. Investments in AI could pay off handsomely up and down the supply chain. That is all fully compatible with the AGI-only position.

如果你确实相信仅 AGI(通用人工智能)的立场,我有两个请求。

If you do believe the AGI-only position, I have two requests.

第一,认真对待你认为 AI 将能够做到的事情所带来的影响,不要退缩,也不要试图假设一切都会神奇地解决,而人们的生活体验不会有太大变化。这意味着要支持我们现在需要采取的行动,以减轻所有这些风险,并帮助事情顺利发展。

First, seriously tackle the implications of what you do think AI is going to be able to do, without flinching, and without trying to assume everything magically works out and somehow people's life experiences do not much change. That means then supporting actions now that we need to mitigate the risks of all that, and help make things go well.

第二,写下阻止你接受 ASI(超级智能)信念的关键点。特别是,写下(在这里的评论中会很好)AI 永远无法做到的最不令人惊讶或印象深刻的事情,或者会改变你对这件事走向的看法的事情,以及哪些其他近期观察会让你确信自己是对的,或者意识到自己错了。

Second, write down what are your cruxes that are stopping you from taking the ASI pill. In particular, write down (in the comments here would be great) what is the least surprising or impressive thing an AI will never be able to do, or that would change your mind about where this is going, and what other near term observations would cause you to either be confident you are right, or realize you are wrong.

在那之前,有两个有效的选择:你可以停留在 AGI(通用人工智能)信念,或者完全转向 ASI(超级智能)。

Until then, there are two valid choices: You can stop at the AGI pill, or go full ASI.

还有两个常见但无效的选择:认为 AI 到此为止,或者否认现实。

There are also two common but invalid choices: Think AI stops here, or deny reality.

互动版:针对本篇提问 →

接着读 · Read next