达里奥·阿莫代——爱的恩典机器

仁慈的机器 1 Machines of Loving Grace1(https://darioamodei.com/essay/machines-of-loving-grace#fn:1)

1https://allpoetry.com/All-Watched-Over-By-Machines-Of-Loving-Grace

1https://allpoetry.com/All-Watched-Over-By-Machines-Of-Loving-Grace

人工智能如何让世界变得更美好

How AI Could Transform the World for the Better

我经常思考和谈论强大 AI 的风险。我担任 CEO 的公司 Anthropic 做了大量研究来降低这些风险。因此，人们有时会得出结论，认为我是一个悲观主义者或“末日论者”，觉得 AI 主要是坏的或危险的。我完全不这么认为。事实上，我关注风险的主要原因之一是，风险是唯一阻碍我们走向我认为本质上是积极未来的东西。我认为大多数人低估了 AI 可能带来的积极影响，就像我认为大多数人低估了风险可能有多严重一样。

I think and talk a lot about the risks of powerful AI. The company I’m the CEO of, Anthropic, does a lot of research on how to reduce these risks. Because of this, people sometimes draw the conclusion that I’m a pessimist or “doomer” who thinks AI will be mostly bad or dangerous. I don’t think that at all. In fact, one of my main reasons for focusing on risks is that they’re the only thing standing between us and what I see as a fundamentally positive future. I think that most people are underestimating just how radical the upside of AI could be, just as I think most people are underestimating how bad the risks could be.

在这篇文章中，我试图勾勒出这种积极影响可能的样子——如果一切顺利，拥有强大 AI 的世界会是什么样子。当然，没有人能确定或精确地预知未来，强大 AI 的影响可能比过去的技术变革更难以预测，因此所有这些都不可避免地由猜测组成。但我的目标是至少做出有根据且有用的猜测，捕捉将要发生的事情的基调，即使大多数细节最终是错误的。我包含大量细节，主要是因为我认为一个具体的愿景比一个高度规避和抽象的愿景更能推动讨论。

In this essay I try to sketch out what that upside might look like—what a world with powerful AI might look like if everything goes _right_. Of course no one can know the future with any certainty or precision, and the effects of powerful AI are likely to be even more unpredictable than past technological changes, so all of this is unavoidably going to consist of guesses. But I am aiming for at least educated and useful guesses, which capture the flavor of what will happen even if most details end up being wrong. I’m including lots of details mainly because I think a concrete vision does more to advance discussion than a highly hedged and abstract one.

然而，首先我想简要解释一下为什么我和 Anthropic 没有过多谈论强大 AI 的积极面，以及为什么我们总体上可能会继续大量谈论风险。具体来说，我做出这个选择是出于以下愿望：

First, however, I wanted to briefly explain why I and Anthropic haven’t talked that much about powerful AI’s upsides, and why we’ll probably continue, overall, to talk a lot about risks. In particular, I’ve made this choice out of a desire to:

* 最大化杠杆作用。AI 技术的基本发展及其许多（并非全部）好处似乎是不可避免的（除非风险破坏一切），并且从根本上由强大的市场力量驱动。另一方面，风险并非预先确定，我们的行动可以极大地改变其可能性。

* Maximize leverage. The basic development of AI technology and many (not all) of its benefits seems inevitable (unless the risks derail everything) and is fundamentally driven by powerful market forces. On the other hand, the risks are not predetermined and our actions can greatly change their likelihood.

* 避免被视为宣传。AI 公司谈论 AI 的所有惊人好处可能会显得像宣传者，或者像是在试图分散人们对负面影响的注意力。我也认为，从原则上讲，花太多时间“推销自己的观点”对你的灵魂不好。

* Avoid perception of propaganda. AI companies talking about all the amazing benefits of AI can come off like propagandists, or as if they’re attempting to distract from downsides. I also think that as a matter of principle it’s bad for your soul to spend too much of your time “talking your book”.

* 避免自大。我常常对许多 AI 风险公众人物（更不用说 AI 公司领导者）谈论后 AGI 世界的方式感到反感，好像他们的使命是单枪匹马地实现它，就像带领人民走向救赎的先知一样。我认为将公司视为单方面塑造世界是危险的，将实际的技术目标本质上视为宗教术语也是危险的。

* Avoid grandiosity. I am often turned off by the way many AI risk public figures (not to mention AI company leaders) talk about the post-AGI world, as if it’s their mission to single-handedly bring it about like a prophet leading their people to salvation. I think it’s dangerous to view companies as unilaterally shaping the world, and dangerous to view practical technological goals in essentially religious terms.

* 避免“科幻”包袱。虽然我认为大多数人低估了强大 AI 的积极面，但少数讨论激进 AI 未来的人往往以过度“科幻”的语调进行（例如，涉及上传意识、太空探索或一般的赛博朋克氛围）。我认为这导致人们不太认真地对待这些说法，并给它们注入一种不真实感。需要明确的是，问题不在于所描述的技术是否可能或可行（主要文章对此进行了详细讨论）——更多的是这种“氛围”在内涵上偷偷带入了大量文化包袱和关于什么样的未来是可取的、各种社会问题将如何发展等的未明说假设。结果往往读起来像是一个狭隘亚文化的幻想，同时让大多数人反感。

* Avoid “sci-fi” baggage. Although I think most people underestimate the upside of powerful AI, the small community of people who do discuss radical AI futures often does so in an excessively “sci-fi” tone (featuring e.g. uploaded minds, space exploration, or general cyberpunk vibes). I think this causes people to take the claims less seriously, and to imbue them with a sort of unreality. To be clear, the issue isn’t whether the technologies described are possible or likely (the main essay discusses this in granular detail)—it’s more that the “vibe” connotatively smuggles in a bunch of cultural baggage and unstated assumptions about what kind of future is desirable, how various societal issues will play out, etc. The result often ends up reading like a fantasy for a narrow subculture, while being off-putting to most people.

然而，尽管有上述所有担忧，我确实认为讨论一个拥有强大 AI 的美好世界可能是什么样子很重要，同时尽力避免上述陷阱。事实上，我认为拥有一个真正鼓舞人心的未来愿景至关重要，而不仅仅是制定一个灭火计划。强大 AI 的许多影响是对抗性的或危险的，但归根结底，我们必须有为之奋斗的东西，一些让每个人都受益的正和结果，一些让人们超越争吵、迎接未来挑战的东西。恐惧是一种动力，但还不够：我们还需要希望。

Yet despite all of the concerns above, I really do think it’s important to discuss what a good world with powerful AI could look like, while doing our best to avoid the above pitfalls. In fact I think it is critical to have a genuinely inspiring vision of the future, and not _just_ a plan to fight fires. Many of the implications of powerful AI are adversarial or dangerous, but at the end of it all, there has to be something we’re fighting _for_, some positive-sum outcome where everyone is better off, something to rally people to rise above their squabbles and confront the challenges ahead. Fear is one kind of motivator, but it’s not enough: we need hope as well.

强大 AI 的积极应用列表非常长（包括机器人、制造、能源等），但我将专注于少数几个在我看来最有潜力直接改善人类生活质量的领域。我最兴奋的五个类别是：

The list of positive applications of powerful AI is extremely long (and includes robotics, manufacturing, energy, and much more), but I’m going to focus on a small number of areas that seem to me to have the greatest potential to directly improve the quality of human life. The five categories I am most excited about are:

我的预测按照大多数标准（除了科幻“奇点”愿景 2

My predictions are going to be radical as judged by most standards (other than sci-fi “singularity” visions2

2 我预计少数人的反应会是“这相当温和”。我认为这些人需要，用推特的话说，“接触现实”。但更重要的是，从社会角度来看，温和是好的。我认为人们一次只能承受有限的变化，我所描述的速度可能接近社会能够吸收而不产生极端动荡的极限。

2 I do anticipate some minority of people’s reaction will be “this is pretty tame”. I think those people need to, in Twitter parlance, “touch grass”. But more importantly, tame is good from a societal perspective. I think there’s only so much change people can handle at once, and the pace I’m describing is probably close to the limits of what society can absorb without extreme turbulence.

）来看将是激进的，但我是认真而真诚地表达它们。我所说的一切都很容易出错（重复我上面的观点），但我至少试图将我的观点基于对各个领域进步可能加速多少以及这在实践中意味着什么的半分析性评估。我很幸运在生物学和神经科学方面有专业经验，并且是经济发展领域的有见识的业余爱好者，但我确信我会在很多事情上犯错。写这篇文章让我意识到的一件事是，召集一组领域专家（生物学、经济学、国际关系等领域）来撰写一个比我这里所写的更好、更有依据的版本将是有价值的。最好将我这里的努力视为对该小组的起始提示。

), but I mean them earnestly and sincerely. Everything I’m saying could very easily be wrong (to repeat my point from above), but I’ve at least attempted to ground my views in a semi-analytical assessment of how much progress in various fields might speed up and what that might mean in practice. I am fortunate to have professional experience in both biology and neuroscience, and I am an informed amateur in the field of economic development, but I am sure I will get plenty of things wrong. One thing writing this essay has made me realize is that it would be valuable to bring together a group of domain experts (in biology, economics, international relations, and other areas) to write a much better and more informed version of what I’ve produced here. It’s probably best to view my efforts here as a starting prompt for that group.

基本假设与框架 Basic assumptions and framework

为了使整篇文章更加精确和扎实，明确我们所说的强大 AI（即 5-10 年倒计时开始的阈值）的含义，并为此类 AI 出现后的影响提供一个思考框架，是很有帮助的。

To make this whole essay more precise and grounded, it’s helpful to specify clearly what we mean by powerful AI (i.e. the threshold at which the 5-10 year clock starts counting), as well as laying out a framework for thinking about the effects of such AI once it’s present.

强大的 AI（我不喜欢 AGI 这个术语）³

What powerful AI (I dislike the term AGI)3

³ 我发现 AGI 是一个不精确的术语，它积累了很多科幻包袱和炒作。我更喜欢“强大 AI”或“专家级科学与工程”，这些术语没有炒作，更能表达我的意思。

3 I find AGI to be an imprecise term that has gathered a lot of sci-fi baggage and hype. I prefer "powerful AI" or "Expert-Level Science and Engineering" which get at what I mean without the hype.

会是什么样子，以及何时（或是否）会到来，本身就是一个巨大的话题。这是我公开讨论过的话题，我可能会专门写一篇独立的文章（也许将来会）。显然，很多人怀疑强大 AI 会很快被构建出来，有些人甚至怀疑它永远不会被构建出来。我认为它最早可能在 2026 年出现，但也可能需要更长时间。但为了本文的目的，我想把这些争议放在一边，假设它会在相当快的时间内到来，并专注于那之后的 5-10 年会发生什么。我还想假设这样一个系统会是什么样子，它的能力以及它如何交互，尽管在这方面存在分歧。

will look like, and when (or if) it will arrive, is a huge topic in itself. It’s one I’ve discussed publicly and could write a completely separate essay on (I probably will at some point). Obviously, many people are skeptical that powerful AI will be built soon and some are skeptical that it will ever be built at all. I think it could come as early as 2026, though there are also ways it could take much longer. But for the purposes of this essay, I’d like to put these issues aside, assume it will come reasonably soon, and focus on what happens in the 5-10 years after that. I also want to assume a definition of what such a system _will look like,_ what its capabilities are and how it interacts, even though there is room for disagreement on this.

所谓强大 AI，我指的是一种 AI 模型——可能在形式上类似于今天的 LLM，尽管它可能基于不同的架构，可能涉及多个交互模型，并且可能以不同的方式训练——具有以下特性：

By _powerful AI_, I have in mind an AI model—likely similar to today’s LLMs in form, though it might be based on a different architecture, might involve several interacting models, and might be trained differently—with the following properties:

* 在纯智能⁴方面，它在大多数相关领域——生物学、编程、数学、工程、写作等——比诺贝尔奖得主更聪明。这意味着它可以证明未解决的数学定理，写出极其优秀的小说，从头编写困难的代码库等。

* In terms of pure intelligence44 In this essay, I use "intelligence" to refer to a general problem-solving capability that can be applied across diverse domains. This includes abilities like reasoning, learning, planning, and creativity. While I use "intelligence" as a shorthand throughout this essay, I acknowledge that the nature of intelligence is a complex and debated topic in cognitive science and AI research. Some researchers argue that intelligence isn't a single, unified concept but rather a collection of separate cognitive abilities. Others contend that there's a general factor of intelligence (g factor) underlying various cognitive skills. That’s a debate for another time. , it is smarter than a Nobel Prize winner across most relevant fields – biology, programming, math, engineering, writing, etc. This means it can prove unsolved mathematical theorems, write extremely good novels, write difficult codebases from scratch, etc.

* 除了只是一个“你可以与之交谈的聪明东西”之外，它拥有虚拟工作的人类可用的所有“接口”，包括文本、音频、视频、鼠标和键盘控制以及互联网访问。它可以参与通过此接口实现的任何行动、通信或远程操作，包括在互联网上采取行动、向人类下达或接受指令、订购材料、指导实验、观看视频、制作视频等。它完成所有这些任务的能力再次超过了世界上最能干的人类。

* In addition to just being a “smart thing you talk to”, it has all the “interfaces” available to a human working virtually, including text, audio, video, mouse and keyboard control, and internet access. It can engage in any actions, communications, or remote operations enabled by this interface, including taking actions on the internet, taking or giving directions to humans, ordering materials, directing experiments, watching videos, making videos, and so on. It does all of these tasks with, again, a skill exceeding that of the most capable humans in the world.

* 它不仅仅被动回答问题；相反，它可以被赋予需要数小时、数天或数周才能完成的任务，然后自主地去完成这些任务，就像聪明的员工一样，必要时会要求澄清。

* It does not just passively answer questions; instead, it can be given tasks that take hours, days, or weeks to complete, and then goes off and does those tasks autonomously, in the way a smart employee would, asking for clarification as necessary.

* 它没有物理实体（除了存在于计算机屏幕上），但可以通过计算机控制现有的物理工具、机器人或实验室设备；理论上，它甚至可以为自己设计机器人或设备来使用。

* It does not have a physical embodiment (other than living on a computer screen), but it can control existing physical tools, robots, or laboratory equipment through a computer; in theory it could even design robots or equipment for itself to use.

* 用于训练模型的资源可以重新用于运行数百万个实例（这与约 2027 年的预期集群规模相匹配），并且模型可以以大约人类速度的 10-100 倍吸收信息和生成行动⁵。然而，它可能受到物理世界或与之交互的软件响应时间的限制。

* The resources used to train the model can be repurposed to _run_ millions of instances of it (this matches projected cluster sizes by ~2027), and the model can absorb information and generate actions at roughly 10x-100x human speed55 This is roughly the current speed of AI systems – for example they can read a page of text in a couple seconds and write a page of text in maybe 20 seconds, which is 10-100x the speed at which humans can do these things. Over time larger models tend to make this slower but more powerful chips tend to make it faster; to date the two effects have roughly canceled out. . It may however be limited by the response time of the physical world or of software it interacts with.

* 这数百万个副本中的每一个都可以独立处理不相关的任务，或者如果需要，可以像人类协作一样一起工作，也许不同的子群体经过微调，特别擅长特定任务。

* Each of these million copies can act independently on unrelated tasks, or if needed can all work together in the same way humans would collaborate, perhaps with different subpopulations fine-tuned to be especially good at particular tasks.

我们可以将其概括为“数据中心里的天才之国”。

We could summarize this as a “country of geniuses in a datacenter”.

显然，这样的实体将能够非常快速地解决非常困难的问题，但要弄清楚有多快并非易事。两种“极端”立场在我看来似乎都是错误的。首先，你可能认为世界会在几秒或几天内瞬间转变（“奇点”），因为超级智能会自我迭代，几乎立即解决所有可能的科学、工程和操作任务。问题在于，存在真实的物理和实际限制，例如围绕硬件构建或进行生物实验。即使是一个新的天才之国也会遇到这些限制。智能可能非常强大，但它不是魔法仙尘。

Clearly such an entity would be capable of solving very difficult problems, very fast, but it is not trivial to figure out how fast. Two “extreme” positions both seem false to me. First, you might think that the world would be instantly transformed on the scale of seconds or days (“the Singularity”), as superior intelligence builds on itself and solves every possible scientific, engineering, and operational task almost immediately. The problem with this is that there are real physical and practical limits, for example around building hardware or conducting biological experiments. Even a new country of geniuses would hit up against these limits. Intelligence may be very powerful, but it isn’t magic fairy dust.

其次，相反地，你可能认为技术进步已经饱和，或者受到现实世界数据或社会因素的速率限制，而超越人类的智能几乎不会增加什么⁶。

Second, and conversely, you might believe that technological progress is saturated or rate-limited by real world data or by social factors, and that better-than-human intelligence will add very little6

⁶ 这可能看起来像一个稻草人立场，但像 Tyler Cowen 和 Matt Yglesias 这样的谨慎思想家已经将其作为一个严肃的问题提出（尽管我认为他们并不完全持有这种观点），而且我认为这并非疯狂。

6 This might seem like a strawman position, but careful thinkers like Tyler Cowen and Matt Yglesias have raised it as a serious concern (though I don’t think they fully hold the view), and I don’t think it is crazy.

这在我看来同样不可信——我可以想到数百个科学甚至社会问题，一大群非常聪明的人会极大地加速进展，特别是如果他们不局限于分析，而是能够在现实世界中有所作为（我们假设的天才之国可以做到，包括通过指导或协助人类团队）。

. This seems equally implausible to me—I can think of hundreds of scientific or even social problems where a large group of really smart people would drastically speed up progress, especially if they aren’t limited to analysis and can make things happen in the real world (which our postulated country of geniuses can, including by directing or assisting teams of humans).

我认为真相可能是这两种极端图景的某种混乱混合，因任务和领域而异，并且在细节上非常微妙。我相信我们需要新的框架来以富有成效的方式思考这些细节。

I think the truth is likely to be some messy admixture of these two extreme pictures, something that varies by task and field and is very subtle in its details. I believe we need new frameworks to think about these details in a productive way.

经济学家经常谈论“生产要素”：如劳动力、土地和资本。“劳动力/土地/资本的边际回报”这个短语捕捉了这样一种思想：在特定情况下，某个特定因素可能是或可能不是限制因素——例如，空军既需要飞机也需要飞行员，如果你没有飞机了，雇佣更多飞行员也无济于事。我相信在 AI 时代，我们应该谈论智能的边际回报⁷，并试图找出与智能互补的其他因素，这些因素在智能非常高时成为限制因素。我们不习惯这样思考——不习惯问“更聪明对这项任务有多大帮助，以及在什么时间尺度上？”——但这似乎是概念化一个拥有非常强大 AI 的世界的正确方式。

Economists often talk about “factors of production”: things like labor, land, and capital. The phrase “marginal returns to labor/land/capital” captures the idea that in a given situation, a given factor may or may not be the limiting one – for example, an air force needs both planes and pilots, and hiring more pilots doesn’t help much if you’re out of planes. I believe that in the AI age, we should be talking about _the marginal returns to intelligence_7

⁷ 我所知道的与这个问题最接近的经济学工作是关于“通用目的技术”和作为通用目的技术补充的“无形投资”的研究。

7 The closest economics work that I’m aware of to tackling this question is work on “general purpose technologies” and “intangible investments” that serve as complements to general purpose technologies.

我猜测的限制或与智能互补的因素列表包括：

, and trying to figure out what the other factors are that are complementary to intelligence and that become limiting factors when intelligence is very high. We are not used to thinking in this way—to asking “how much does being smarter help with this task, and on what timescale?”—but it seems like the right way to conceptualize a world with very powerful AI.

* 外部世界的速度。智能体需要与世界交互操作才能完成任务，同时也为了学习⁸。但世界只能以一定的速度运行。细胞和动物以固定速度运行，因此对它们进行的实验需要一定的时间，这可能是不可减少的。硬件、材料科学、任何涉及与人沟通的事情，甚至我们现有的软件基础设施也是如此。此外，在科学中，许多实验通常需要按顺序进行，每个实验都从上一个实验中学习或建立在其基础上。所有这些意味着，一个重大项目——例如开发癌症疗法——可以完成的速度可能有一个不可减少的最小值，即使智能继续增加，这个最小值也无法进一步降低。

My guess at a list of factors that limit or are complementary to intelligence includes:

* 对数据的需求。有时原始数据缺乏，没有数据，更多的智能也无济于事。今天的粒子物理学家非常聪明，并发展出了广泛的理论，但由于粒子加速器数据非常有限，缺乏数据来在它们之间做出选择。如果他们具有超级智能，他们是否会做得更好并不清楚——除了可能通过加速建造更大的加速器。

* Speed of the outside world. Intelligent agents need to operate interactively in the world in order to accomplish things and also to learn88 This learning can include temporary, in-context learning, or traditional training; both will be rate-limited by the physical world. . But the world only moves so fast. Cells and animals run at a fixed speed so experiments on them take a certain amount of time which may be irreducible. The same is true of hardware, materials science, anything involving communicating with people, and even our existing software infrastructure. Furthermore, in science many experiments are often needed in sequence, each learning from or building on the last. All of this means that the speed at which a major project—for example developing a cancer cure—can be completed may have an irreducible minimum that cannot be decreased further even as intelligence continues to increase.

* 内在复杂性。有些事情本质上是不可预测或混沌的，即使是最强大的 AI 也无法比今天的人类或计算机更好地预测或解开它们。例如，即使是非常强大的 AI，在一般情况下，也只能比今天的人类和计算机稍微提前一点预测混沌系统（如三体问题）⁹。

* Need for data. Sometimes raw data is lacking and in its absence more intelligence does not help. Today’s particle physicists are very ingenious and have developed a wide range of theories, but lack the data to choose between them because particle accelerator data is so limited. It is not clear that they would do drastically better if they were superintelligent—other than perhaps by speeding up the construction of a bigger accelerator.

* 来自人类的约束。许多事情不能在不违反法律、伤害人类或扰乱社会的情况下完成。对齐的 AI 不会想做这些事情（如果我们有一个未对齐的 AI，我们又回到了谈论风险）。许多人类社会结构效率低下甚至有害，但在尊重诸如临床试验的法律要求、人们改变习惯的意愿或政府行为等约束的情况下，很难改变。在技术意义上运行良好，但其影响因法规或不当恐惧而大大降低的进展例子包括核能、超音速飞行，甚至电梯。

* Intrinsic complexity. Some things are inherently unpredictable or chaotic and even the most powerful AI cannot predict or untangle them substantially better than a human or a computer today. For example, even incredibly powerful AI could predict only marginally further ahead in a chaotic system (such as the three-body problem) in the general case,99 In a chaotic system, small errors compound exponentially over time, so that even an enormous increase in computing power leads to only a small improvement in how far ahead it is possible to predict, and in practice measurement error may degrade this further. as compared to today’s humans and computers.

* 物理定律。这是第一点的更严格版本。有些物理定律似乎是不可打破的。不可能比光速更快。布丁不会自动搅匀。芯片每平方厘米只能有那么多晶体管，否则就会变得不可靠。计算需要每擦除一位信息消耗一定的最小能量，限制了世界上的计算密度。

* Constraints from humans. Many things cannot be done without breaking laws, harming humans, or messing up society. An aligned AI would not want to do these things (and if we have an unaligned AI, we’re back to talking about risks). Many human societal structures are inefficient or even actively harmful, but are hard to change while respecting constraints like legal requirements on clinical trials, people’s willingness to change their habits, or the behavior of governments. Examples of advances that work well in a technical sense, but whose impact has been substantially reduced by regulations or misplaced fears, include nuclear power, supersonic flight, and even elevators.

还有一个基于时间尺度的进一步区分。短期内是硬约束的东西，长期来看可能变得更容易被智能改变。例如，智能可能被用来开发一种新的实验范式，使我们能够在体外学习以前需要活体动物实验的东西，或者构建收集新数据所需的工具（例如更大的粒子加速器），或者（在伦理限制内）找到绕过人类约束的方法（例如帮助改进临床试验系统，帮助创建临床试验官僚机构较少的新司法管辖区，或者改进科学本身，使人体临床试验变得不那么必要或更便宜）。

* Physical laws. This is a starker version of the first point. There are certain physical laws that appear to be unbreakable. It’s not possible to travel faster than light. Pudding does not unstir). Chips can only have so many transistors per square centimeter before they become unreliable. Computation requires a certain minimum energy per bit erased, limiting the density of computation in the world.

因此，我们应该想象这样一幅图景：智能最初受到其他生产要素的严重瓶颈，但随着时间的推移，智能本身越来越多地绕过其他因素，即使它们从未完全消失（有些东西如物理定律是绝对的）¹⁰。关键问题是这一切发生的速度有多快以及以什么顺序发生。

There is a further distinction based on _timescales_. Things that are hard constraints in the short run may become more malleable to intelligence in the long run. For example, intelligence might be used to develop a new experimental paradigm that allows us to learn _in vitro_ what used to require live animal experiments, or to build the tools needed to collect new data (e.g. the bigger particle accelerator), or to (within ethical limits) find ways around human-based constraints (e.g. helping to improve the clinical trial system, helping to create new jurisdictions where clinical trials have less bureaucracy, or improving the science itself to make human clinical trials less necessary or cheaper).

¹⁰ 另一个因素当然是，强大 AI 本身可能被用来创造更强大的 AI。我的假设是，这可能会发生（事实上，很可能发生），但其影响会比你可能想象的要小，正是由于这里讨论的“智能的边际回报递减”。换句话说，AI 将继续快速变得更聪明，但其效果最终将受到非智能因素的限制，分析这些因素对于 AI 之外的科学进步速度最为重要。

Thus, we should imagine a picture where intelligence is initially heavily bottlenecked by the other factors of production, but over time intelligence itself increasingly routes around the other factors, even if they never fully dissolve (and some things like physical laws are absolute)10

有了上述框架，我将尝试回答引言中提到的五个领域的问题。

10 Another factor is of course that powerful AI itself can potentially be used to create even more powerful AI. My assumption is that this might (in fact, probably will) occur, but that its effect will be smaller than you might imagine, precisely because of the “decreasing marginal returns to intelligence” discussed here. In other words, AI will continue to get smarter quickly, but its effect will eventually be limited by non-intelligence factors, and analyzing those is what matters most to the speed of scientific progress outside AI.

. The key question is how fast it all happens and in what order.

With the above framework in mind, I’ll try to answer that question for the five areas mentioned in the introduction.

1. 生物学与健康 1. Biology and health

生物学可能是科学进步最有可能直接且明确地改善人类生活质量的领域。在上个世纪，一些最古老的人类疾病（如天花）终于被征服，但还有许多疾病依然存在，战胜它们将是巨大的人道主义成就。除了治愈疾病，生物学科学原则上还可以通过延长健康人类寿命、增强对我们自身生物过程的控制与自由，以及解决我们目前视为人类状况中不可改变部分的日常问题，来改善人类健康的基线水平。

Biology is probably the area where scientific progress has the greatest potential to directly and unambiguously improve the quality of human life. In the last century some of the most ancient human afflictions (such as smallpox) have finally been vanquished, but many more still remain, and defeating them would be an enormous humanitarian accomplishment. Beyond even curing disease, biological science can in principle improve the _baseline_ quality of human health, by extending the healthy human lifespan, increasing control and freedom over our own biological processes, and addressing everyday problems that we currently think of as immutable parts of the human condition.

用上一节中“限制因素”的语言来说，将智能直接应用于生物学的主要挑战是数据、物理世界的速度以及内在复杂性（事实上，这三者相互关联）。在后期阶段，当涉及临床试验时，人类约束也发挥作用。让我们逐一分析这些因素。

In the “limiting factors” language of the previous section, the main challenges with directly applying intelligence to biology are data, the speed of the physical world, and intrinsic complexity (in fact, all three are related to each other). Human constraints also play a role at a later stage, when clinical trials are involved. Let’s take these one by one.

对细胞、动物甚至化学过程的实验受到物理世界速度的限制：许多生物学方案涉及培养细菌或其他细胞，或者仅仅等待化学反应发生，这有时可能需要数天甚至数周，且没有明显的加速方法。动物实验可能需要数月（或更长时间），而人体实验通常需要数年（甚至数十年用于长期结果研究）。与此相关的是，数据常常缺乏——不是数量不足，而是质量不高：总是缺乏清晰、明确的数据，能够将感兴趣的生物学效应与其他一万个混杂因素隔离开来，或者对给定过程进行因果干预，或者直接测量某种效应（而不是通过某种间接或嘈杂的方式推断其后果）。即使是大规模的定量分子数据，比如我在研究质谱技术时收集的蛋白质组学数据，也是嘈杂且遗漏很多的（这些蛋白质存在于哪种细胞中？细胞的哪个部分？细胞周期的哪个阶段？）。

Experiments on cells, animals, and even chemical processes are limited by the speed of the physical world: many biological protocols involve culturing bacteria or other cells, or simply waiting for chemical reactions to occur, and this can sometimes take days or even weeks, with no obvious way to speed it up. Animal experiments can take months (or more) and human experiments often take years (or even decades for long-term outcome studies). Somewhat related to this, data is often lacking—not so much in quantity, but quality: there is always a dearth of clear, unambiguous data that isolates a biological effect of interest from the other 10,000 confounding things that are going on, or that intervenes causally in a given process, or that directly measures some effect (as opposed to inferring its consequences in some indirect or noisy way). Even massive, quantitative molecular data, like the proteomics data that I collected while working on mass spectrometry techniques, is noisy and misses a lot (which types of cells were these proteins in? Which part of the cell? At what phase in the cell cycle?).

这些数据问题部分归因于内在复杂性：如果你见过显示人类新陈代谢生物化学的图表，你就会知道很难隔离这个复杂系统中任何部分的影响，更难以精确或可预测的方式对系统进行干预。最后，除了进行人体实验所需的内在时间外，实际的临床试验还涉及大量的官僚主义和监管要求，这些（在我看来，以及许多其他人看来）增加了不必要的额外时间并阻碍了进展。

In part responsible for these problems with data is intrinsic complexity: if you’ve ever seen a diagram showing the biochemistry of human metabolism, you’ll know that it’s very hard to isolate the effect of any part of this complex system, and even harder to intervene on the system in a precise or predictable way. And finally, beyond just the intrinsic time that it takes to run an experiment on humans, actual clinical trials involve a lot of bureaucracy and regulatory requirements that (in the opinion of many people, including me) add unnecessary additional time and delay progress.

鉴于这一切，许多生物学家长期以来一直对人工智能以及更广泛的“大数据”在生物学中的价值持怀疑态度。历史上，过去 30 年中将技能应用于生物学的数学家、计算机科学家和物理学家取得了相当的成功，但并未产生最初期望的真正变革性影响。AlphaFold（它当之无愧地为其创造者赢得了诺贝尔化学奖）和 AlphaProteo 等重大革命性突破在一定程度上减少了这种怀疑。

Given all this, many biologists have long been skeptical of the value of AI and “big data” more generally in biology. Historically, mathematicians, computer scientists, and physicists who have applied their skills to biology over the last 30 years have been quite successful, but have not had the truly transformative impact initially hoped for. Some of the skepticism has been reduced by major and revolutionary breakthroughs like AlphaFold (which has just deservedly won its creators the Nobel Prize in Chemistry) and AlphaProteo11

这些成就对我而言是激励，也许也是人工智能用于变革生物学的最有力现有例证。

11 These achievements have been an inspiration to me and perhaps the most powerful existing example of AI being used to transform biology.

但仍然存在一种看法，认为人工智能只在有限的情况下有用（并将继续如此）。一种常见的说法是：“人工智能可以更好地分析你的数据，但它不能产生更多数据或提高数据质量。垃圾进，垃圾出。”

, but there’s still a perception that AI is (and will continue to be) useful in only a limited set of circumstances. A common formulation is “AI can do a better job analyzing your data, but it can’t produce more data or improve the quality of the data. Garbage in, garbage out”.

但我认为这种悲观的观点是以错误的方式看待人工智能。如果我们关于人工智能进展的核心假设是正确的，那么看待人工智能的正确方式不是将其作为一种数据分析方法，而是作为一个虚拟的生物学家，它执行生物学家所做的所有任务，包括设计和在现实世界中运行实验（通过控制实验室机器人或直接告诉人类运行哪些实验——就像首席研究员对其研究生所做的那样），发明新的生物学方法或测量技术，等等。正是通过加速整个研究过程，人工智能才能真正加速生物学的发展。我想重复这一点，因为这是我在谈论人工智能变革生物学能力时遇到的最常见误解：我并不是在谈论人工智能仅仅作为分析数据的工具。根据本文开头对强大人工智能的定义，我谈论的是使用人工智能来执行、指导和改进生物学家所做的几乎所有事情。

But I think that pessimistic perspective is thinking about AI in the wrong way. If our core hypothesis about AI progress is correct, then the right way to think of AI is not as a method of data analysis, but as a virtual biologist who performs _all_ the tasks biologists do, including designing and running experiments in the real world (by controlling lab robots or simply telling humans which experiments to run – as a Principal Investigator would to their graduate students), inventing new biological methods or measurement techniques, and so on. It is by speeding up _the whole research process_ that AI can truly accelerate biology. I want to repeat this because it’s the most common misconception that comes up when I talk about AI’s ability to transform biology: I am _not_ talking about AI as merely a tool to analyze data. In line with the definition of powerful AI at the beginning of this essay, I’m talking about using AI to perform, direct, and improve upon nearly everything biologists do.

为了更具体地说明我认为加速可能来自何处，生物学进展中相当大的一部分实际上来自极少数发现，这些发现通常与允许对生物系统进行精确但通用或可编程干预的广泛测量工具或技术有关。

To get more specific on where I think acceleration is likely to come from, a surprisingly large fraction of the progress in biology has come from a truly tiny number of discoveries, often related to broad measurement tools or techniques12

“科学进步依赖于新技术、新发现和新思想，可能按此顺序。”——悉尼·布伦纳

12 “Progress in science depends on new techniques, new discoveries and new ideas, probably in that order.” - Sydney Brenner

大约每年有 1 个这样的重大发现，它们共同推动了生物学进展的 50%以上。这些发现之所以如此强大，正是因为它们突破了内在复杂性和数据限制，直接增强了我们对生物过程的理解和控制。每十年有几个发现不仅使我们获得了对生物学的基本科学理解，还推动了许多最强大的医学治疗。

that allow precise but generalized or programmable intervention in biological systems. There’s perhaps ~1 of these major discoveries per year and collectively they arguably drive >50% of progress in biology. These discoveries are so powerful precisely because they cut through intrinsic complexity and data limitations, directly increasing our understanding and control over biological processes. A few discoveries per decade have enabled both the bulk of our basic scientific understanding of biology, and have driven many of the most powerful medical treatments.

* CRISPR：一种允许在活生物体中实时编辑任何基因的技术（将任意基因序列替换为任意其他序列）。自原始技术开发以来，不断有改进以靶向特定细胞类型、提高准确性并减少错误基因的编辑——所有这些对于在人类中安全使用都是必需的。

* CRISPR: a technique that allows live editing of any gene in living organisms (replacement of any arbitrary gene sequence with any other arbitrary sequence). Since the original technique was developed, there have been constant improvements to target specific cell types, increasing accuracy, and reducing edits of the wrong gene—all of which are needed for safe use in humans.

* 各种类型的显微镜，用于精确观察正在发生的事情：先进的光学显微镜（具有各种荧光技术、特殊光学器件等）、电子显微镜、原子力显微镜等。

* Various kinds of microscopy for watching what is going on at a precise level: advanced light microscopes (with various kinds of fluorescent techniques, special optics, etc), electron microscopes, atomic force microscopes, etc.

* 基因组测序和合成，其成本在过去几十年中下降了数个数量级。

* Genome sequencing and synthesis, which has dropped in cost by several orders of magnitude in the last couple decades.

* 光遗传学技术，允许通过光照使神经元放电。

* Optogenetic techniques that allow you to get a neuron to fire by shining a light on it.

* mRNA 疫苗，原则上允许我们设计针对任何事物的疫苗并快速调整（mRNA 疫苗当然在 COVID 期间变得著名）。

* mRNA vaccines that, in principle, allow us to design a vaccine against anything and then quickly adapt it (mRNA vaccines of course became famous during COVID).

* 细胞疗法，如 CAR-T，允许将免疫细胞从体内取出并“重新编程”以攻击任何事物。

* Cell therapies such as CAR-T that allow immune cells to be taken out of the body and “reprogrammed” to attack, in principle, anything.

* 概念性见解，如疾病的细菌理论或免疫系统与癌症之间联系的实现。感谢 Parag Mallick 提出这一点。

* Conceptual insights like the germ theory of disease or the realization of a link between the immune system and cancer1313 Thanks to Parag Mallick for suggesting this point. .

我费心列出所有这些技术，是因为我想对它们提出一个关键主张：我认为如果拥有更多有才华、有创造力的研究人员，它们的发现速度可以提高 10 倍或更多。或者换句话说，我认为这些发现的智能回报很高，而生物学和医学中的其他一切都主要源于它们。

I’m going to the trouble of listing all these technologies because I want to make a crucial claim about them: I think their rate of discovery could be increased by 10x or more if there were a lot more talented, creative researchers_._ Or, put another way, I think the returns to intelligence are high for these discoveries, and that everything else in biology and medicine mostly follows from them.

我为什么这么认为？因为当我们试图确定“智能回报”时，我们应该养成提出一些问题的习惯。首先，这些发现通常由极少数研究人员做出，通常是同一个人反复做出，这表明是技能而非随机搜索（后者可能表明漫长的实验是限制因素）。其次，它们通常“本可以”更早几年被做出：例如，CRISPR 是细菌免疫系统中一种自然存在的成分，自 20 世纪 80 年代以来就已知，但人们又花了 25 年才意识到它可以被重新用于通用基因编辑。它们也常常因科学界对有前途方向缺乏支持而延迟多年（参见关于 mRNA 疫苗发明者的简介；类似故事比比皆是）。第三，成功的项目通常是临时拼凑的，或者是人们最初认为没有前途的“事后想法”，而不是大规模资助的努力。这表明推动发现的不仅仅是大量资源集中，还有独创性。

Why do I think this? Because of the answers to some questions that we should get in the habit of asking when we’re trying to determine “returns to intelligence”. First, these discoveries are generally made by a tiny number of researchers, often the same people repeatedly, suggesting skill and not random search (the latter might suggest lengthy experiments are the limiting factor). Second, they often “could have been made” years earlier than they were: for example, CRISPR was a naturally occurring component of the immune system in bacteria that’s been known since the 1980s, but it took another 25 years for people to realize it could be repurposed for general gene editing. They also are often delayed many years by lack of support from the scientific community for promising directions (see this profile on the inventor of mRNA vaccines; similar stories abound). Third, successful projects are often scrappy or were afterthoughts that people didn’t initially think were promising, rather than massively funded efforts. This suggests that it’s not just massive resource concentration that drives discoveries, but ingenuity.

最后，尽管其中一些发现具有“序列依赖性”（你需要先做出发现 A 才能拥有做出发现 B 的工具或知识）——这同样可能造成实验延迟——但许多（也许是大多数）是独立的，意味着许多发现可以同时并行进行。这些事实以及我作为生物学家的普遍经验强烈表明，如果科学家更聪明，并且能够更好地连接人类拥有的庞大生物学知识（再次考虑 CRISPR 的例子），那么还有数百个这样的发现等待被做出。AlphaFold/AlphaProteo 在解决重要问题方面比人类更有效，尽管经过了数十年的精心设计的物理建模，这提供了一个原理证明（尽管是在狭窄领域中的狭窄工具），应该指明前进的方向。

Finally, although some of these discoveries have “serial dependence” (you need to make discovery A first in order to have the tools or knowledge to make discovery B)—which again might create experimental delays—many, perhaps most, are independent, meaning many at once can be worked on in parallel. Both these facts, and my general experience as a biologist, strongly suggest to me that there are hundreds of these discoveries waiting to be made if scientists were smarter and better at making connections between the vast amount of biological knowledge humanity possesses (again consider the CRISPR example). The success of AlphaFold/AlphaProteo at solving important problems much more effectively than humans, despite decades of carefully designed physics modeling, provides a proof of principle (albeit with a narrow tool in a narrow domain) that should point the way forward.

因此，我猜测强大的人工智能至少可以将这些发现的速率提高 10 倍，使我们在 5-10 年内获得未来 50-100 年的生物学进展。

Thus, it’s my guess that powerful AI could at least 10x the rate of these discoveries, giving us the next 50-100 years of biological progress in 5-10 years.14

我不想用关于 AI 驱动的科学可能做出哪些具体未来发现的猜测来堵塞正文，但这里有一些可能性的头脑风暴：

14 I didn't want to clog up the text with speculation about what specific future discoveries AI-enabled science could make, but here is a brainstorm of some possibilities:

— 设计更好的计算工具，如 AlphaFold 和 AlphaProteo——即一个通用 AI 系统加速我们制造专门的 AI 计算生物学工具的能力。

— Design of better computational tools like AlphaFold and AlphaProteo — that is, a general AI system speeding up our ability to make specialized AI computational biology tools.

— 材料科学和微型化突破，导致更好的植入设备。

— Materials science and miniaturization breakthroughs leading to better implanted devices.

— 对干细胞、细胞分化和去分化的更好控制，以及由此产生的再生或重塑组织的能力。

— Better control over stem cells, cell differentiation, and de-differentiation, and a resulting ability to regrow or reshape tissue.

— 对免疫系统的更好控制：选择性激活以应对癌症和传染病，选择性关闭以应对自身免疫疾病。

— Better control over the immune system: turning it on selectively to address cancer and infectious disease, and turning it off selectively to address autoimmune diseases.

为什么不是 100 倍？也许这是可能的，但在这里，序列依赖性和实验时间都变得重要：在 1 年内取得 100 年的进展需要很多事情第一次就做对，包括动物实验以及设计显微镜或昂贵的实验室设施之类的事情。我实际上对（可能听起来荒谬的）想法持开放态度，即我们可以在 5-10 年内取得 1000 年的进展，但我非常怀疑我们能否在 1 年内取得 100 年的进展。另一种说法是，我认为存在一个不可避免的恒定延迟：实验和硬件设计具有一定的“延迟”，并且需要迭代一定“不可减少”的次数，以学习无法逻辑推导的东西。但在此基础上可能实现大规模并行。

Why not 100x? Perhaps it is possible, but here both serial dependence and experiment times become important: getting 100 years of progress in 1 year requires a lot of things to go right the first time, including animal experiments and things like designing microscopes or expensive lab facilities. I’m actually open to the (perhaps absurd-sounding) idea that we could get _1000_ years of progress in 5-10 years, but very skeptical that we can get 100 years in 1 year. Another way to put it is I think there’s an unavoidable constant delay: experiments and hardware design have a certain “latency” and need to be iterated upon a certain “irreducible” number of times in order to learn things that can’t be deduced logically. But massive parallelism may be possible on top of that15

AI 当然也可能有助于更聪明地选择运行哪些实验：改进实验设计，从第一轮实验中学习更多，以便第二轮可以缩小关键问题范围，等等。

15 AI may of course also help with being smarter about choosing what experiments to run: improving experimental design, learning more from a first round of experiments so that the second round can narrow in on key questions, and so on.

临床试验呢？尽管存在大量官僚主义和放缓，但事实是，它们的大部分（尽管绝非全部！）缓慢最终源于需要严格评估几乎不起作用或作用模糊的药物。可悲的是，今天大多数疗法都是如此：平均癌症药物将生存期延长几个月，同时具有需要仔细测量的显著副作用（阿尔茨海默病药物也有类似情况）。这导致大规模研究（以获得统计功效）和困难的权衡，监管机构通常不擅长做出这些权衡，同样是由于官僚主义和竞争利益的复杂性。

What about clinical trials? Although there is a lot of bureaucracy and slowdown associated with them, the truth is that a lot (though by no means all!) of their slowness ultimately derives from the need to rigorously evaluate drugs that barely work or ambiguously work. This is sadly true of most therapies today: the average cancer drug increases survival by a few months while having significant side effects that need to be carefully measured (there’s a similar story for Alzheimer’s drugs). This leads to huge studies (in order to achieve statistical power) and difficult tradeoffs which regulatory agencies generally aren’t great at making, again because of bureaucracy and the complexity of competing interests.

当某样东西效果非常好时，它会快得多：存在加速批准通道，并且当效应量更大时，批准更容易。用于 COVID 的 mRNA 疫苗在 9 个月内获得批准——比通常速度快得多。尽管如此，即使在这些条件下，临床试验仍然太慢——mRNA 疫苗可以说应该在约 2 个月内获得批准。但这些类型的延迟（药物从端到端约 1 年）结合大规模并行化和需要一些但不过多的迭代（“几次尝试”），与 5-10 年内的根本性变革非常兼容。更乐观的是，AI 驱动的生物科学可能通过开发更好的动物和细胞实验模型（甚至模拟）来减少临床试验中迭代的需求，这些模型在预测人类会发生什么方面更准确。这对于开发针对衰老过程的药物尤其重要，衰老过程持续数十年，我们需要更快的迭代循环。

When something works really well, it goes much faster: there’s an accelerated approval track and the ease of approval is much greater when effect sizes are larger. mRNA vaccines for COVID were approved in 9 months—much faster than the usual pace. That said, even under these conditions clinical trials are still too slow—mRNA vaccines arguably _should_ have been approved in ~2 months. But these kinds of delays (~1 year end-to-end for a drug) combined with massive parallelization and the need for some but not too much iteration (“a few tries”) are very compatible with radical transformation in 5-10 years. Even more optimistically, it is possible that AI-enabled biological science will reduce the need for iteration in clinical trials by developing better animal and cell experimental models (or even simulations) that are more accurate in predicting what will happen in humans. This will be particularly important in developing drugs against the aging process, which plays out over decades and where we need a faster iteration loop.

最后，关于临床试验和社会障碍的话题，值得明确指出的是，生物医学创新在某些方面具有异常强大的成功部署记录，这与一些其他技术形成对比。

Finally, on the topic of clinical trials and societal barriers, it is worth pointing out explicitly that in some ways biomedical innovations have an unusually _strong_ track record of being successfully deployed, in contrast to some other technologies16

感谢 Matthew Yglesias 提出这一点。

16 Thanks to Matthew Yglesias for suggesting this point.

正如引言中提到的，许多技术尽管在技术上运行良好，但受到社会因素的阻碍。这可能表明对 AI 能实现什么持悲观态度。但生物医学是独特的，尽管开发药物的过程过于繁琐，但一旦开发出来，它们通常会被成功部署和使用。

. As mentioned in the introduction, many technologies are hampered by societal factors despite working well technically. This might suggest a pessimistic perspective on what AI can accomplish _._ But biomedicine is unique in that although the process of developing drugs is overly cumbersome, once developed they generally are successfully deployed and used.

总结以上内容，我的基本预测是，AI 驱动的生物学和医学将使我们能够将人类生物学家在未来 50-100 年内取得的进展压缩到 5-10 年内。我将其称为“压缩的 21 世纪”：即在开发出强大 AI 后，我们将在几年内取得我们在整个 21 世纪本应取得的生物学和医学进展。

To summarize the above, my basic prediction is that AI-enabled biology and medicine will allow us to compress the progress that human biologists would have achieved over the next 50-100 years into 5-10 years. I’ll refer to this as the “compressed 21st century”: the idea that after powerful AI is developed, we will in a few years make all the progress in biology and medicine that we would have made in the whole 21st century.

尽管预测强大 AI 在几年内能做什么仍然固有地困难和推测性，但问“人类在无辅助的情况下在未来 100 年内能做什么”具有一定具体性。简单地看看我们在 20 世纪取得的成就，或从 21 世纪头 20 年进行外推，或问“10 个 CRISPR 和 50 个 CAR-T”会带来什么，所有这些都提供了实用、接地气的方法来估计我们可能期望从强大 AI 获得的总体进展水平。

Although predicting what powerful AI can do in a few years remains inherently difficult and speculative, there is some concreteness to asking “what could humans do unaided in the next 100 years?”. Simply looking at what we’ve accomplished in the 20th century, or extrapolating from the first 2 decades of the 21st, or asking what “10 CRISPRs and 50 CAR-Ts” would get us, all offer practical, grounded ways to estimate the general level of progress we might expect from powerful AI.

下面我尝试列出我们可能期望的内容。这不是基于任何严格的方法论，并且在细节上几乎肯定会被证明是错误的，但它试图传达我们应该期望的根本性变革的总体水平：

Below I try to make a list of what we might expect. This is not based on any rigorous methodology, and will almost certainly prove wrong in the details, but it’s trying to get across the general _level_ of radicalism we should expect:

* 可靠地预防和治疗几乎所有自然传染病。快速进化的疾病，如多重耐药菌株，它们基本上利用医院作为进化实验室，不断提高对治疗的抵抗力，可能特别难以对付，并且可能是阻止我们达到 100%的那种事情。鉴于 20 世纪在传染病方面取得的巨大进展，想象我们在一个压缩的 21 世纪中或多或少“完成这项工作”并不激进。mRNA 疫苗和类似技术已经为“针对任何事物的疫苗”指明了道路。传染病是否从世界上完全根除（而不仅仅是在某些地方）取决于贫困和不平等问题，这些将在第 3 节中讨论。

* Reliable prevention and treatment of nearly all1717 Fast evolving diseases, like the multidrug resistant strains that essentially use hospitals as an evolutionary laboratory to continually improve their resistance to treatment, could be especially stubborn to deal with, and could be the kind of thing that prevents us from getting to 100%. natural infectious disease. Given the enormous advances against infectious disease in the 20th century, it is not radical to imagine that we could more or less “finish the job” in a compressed 21st. mRNA vaccines and similar technology already point the way towards “vaccines for anything”. Whether infectious disease is _fully eradicated from the world_ (as opposed to just in some places) depends on questions about poverty and inequality, which are discussed in Section 3.

* 消除大多数癌症。过去几十年中，癌症死亡率每年下降约 2%；因此，按照当前人类科学的速度，我们有望在 21 世纪消除大多数癌症。一些亚型已经基本治愈（例如某些类型的白血病通过 CAR-T 疗法），我甚至可能对非常选择性的药物更兴奋，这些药物在癌症早期就靶向它并防止其生长。AI 还将使非常精细地适应癌症个体化基因组的治疗方案成为可能——这些在今天也是可能的，但在时间和人类专业知识方面极其昂贵，而 AI 应该使我们能够扩展这些方案。死亡率和发病率降低 95%或更多似乎是可能的。尽管如此，癌症极其多样且适应性强，可能是最难完全摧毁的疾病。如果一些罕见的、难治的恶性肿瘤持续存在，那也不足为奇。

* Elimination of most cancer. Death rates from cancer have been dropping ~2% per year for the last few decades; thus we are on track to eliminate most cancer in the 21st century at the current pace of human science. Some subtypes have already been largely cured (for example some types of leukemia with CAR-T therapy), and I’m perhaps even more excited for very selective drugs that target cancer in its infancy and prevent it from ever growing. AI will also make possible treatment regimens very finely adapted to the individualized genome of the cancer—these are possible today, but hugely expensive in time and human expertise, which AI should allow us to scale. Reductions of 95% or more in both mortality and incidence seem possible. That said, cancer is extremely varied and adaptive, and is likely the hardest of these diseases to fully destroy. It would not be surprising if an assortment of rare, difficult malignancies persists.

* 非常有效的预防和有效的遗传病治愈。大大改进的胚胎筛查可能使预防大多数遗传病成为可能，而更安全、更可靠的 CRISPR 后代可能治愈现有患者中的大多数遗传病。然而，影响大部分细胞的全身性疾病可能是最后的顽固堡垒。

* Very effective prevention and effective cures for genetic disease. Greatly improved embryo screening will likely make it possible to prevent most genetic disease, and some safer, more reliable descendant of CRISPR may cure most genetic disease in existing people. Whole-body afflictions that affect a large fraction of cells may be the last holdouts, however.

* 预防阿尔茨海默病。我们一直很难弄清楚阿尔茨海默病的病因（它与β-淀粉样蛋白有关，但实际细节似乎非常复杂）。这似乎正是那种可以通过更好的测量工具来隔离生物学效应的问题；因此我看好 AI 解决它的能力。一旦我们真正理解了正在发生的事情，很有可能最终可以通过相对简单的干预来预防它。尽管如此，已经存在的阿尔茨海默病造成的损害可能非常难以逆转。

* Prevention of Alzheimer’s. We’ve had a very hard time figuring out what causes Alzheimer’s (it is somehow related to beta-amyloid protein, but the actual details seem to be very complex). It seems like exactly the type of problem that can be solved with better measurement tools that isolate biological effects; thus I am bullish about AI’s ability to solve it. There is a good chance it can eventually be prevented with relatively simple interventions, once we actually understand what is going on. That said, damage from already existing Alzheimer’s may be very difficult to reverse.

* 改善大多数其他疾病的治疗。这是一个包含其他疾病的类别，包括糖尿病、肥胖症、心脏病、自身免疫疾病等。其中大多数似乎比癌症和阿尔茨海默病“更容易”解决，并且在许多情况下已经大幅下降。例如，心脏病死亡率已经下降超过 50%，像 GLP-1 激动剂这样的简单干预已经在肥胖症和糖尿病方面取得了巨大进展。

* Improved treatment of most other ailments. This is a catch-all category for other ailments including diabetes, obesity, heart disease, autoimmune diseases, and more. Most of these seem “easier” to solve than cancer and Alzheimer’s and in many cases are already in steep decline. For example, deaths from heart disease have already declined over 50%, and simple interventions like GLP-1 agonists have already made huge progress against obesity and diabetes.

* 生物自由。过去 70 年见证了避孕、生育、体重管理等方面的进步。但我怀疑 AI 加速的生物学将大大扩展可能性：体重、外貌、生殖和其他生物过程将完全在人们的控制之下。我们将这些归入生物自由的标题下：即每个人都应该有权选择自己想成为什么，并以最吸引他们的方式生活。当然，关于全球平等获取的重要问题将存在；参见第 3 节。

* Biological freedom. The last 70 years featured advances in birth control, fertility, management of weight, and much more. But I suspect AI-accelerated biology will greatly expand what is possible: weight, physical appearance, reproduction, and other biological processes will be fully under people’s control. We’ll refer to these under the heading of _biological freedom:_ the idea that everyone should be empowered to choose what they want to become and live their lives in the way that most appeals to them. There will of course be important questions about global equality of access; see Section 3 for these.

* 人类寿命翻倍。注意，在 5-10 年内可能很难知道我们已经将人类寿命翻倍。虽然我们可能已经实现了，但在研究时间框架内我们可能还不知道。这似乎很激进，但 20 世纪预期寿命几乎翻了一番（从约 40 岁到 75 岁），因此“压缩的 21 世纪”再次将其翻倍至 150 岁是“符合趋势的”。显然，涉及减缓实际衰老过程的干预措施将与上个世纪预防（主要是儿童）因疾病过早死亡所需的干预措施不同，但变化的幅度并非前所未有。这是我愿意，尽管在治愈疾病和减缓衰老过程本身之间存在明显的生物学差异，但从更远的距离看统计趋势并说“即使细节不同，我认为人类科学可能会找到继续这一趋势的方法；毕竟，任何复杂事物中的平滑趋势必然是通过累加非常异质的组成部分而构成的。具体来说，已经存在将大鼠最大寿命延长 25-50%且副作用有限的药物。而且一些动物（例如某些类型的乌龟）已经活了 200 年，因此人类显然没有达到某个理论上限。猜测一下，最需要的东西可能是可靠、非古德哈特的人类衰老生物标志物，因为这将允许在实验和临床试验中快速迭代。一旦人类寿命达到 150 岁，我们可能能够达到“逃逸速度”，为目前活着的大多数人赢得足够的时间，使他们能够活到他们想要的时间，尽管当然不能保证这在生物学上是可能的。

* Doubling of the human lifespan1818 Note it may be hard to know that we have doubled the human lifespan within the 5-10 years. While we might have accomplished it, we may not know it yet within the study time frame. . This might seem radical, but life expectancy increased almost 2x in the 20th century (from ~40 years to ~75), so it’s “on trend” that the “compressed 21st” would double it again to 150. Obviously the interventions involved in slowing the actual aging process will be different from those that were needed in the last century to prevent (mostly childhood) premature deaths from disease, but the magnitude of change is not unprecedented1919 This is one place where I am willing, despite the obvious biological differences between curing diseases and slowing down the aging process itself, to instead look from a greater distance at the statistical trend and say “even though the details are different, I think human science would probably find a way to continue this trend; after all, smooth trends in anything complex are necessarily made by adding up very heterogeneous components. . Concretely, there already exist drugs that increase maximum lifespan in rats by 25-50% with limited ill effects. And some animals (e.g. some types of turtle) already live 200 years, so humans are manifestly not at some theoretical upper limit. At a guess, the most important thing that is needed might be reliable, non-Goodhart-able biomarkers of human aging, as that will allow fast iteration on experiments and clinical trials. Once human lifespan is 150, we may be able to reach “escape velocity”, buying enough time that most of those currently alive today will be able to live as long as they want, although there’s certainly no guarantee this is biologically possible.

值得看看这个列表，并思考如果所有这些都在 7-12 年后实现（这将符合激进的 AI 时间线），世界将会多么不同。不言而喻，这将是一个难以想象的人道主义胜利，一举消除困扰人类数千年的绝大多数祸害。我的许多朋友和同事正在抚养孩子，当这些孩子长大时，我希望任何关于疾病的提及对他们来说听起来就像坏血病、天花或鼠疫对我们来说一样。那一代还将受益于增强的生物自由和自我表达，并且运气好的话，也可能能够活到他们想要的时间。

It is worth looking at this list and reflecting on how different the world will be if all of it is achieved 7-12 years from now (which would be in line with an aggressive AI timeline). It goes without saying that it would be an unimaginable humanitarian triumph, the elimination all at once of most of the scourges that have haunted humanity for millennia. Many of my friends and colleagues are raising children, and when those children grow up, I hope that any mention of disease will sound to them the way scurvy, smallpox, or bubonic plague sounds to us. That generation will also benefit from increased biological freedom and self-expression, and with luck may also be able to live as long as they want.

很难高估这些变化对除了预期强大 AI 的小社区之外的所有人来说将有多么令人惊讶。例如，美国成千上万的经济学家和政策专家目前争论如何保持社会保障和医疗保险的偿付能力，以及更广泛地如何降低医疗保健成本（主要由 70 岁以上的人消耗，尤其是那些患有癌症等绝症的人）。如果这一切成为现实，这些项目的情况很可能会得到根本改善。

It’s hard to overestimate how surprising these changes will be to everyone except the small community of people who expected powerful AI. For example, thousands of economists and policy experts in the US currently debate how to keep Social Security and Medicare solvent, and more broadly how to keep down the cost of healthcare (which is mostly consumed by those over 70 and especially those with terminal illnesses such as cancer). The situation for these programs is likely to be radically improved if all this comes to pass20

例如，我被告知，每年生产率增长提高 1%甚至 0.5%将对与这些项目相关的预测产生变革性影响。如果本文中设想的事情成为现实，生产率收益可能远大于此。

20 As an example, I’m told that an increase in productivity growth per year of 1% or even 0.5% would be transformative in projections related to these programs. If the ideas contemplated in this essay come to pass, productivity gains could be much larger than this.

，因为工作年龄人口与退休人口的比例将发生巨大变化。毫无疑问，这些挑战将被其他挑战所取代，例如如何确保新技术的广泛普及，但值得反思的是，即使生物学是唯一被 AI 成功加速的领域，世界将会发生多大的变化。

, as the ratio of working age to retired population will change drastically. No doubt these challenges will be replaced with others, such as how to ensure widespread access to the new technologies, but it is worth reflecting on how much the world will change even if biology is the _only_ area to be successfully accelerated by AI.

2. 神经科学与心智 2. Neuroscience and mind

在上一节中，我主要关注的是_生理_疾病和一般生物学，没有涉及神经科学或心理健康。但神经科学是生物学的一个分支学科，心理健康与生理健康同样重要。事实上，心理健康甚至比生理健康更直接地影响人类福祉。数亿人因成瘾、抑郁、精神分裂症、低功能自闭症、创伤后应激障碍、精神病态等问题而生活质量极低。

In the previous section I focused on _physical_ diseases and biology in general, and didn’t cover neuroscience or mental health. But neuroscience is a subdiscipline of biology and mental health is just as important as physical health. In fact, if anything, mental health affects human well-being even more directly than physical health. Hundreds of millions of people have very low quality of life due to problems like addiction, depression, schizophrenia, low-functioning autism, PTSD, psychopathy21

媒体喜欢描绘高地位的精神病态者，但普通精神病态者可能是一个经济前景不佳、冲动控制差、最终在监狱中度过大量时间的人。

21 The media loves to portray high status psychopaths), but the average psychopath is probably a person with poor economic prospects and poor impulse control who ends up spending significant time in prison.

或智力残疾。还有数十亿人挣扎于日常问题，这些问题通常可以被解释为这些严重临床障碍的较温和版本。与一般生物学一样，我们可能不仅能够解决问题，还能改善人类体验的基线水平。

, or intellectual disabilities. Billions more struggle with everyday problems that can often be interpreted as much milder versions of one of these severe clinical disorders. And as with general biology, it may be possible to go beyond addressing problems to improving the baseline quality of human experience.

我为生物学制定的基本框架同样适用于神经科学。该领域由少量发现推动，这些发现通常与测量或精确干预工具相关——在上述列表中，光遗传学是一项神经科学发现，而最近的 CLARITY 和扩展显微镜也是同一方向的进展，此外许多通用细胞生物学方法直接应用于神经科学。我认为这些进展的速度将因 AI 而类似地加速，因此“100 年进展在 5-10 年内完成”的框架同样适用于神经科学，原因与生物学相同。与生物学一样，20 世纪神经科学的进展是巨大的——例如，直到 20 世纪 50 年代我们才理解神经元如何以及为何放电。因此，有理由预期 AI 加速的神经科学将在几年内产生快速进展。

The basic framework that I laid out for biology applies equally to neuroscience. The field is propelled forward by a small number of discoveries often related to tools for measurement or precise intervention – in the list of those above, optogenetics was a neuroscience discovery, and more recently CLARITY and expansion microscopy%20is%20a,them%20using%20a%20polymer%20system.) are advances in the same vein, in addition to many of the general cell biology methods directly carrying over to neuroscience. I think the rate of these advances will be similarly accelerated by AI and therefore that the framework of “100 years of progress in 5-10 years” applies to neuroscience in the same way it does to biology and for the same reasons. As in biology, the progress in 20th century neuroscience was enormous – for example we didn’t even understand how or why neurons fired until the 1950s. Thus, it seems reasonable to expect AI-accelerated neuroscience to produce rapid progress over a few years.

在这个基本图景中，我们还需要补充一点：过去几年我们关于 AI 本身学到的一些东西（或正在学习的东西）很可能有助于推动神经科学的发展，即使它仍然只由人类进行。可解释性是一个明显的例子：尽管生物神经元与人工神经元在表面上运作方式完全不同（它们通过尖峰和尖峰频率通信，因此存在人工神经元中没有的时间元素，并且许多与细胞生理学和神经递质相关的细节显著改变了它们的运作），但“分布式、经过训练的简单单元网络如何执行组合线性/非线性运算以共同完成重要计算”这一基本问题是相同的，我强烈怀疑在大多数关于计算和回路的有趣问题中，单个神经元通信的细节将被抽象化。

There is one thing we should add to this basic picture, which is that some of the things we’ve learned (or are learning) about AI itself in the last few years are likely to help advance neuroscience, even if it continues to be done only by humans. Interpretability is an obvious example: although biological neurons superficially operate in a completely different manner from artificial neurons (they communicate via spikes and often spike rates, so there is a time element not present in artificial neurons, and a bunch of details relating to cell physiology and neurotransmitters modifies their operation substantially), the basic question of “how do distributed, trained networks of simple units that perform combined linear/non-linear operations work together to perform important computations” is the same, and I strongly suspect the details of individual neuron communication will be abstracted away in most of the interesting questions about computation and circuits22

我认为这有点类似于我们从可解释性中学到的许多结果（尽管可能不是全部）即使我们当前人工神经网络的某些架构细节（如注意力机制）被改变或替换，也仍然相关。

22 I think this is somewhat analogous to the fact that many, though likely not all, of the results we’re learning from interpretability would continue to be relevant even if some of the architectural details of our current artificial neural nets, such as the attention mechanism, were changed or replaced in some way.

仅举一例，AI 系统中可解释性研究人员发现的一种计算机制最近在小鼠大脑中被重新发现。

. As just one example of this, a computational mechanism discovered by interpretability researchers in AI systems was recently rediscovered in the brains of mice.

在人工神经网络上进行实验比在真实神经网络上容易得多（后者通常需要切割动物大脑），因此可解释性很可能成为提高我们对神经科学理解的一种工具。此外，强大的 AI 本身可能比人类更能开发和运用这一工具。

It is much easier to do experiments on artificial neural networks than on real ones (the latter often requires cutting into animal brains), so interpretability may well become a tool for improving our understanding of neuroscience. Furthermore, powerful AIs will themselves probably be able to develop and apply this tool better than humans can.

然而，除了可解释性之外，我们从 AI 中学到的关于智能系统如何_训练_的知识应该（尽管我不确定是否已经）引发神经科学的一场革命。当我在神经科学领域工作时，许多人关注的是我现在认为是关于学习的错误问题，因为 Scaling 假设/苦涩的教训的概念当时还不存在。简单的目标函数加上大量数据可以驱动极其复杂的行为这一想法，使得理解目标函数和架构偏差变得更有趣，而理解涌现计算的细节则不那么有趣。我近年来没有密切跟进该领域，但我隐约感觉计算神经科学家仍未完全吸收这一教训。我对 Scaling 假设的态度一直是“啊哈——这是对智能如何运作以及它为何如此容易进化的一种高层次解释”，但我不认为这是普通神经科学家的观点，部分原因是作为“智能的秘密”的 Scaling 假设即使在 AI 内部也未被完全接受。

Beyond just interpretability though, what we have learned from AI about how intelligent systems are _trained_ should (though I am not sure it _has_ yet) cause a revolution in neuroscience. When I was working in neuroscience, a lot of people focused on what I would now consider the wrong questions about learning, because the concept of the scaling hypothesis / bitter lesson didn’t exist yet. The idea that a simple objective function plus a lot of data can drive incredibly complex behaviors makes it more interesting to understand the objective functions and architectural biases and less interesting to understand the details of the emergent computations. I have not followed the field closely in recent years, but I have a vague sense that computational neuroscientists have still not fully absorbed the lesson. My attitude to the scaling hypothesis has always been “aha – this is an explanation, at a high level, of how intelligence works and how it so easily evolved”, but I don’t think that’s the average neuroscientist’s view, in part because the scaling hypothesis as “the secret to intelligence” isn’t fully accepted even within AI.

我认为神经科学家应该尝试将这一基本见解与人脑的特殊性（生物物理限制、进化历史、拓扑结构、运动和感觉输入/输出的细节）结合起来，以试图解决神经科学的一些关键谜题。有些人可能正在这样做，但我怀疑这还不够，AI 神经科学家将能够更有效地利用这一角度来加速进展。

I think that neuroscientists should be trying to combine this basic insight with the particularities of the human brain (biophysical limitations, evolutionary history, topology, details of motor and sensory inputs/outputs) to try to figure out some of neuroscience’s key puzzles. Some likely are, but I suspect it’s not enough yet, and that AI neuroscientists will be able to more effectively leverage this angle to accelerate progress.

我预计 AI 将通过四条不同的途径加速神经科学的进展，所有这些途径有望共同治愈精神疾病并改善功能：

I expect AI to accelerate neuroscientific progress along four distinct routes, all of which can hopefully work together to cure mental illness and improve function:

* 传统分子生物学、化学和遗传学。这本质上与第 1 节中一般生物学的情况相同，AI 可能通过相同的机制加速它。有许多药物通过调节神经递质来改变大脑功能、影响警觉性或感知、改变情绪等，AI 可以帮助我们发明更多此类药物。AI 也可能加速对精神疾病遗传基础的研究。

* Traditional molecular biology, chemistry, and genetics. This is essentially the same story as general biology in Section 1, and AI can likely speed it up via the same mechanisms. There are many drugs that modulate neurotransmitters in order to alter brain function, affect alertness or perception, change mood, etc., and AI can help us invent many more. AI can probably also accelerate research on the genetic basis of mental illness.

* 精细神经测量和干预。这是测量大量单个神经元或神经元回路活动并干预以改变其行为的能力。光遗传学和神经探针是能够在活体生物中进行测量和干预的技术，许多非常先进的方法（如分子记录带以读取大量单个神经元的放电模式）也已被提出，并且在原则上似乎是可行的。

* Fine-grained neural measurement and intervention. This is the ability to measure what a lot of individual neurons or neuronal circuits are doing, and intervene to change their behavior. Optogenetics and neural probes are technologies capable of both measurement and intervention in live organisms, and a number of very advanced methods (such as molecular ticker tapes to read out the firing patterns of large numbers of individual neurons) have also been proposed and seem possible in principle.

* 先进的计算神经科学。如上所述，现代 AI 的具体见解和_整体观_很可能可以有效地应用于系统神经科学的问题，包括可能揭示精神病或情绪障碍等复杂疾病的真正原因和动态。

* Advanced computational neuroscience. As noted above, both the specific insights and the _gestalt_ of modern AI can probably be applied fruitfully to questions in systems neuroscience00099-2), including perhaps uncovering the real causes and dynamics of complex diseases like psychosis or mood disorders.

* 行为干预。鉴于对神经科学生物学方面的关注，我很少提及这一点，但精神病学和心理学在 20 世纪当然发展出了广泛的行为干预手段；有理由认为 AI 也可以加速这些干预，包括新方法的开发和帮助患者坚持现有方法。更广泛地说，“AI 教练”的概念——它始终帮助你成为最好的自己，研究你的互动并帮助你学习更有效——似乎非常有前景。

* Behavioral interventions. I haven’t much mentioned it given the focus on the biological side of neuroscience, but psychiatry and psychology have of course developed a wide repertoire of behavioral interventions over the 20th century; it stands to reason that AI could accelerate these as well, both the development of new methods and helping patients to adhere to existing methods. More broadly, the idea of an “AI coach” who always helps you to be the best version of yourself, who studies your interactions and helps you learn to be more effective, seems very promising.

我的猜测是，这四条进展途径共同作用，与生理疾病一样，即使没有 AI，也有望在未来 100 年内治愈或预防大多数精神疾病——因此，在 AI 加速的 5-10 年内合理完成。具体来说，我对将要发生的事情的猜测如下：

It’s my guess that these four routes of progress working together would, as with physical disease, be on track to lead to the cure or prevention of most mental illness in the next 100 years even if AI was not involved – and thus might reasonably be completed in 5-10 AI-accelerated years. Concretely my guess at what will happen is something like:

* 大多数精神疾病可能可以治愈。我不是精神疾病专家（我在神经科学领域的时间是建造探针来研究小群神经元），但我的猜测是，像创伤后应激障碍、抑郁、精神分裂症、成瘾等疾病可以通过上述四个方向的某种组合被弄清楚并得到非常有效的治疗。答案很可能是“生化方面出了问题”（尽管可能非常复杂）和“神经网络在高层面上出了问题”的某种组合。也就是说，这是一个系统神经科学问题——但这并不否定上述行为干预的影响。测量和干预工具，尤其是在活体人类中，似乎可能导致快速迭代和进展。

* Most mental illness can probably be cured. I’m not an expert in psychiatric disease (my time in neuroscience was spent building probes to study small groups of neurons) but it’s my guess that diseases like PTSD, depression, schizophrenia, addiction, etc. can be figured out and very effectively treated via some combination of the four directions above. The answer is likely to be some combination of “something went wrong biochemically” (although it could be very complex) and “something went wrong with the neural network, at a high level”. That is, it’s a systems neuroscience question—though that doesn’t gainsay the impact of the behavioral interventions discussed above. Tools for measurement and intervention, especially in live humans, seem likely to lead to rapid iteration and progress.

* 非常“结构性”的疾病可能更困难，但并非不可能。有一些证据表明精神病态与明显的神经解剖学差异相关——精神病态者的一些大脑区域只是更小或发育更差。精神病态者也被认为从小缺乏同理心；无论他们的大脑有什么不同，可能一直都是这样。智力残疾和其他一些疾病可能也是如此。重构大脑听起来很难，但这似乎也是一项对智能回报很高的任务。也许有某种方法可以诱导成人大脑进入更早期或更可塑的状态，从而进行重塑。我对这有多大可能性非常不确定，但我的直觉是对 AI 在此能发明的东西持乐观态度。

* Conditions that are very “structural” may be more difficult, but not impossible. There’s some evidence that psychopathy is associated with obvious neuroanatomical differences – that some brain regions are simply smaller or less developed in psychopaths. Psychopaths are also believed to lack empathy from a young age; whatever is different about their brain, it was probably always that way. The same may be true of some intellectual disabilities, and perhaps other conditions. Restructuring the brain sounds hard, but it also seems like a task with high returns to intelligence. Perhaps there is some way to coax the adult brain into an earlier or more plastic state where it can be reshaped. I’m very uncertain how possible this is, but my instinct is to be optimistic about what AI can invent here.

* 精神疾病的有效遗传预防似乎是可能的。大多数精神疾病是部分遗传的，全基因组关联研究开始识别相关因素，这些因素通常数量众多。通过胚胎筛查可能可以预防大多数这些疾病，类似于生理疾病的情况。一个区别是精神疾病更可能是多基因的（许多基因贡献），因此由于复杂性，存在无意中选择与疾病相关的积极特征的风险增加。然而奇怪的是，近年来的 GWAS 研究似乎表明这些相关性可能被夸大了。无论如何，AI 加速的神经科学可能帮助我们弄清楚这些事情。当然，针对复杂特征的胚胎筛查引发了许多社会问题，并将具有争议性，尽管我猜测大多数人会支持对严重或使人衰弱的精神疾病进行筛查。

* Effective genetic prevention of mental illness seems possible. Most mental illness is partially heritable, and genome-wide association studies are starting to gain traction on identifying the relevant factors, which are often many in number. It will probably be possible to prevent most of these diseases via embryo screening, similar to the story with physical disease. One difference is that psychiatric disease is more likely to be polygenic (many genes contribute), so due to complexity there’s an increased risk of unknowingly selecting against positive traits that are correlated with disease. Oddly however, in recent years GWAS studies seem to suggest that these correlations might have been overstated. In any case, AI-accelerated neuroscience may help us to figure these things out. Of course, embryo screening for complex traits raises a number of societal issues and will be controversial, though I would guess that most people would support screening for severe or debilitating mental illness.

* 我们通常不认为是临床疾病的日常问题也将得到解决。我们大多数人都有日常心理问题，通常不被认为达到临床疾病的程度。有些人容易发怒，其他人难以集中注意力或经常昏昏欲睡，有些人恐惧或焦虑，或者对变化反应不佳。今天，已经存在帮助提高警觉性或注意力的药物（咖啡因、莫达非尼、利他林），但与许多其他先前领域一样，可能还有更多可能性。可能还有许多此类药物存在但尚未被发现，也可能有全新的干预方式，如靶向光刺激（见上文光遗传学）或磁场。鉴于我们在 20 世纪已经开发了那么多调节认知功能和情绪状态的药物，我对“压缩的 21 世纪”非常乐观，届时每个人都能让自己的大脑表现更好，拥有更充实的日常体验。

* Everyday problems that we don’t think of as clinical disease will also be solved. Most of us have everyday psychological problems that are not ordinarily thought of as rising to the level of clinical disease. Some people are quick to anger, others have trouble focusing or are often drowsy, some are fearful or anxious, or react badly to change. Today, drugs already exist to help with e.g. alertness or focus (caffeine, modafinil, ritalin) but as with many other previous areas, much more is likely to be possible. Probably many more such drugs exist and have not been discovered, and there may also be totally new modalities of intervention, such as targeted light stimulation (see optogenetics above) or magnetic fields. Given how many drugs we’ve developed in the 20th century that tune cognitive function and emotional state, I’m very optimistic about the “compressed 21st” where everyone can get their brain to behave a bit better and have a more fulfilling day-to-day experience.

* 人类基线体验可以更好。更进一步，许多人经历过非凡的启示时刻、创造性灵感、同情、满足、超越、爱、美或冥想般的平静。这些体验的特征和频率因人而异，同一个人在不同时间也不同，有时也可以由各种药物触发（尽管常有副作用）。所有这些都表明“可能体验的空间”非常广阔，人们生活中更大比例的时间可以由这些非凡时刻组成。也可能可以全面改善各种认知功能。这或许是“生物自由”或“延长寿命”的神经科学版本。

* Human baseline experience can be much better. Taking this one step further, many people have experienced extraordinary moments of revelation, creative inspiration, compassion, fulfillment, transcendence, love, beauty, or meditative peace. The character and frequency of these experiences differs greatly from person to person and within the same person at different times, and can also sometimes be triggered by various drugs (though often with side effects). All of this suggests that the “space of what is possible to experience” is very broad and that a larger fraction of people’s lives could consist of these extraordinary moments. It is probably also possible to improve various cognitive functions across the board. This is perhaps the neuroscience version of “biological freedom” or “extended lifespans”.

在科幻小说对 AI 的描绘中经常出现的一个话题，但我有意没有在这里讨论的是“意识上传”，即捕捉人脑的模式和动态并将其实例化到软件中。这个话题本身就可以写一篇文章，但只需说，虽然我认为上传在原则上几乎肯定是可能的，但在实践中，即使有强大的 AI，它也面临重大的技术和社会挑战，很可能超出了我们讨论的 5-10 年窗口。

One topic that often comes up in sci-fi depictions of AI, but that I intentionally haven’t discussed here, is “mind uploading”, the idea of capturing the pattern and dynamics of a human brain and instantiating them in software. This topic could be the subject of an essay all by itself, but suffice it to say that while I think uploading is almost certainly possible in principle, in practice it faces significant technological and societal challenges, even with powerful AI, that likely put it outside the 5-10 year window we are discussing.

总之，AI 加速的神经科学很可能极大地改善甚至治愈大多数精神疾病的治疗方法，并极大地扩展“认知和心智自由”以及人类的认知和情感能力。它将与上一节描述的生理健康改善一样彻底。也许世界在外表上不会有明显不同，但人类所体验的世界将是一个更好、更人性化的地方，也是一个提供更多自我实现机会的地方。我还怀疑，改善的心理健康将缓解许多其他社会问题，包括那些看似政治或经济的问题。

In summary, AI-accelerated neuroscience is likely to vastly improve treatments for, or even cure, most mental illness as well as greatly expand “cognitive and mental freedom” and human cognitive and emotional abilities. It will be every bit as radical as the improvements in physical health described in the previous section. Perhaps the world will not be visibly different on the outside, but the world as experienced by humans will be a much better and more humane place, as well as a place that offers greater opportunities for self-actualization. I also suspect that improved mental health will ameliorate a lot of other societal problems, including ones that seem political or economic.

3. 经济发展与贫困 3. Economic development and poverty

前两节讨论的是开发治疗疾病和改善人类生活质量的新技术。然而，从人道主义的角度来看，一个显而易见的问题是：“每个人都能获得这些技术吗？”

The previous two sections are about _developing_ new technologies that cure disease and improve the quality of human life. However an obvious question, from a humanitarian perspective, is: “will everyone have access to these technologies?”

开发一种疾病的治疗方法是一回事，从世界上根除这种疾病则是另一回事。更广泛地说，许多现有的健康干预措施尚未在世界各地得到应用，而且（非健康领域的）技术改进总体上也是如此。换句话说，世界许多地区的生活水平仍然极其贫困：撒哈拉以南非洲的人均 GDP 约为 2000 美元，而美国约为 75000 美元。如果人工智能进一步提高了发达国家的经济增长和生活质量，却对发展中国家帮助甚少，我们应该将其视为可怕的道德失败，以及前两节中真正人道主义胜利的污点。理想情况下，强大的人工智能应该帮助发展中国家赶上发达国家，即使它同时也在彻底改变后者。

It is one thing to develop a cure for a disease, it is another thing to eradicate the disease from the world. More broadly, many existing health interventions have not yet been applied everywhere in the world, and for that matter the same is true of (non-health) technological improvements in general. Another way to say this is that living standards in many parts of the world are still desperately poor: GDP per capita is ~$2,000 in Sub-Saharan Africa as compared to ~$75,000 in the United States. If AI further increases economic growth and quality of life in the developed world, while doing little to help the developing world, we should view that as a terrible moral failure and a blemish on the genuine humanitarian victories in the previous two sections. Ideally, powerful AI should help the developing world _catch up to_ the developed world, even as it revolutionizes the latter.

我不像对人工智能能够发明基础技术那样确信它能解决不平等和经济增长问题，因为技术显然具有很高的智力回报（包括绕过复杂性和数据缺乏的能力），而经济则涉及许多来自人类的约束，以及大量的内在复杂性。我有点怀疑人工智能能否解决著名的“社会主义计算问题”。

I am not as confident that AI can address inequality and economic growth as I am that it can invent fundamental technologies, because technology has such obvious high returns to intelligence (including the ability to route around complexities and lack of data) whereas the economy involves a lot of constraints from humans, as well as a large dose of intrinsic complexity. I am somewhat skeptical that an AI could solve the famous “socialist calculation problem”23

23 我怀疑这有点像经典的混沌系统——受到不可简化的复杂性的困扰，必须以去中心化的方式管理。尽管正如我在本节后面所说，更温和的干预可能是可能的。经济学家埃里克·布林约尔松向我提出的一个反驳观点是，大公司（如沃尔玛或优步）开始拥有足够的集中知识，能够比任何去中心化过程更好地了解消费者，这或许迫使我们修正哈耶克关于谁拥有最佳本地知识的见解。

23 I suspect it is a bit like a classical chaotic system – beset by irreducible complexity that has to be managed in a mostly decentralized manner. Though as I say later in this section, more modest interventions may be possible. A counterargument, made to me by economist Erik Brynjolfsson, is that large companies (such as Walmart or Uber) are starting to have enough centralized knowledge to understand consumers better than any decentralized process could, perhaps forcing us to revise Hayek’s insights about who has the best local knowledge.

而且我认为政府不会（也不应该）将经济政策交给这样一个实体，即使它能够做到。还有一些问题，比如如何说服人们接受有效但他们可能怀疑的治疗方法。

and I don’t think governments will (or should) turn over their economic policy to such an entity, even if it could do so. There are also problems like how to convince people to take treatments that are effective but that they may be suspicious of.

发展中国家面临的挑战因私营和公共部门普遍存在的腐败而变得更加复杂。腐败造成了一个恶性循环：它加剧了贫困，而贫困又滋生了更多的腐败。人工智能驱动的经济发展计划需要应对腐败、薄弱制度和其他非常人性化的挑战。

The challenges facing the developing world are made even more complicated by pervasive corruption in both private and public sectors. Corruption creates a vicious cycle: it exacerbates poverty, and poverty in turn breeds more corruption. AI-driven plans for economic development need to reckon with corruption, weak institutions, and other very human challenges.

尽管如此，我确实看到了乐观的重要理由。疾病确实被根除过，许多国家也确实从贫穷走向了富裕，并且很明显，这些任务中的决策表现出很高的智力回报（尽管存在人类约束和复杂性）。因此，人工智能很可能比当前的做法做得更好。也可能有针对性的干预措施可以绕过人类约束，让人工智能能够集中精力。但更重要的是，我们必须尝试。人工智能公司和发达国家的政策制定者都需要尽自己的一份力量，确保发展中国家不被排除在外；道德责任太大了。因此，在本节中，我将继续提出乐观的理由，但请始终记住，成功并非必然，取决于我们的集体努力。

Nevertheless, I do see significant reasons for optimism. Diseases _have_ been eradicated and many countries _have_ gone from poor to rich, and it is clear that the decisions involved in these tasks exhibit high returns to intelligence (despite human constraints and complexity). Therefore, AI can likely do them better than they are currently being done. There may also be targeted interventions that get around the human constraints and that AI could focus on. More importantly though, _we have_ to try. Both AI companies and developed world policymakers will need to do their part to ensure that the developing world is not left out; the moral imperative is too great. So in this section, I’ll continue to make the optimistic case, but keep in mind everywhere that success is not guaranteed and depends on our collective efforts.

下面我对强大人工智能开发后的 5-10 年内，发展中国家可能的情况做一些猜测：

Below I make some guesses about how I think things may go in the developing world over the 5-10 years after powerful AI is developed:

* 健康干预的分配。我最乐观的领域可能是将健康干预措施分布到世界各地。疾病实际上是通过自上而下的运动被根除的：天花在 20 世纪 70 年代被完全消灭，脊髓灰质炎和麦地那龙线虫病几乎被根除，每年病例不到 100 例。数学上复杂的流行病学建模在疾病根除运动中发挥着积极作用，而且似乎很可能，比人类更聪明的人工智能系统有空间比人类做得更好。分配的物流也可能得到极大优化。作为 GiveWell 的早期捐赠者，我学到的一件事是，一些健康慈善机构比其他机构有效得多；希望人工智能加速的努力会更加有效。此外，一些生物学进展实际上使分配物流变得容易得多：例如，疟疾难以根除，因为它需要在每次感染时进行治疗；一种只需接种一次的疫苗会使物流简单得多（而且针对疟疾的此类疫苗目前正在开发中）。甚至更简单的分配机制也是可能的：一些疾病原则上可以通过针对其动物携带者来根除，例如释放感染了某种细菌的蚊子，这种细菌会阻断它们携带疾病的能力（然后它们会感染所有其他蚊子），或者干脆使用基因驱动来消灭蚊子。这需要一个或几个集中的行动，而不是必须单独治疗数百万人的协调运动。总的来说，我认为 5-10 年是一个合理的时间线，让相当一部分（也许 50%）人工智能驱动的健康益处传播到甚至世界上最贫穷的国家。一个良好的目标可能是，在强大人工智能出现后的 5-10 年，发展中国家至少在健康方面比今天的发达国家好得多，即使它继续落后于发达国家。实现这一目标当然需要全球健康、慈善、政治倡导和许多其他方面的巨大努力，人工智能开发者和政策制定者都应提供帮助。

* Distribution of health interventions. The area where I am perhaps most optimistic is distributing health interventions throughout the world. Diseases have actually been eradicated by top-down campaigns: smallpox was fully eliminated in the 1970s, and polio and guinea worm are nearly eradicated with less than 100 cases per year. Mathematically sophisticated epidemiological modeling plays an active role in disease eradication campaigns, and it seems very likely that there is room for smarter-than-human AI systems to do a better job of it than humans are. The logistics of distribution can probably also be greatly optimized. One thing I learned as an early donor to GiveWell is that some health charities are way more effective than others; the hope is that AI-accelerated efforts would be more effective still. Additionally, some biological advances actually make the logistics of distribution much easier: for example, malaria has been difficult to eradicate because it requires treatment each time the disease is contracted; a vaccine that only needs to be administered once makes the logistics much simpler (and such vaccines for malaria are in fact currently being developed). Even simpler distribution mechanisms are possible: some diseases could in principle be eradicated by targeting their animal carriers, for example releasing mosquitoes infected with a bacterium that blocks their ability to carry a disease (who then infect all the other mosquitos) or simply using gene drives to wipe out the mosquitos. This requires one or a few centralized actions, rather than a coordinated campaign that must individually treat millions. Overall, I think 5-10 years is a reasonable timeline for a good fraction (maybe 50%) of AI-driven health benefits to propagate to even the poorest countries in the world. A good goal might be for the developing world 5-10 years after powerful AI to at least be substantially healthier than the developed world is today, even if it continues to lag behind the developed world. Accomplishing this will of course require a huge effort in global health, philanthropy, political advocacy, and many other efforts, which both AI developers and policymakers should help with.

* 经济增长。发展中国家能否不仅在健康方面，而且在经济上全面迅速赶上发达国家？有一些先例：在 20 世纪的最后几十年，几个东亚经济体实现了持续约 10%的年实际 GDP 增长率，使它们能够赶上发达国家。人类经济规划者做出了导致这一成功的决策，不是通过直接控制整个经济，而是通过拉动几个关键杠杆（例如出口导向型增长的产业政策，以及抵制依赖自然资源财富的诱惑）；“人工智能财政部长和央行行长”有可能复制或超越这一 10%的成就。一个重要的问题是如何让发展中国家政府采纳这些政策，同时尊重自决原则——有些政府可能对此充满热情，但其他政府可能持怀疑态度。乐观的一面是，前一点中的许多健康干预措施可能会自然地促进经济增长：根除艾滋病/疟疾/寄生虫将对生产力产生变革性影响，更不用说一些神经科学干预措施（如改善情绪和注意力）在发达国家和发展中国家都会带来的经济效益。最后，非健康领域的人工智能加速技术（如能源技术、运输无人机、改进的建筑材料、更好的物流和分销等）可能会自然地渗透到世界；例如，即使是手机也通过市场机制迅速渗透到撒哈拉以南非洲，无需慈善努力。更消极的一面是，虽然人工智能和自动化有许多潜在好处，但它们也对经济发展构成挑战，特别是对那些尚未工业化的国家。找到确保这些国家在日益自动化的时代仍能发展和改善经济的方法，是经济学家和政策制定者需要应对的重要挑战。总体而言，一个梦想的情景——也许是一个目标——是发展中国家 20%的年 GDP 增长率，其中 10%来自人工智能驱动的经济决策，10%来自人工智能加速技术的自然传播，包括但不限于健康。如果实现，这将使撒哈拉以南非洲在 5-10 年内达到中国当前的人均 GDP，同时使其他大部分发展中国家达到高于美国当前 GDP 的水平。再次强调，这是一个梦想情景，不是默认发生的：这是我们所有人都必须共同努力使之更有可能实现的事情。

* Economic growth. Can the developing world quickly catch up to the developed world, not just in health, but across the board economically? There is some precedent for this: in the final decades of the 20th century, several East Asian economies achieved sustained ~10% annual real GDP growth rates, allowing them to catch up with the developed world. Human economic planners made the decisions that led to this success, not by directly controlling entire economies but by pulling a few key levers (such as an industrial policy of export-led growth, and resisting the temptation to rely on natural resource wealth); it’s plausible that “AI finance ministers and central bankers” could replicate or exceed this 10% accomplishment. An important question is how to get developing world governments to adopt them while respecting the principle of self-determination—some may be enthusiastic about it, but others are likely to be skeptical. On the optimistic side, many of the health interventions in the previous bullet point are likely to organically increase economic growth: eradicating AIDS/malaria/parasitic worms would have a transformative effect on productivity, not to mention the economic benefits that some of the neuroscience interventions (such as improved mood and focus) would have in developed and developing worlds alike. Finally, non-health AI-accelerated technology (such as energy technology, transport drones, improved building materials, better logistics and distribution, and so on) may simply permeate the world naturally; for example, even cell phones quickly permeated sub-Saharan Africa via market mechanisms, without needing philanthropic efforts. On the more negative side, while AI and automation have many potential benefits, they also pose challenges for economic development, particularly for countries that haven't yet industrialized. Finding ways to ensure these countries can still develop and improve their economies in an age of increasing automation is an important challenge for economists and policymakers to address. Overall, a dream scenario—perhaps a goal to aim for—would be 20% annual GDP growth rate in the developing world, with 10% each coming from AI-enabled economic decisions and the natural spread of AI-accelerated technologies, including but not limited to health. If achieved, this would bring sub-Saharan Africa to the current per-capita GDP of China in 5-10 years, while raising much of the rest of the developing world to levels higher than the current US GDP. Again, this is a dream scenario, not what happens by default: it’s something all of us must work together to make more likely.

* 粮食安全。作物技术的进步，如更好的肥料和杀虫剂、更多的自动化和更高效的土地利用，在 20 世纪大幅提高了作物产量，使数百万人免于饥饿。基因工程目前正在进一步改良许多作物。找到更多的方法来做到这一点——以及使农业供应链更加高效——可以给我们带来人工智能驱动的第二次绿色革命，帮助缩小发展中国家和发达国家之间的差距。

* Food security2424 Thanks to Kevin Esvelt for suggesting this point. . Advances in crop technology like better fertilizers and pesticides, more automation, and more efficient land use drastically increased crop yields across the 20th century, saving millions of people from hunger. Genetic engineering is currently improving many crops even further. Finding even more ways to do this—as well as to make agricultural supply chains even more efficient—could give us an AI-driven second Green Revolution, helping close the gap between the developing and developed world.

* 缓解气候变化。气候变化在发展中国家将更加明显，阻碍其发展。我们可以预期，人工智能将带来减缓或防止气候变化的技术改进，从大气碳去除和清洁能源技术到实验室培育的肉类，减少我们对碳密集型工厂化农业的依赖。当然，如上所述，技术并不是限制气候变化进展的唯一因素——与本文讨论的所有其他问题一样，人类社会因素也很重要。但有充分理由认为，人工智能增强的研究将为我们提供手段，使缓解气候变化的成本大大降低，破坏性更小，使许多反对意见变得无关紧要，并释放发展中国家以取得更多经济进步。

* Mitigating climate change. Climate change will be felt much more strongly in the developing world, hampering its development. We can expect that AI will lead to improvements in technologies that slow or prevent climate change, from atmospheric carbon-removal and clean energy technology to lab-grown meat that reduces our reliance on carbon-intensive factory farming. Of course, as discussed above, technology isn’t the only thing restricting progress on climate change—as with all of the other issues discussed in this essay, human societal factors are important. But there’s good reason to think that AI-enhanced research will give us the means to make mitigating climate change far less costly and disruptive, rendering many of the objections moot and freeing up developing countries to make more economic progress.

* 国家内部的不平等。我主要将不平等作为一个全球现象来讨论（我认为这是其最重要的表现形式），但当然不平等也存在于国家内部。随着先进的健康干预措施，特别是寿命的显著延长或认知增强药物，肯定会有合理的担忧，即这些技术“只适用于富人”。我对国家内部的不平等更加乐观，尤其是在发达国家，原因有二。首先，发达国家的市场运作更好，而市场通常擅长随着时间的推移降低高价值技术的成本。其次，发达国家的政治机构对其公民更负责，并且有更大的国家能力来执行普遍接入计划——我预计公民会要求获得那些能如此彻底改善生活质量的技术。当然，这些要求能否成功并非预先确定——这里是我们必须尽一切努力确保公平社会的另一个地方。财富不平等（相对于获得救生和增强生命技术的不平等）是一个单独的问题，似乎更难解决，我将在第 5 节讨论。

* Inequality within countries. I’ve mostly talked about inequality as a global phenomenon (which I do think is its most important manifestation), but of course inequality also exists _within_ countries. With advanced health interventions and especially radical increases in lifespan or cognitive enhancement drugs, there will certainly be valid worries that these technologies are “only for the rich”. I am more optimistic about within-country inequality especially in the developed world, for two reasons. First, markets function better in the developed world, and markets are typically good at bringing down the cost of high-value technologies over time2525 For example, cell phones were initially a technology for the rich, but quickly became very cheap with year-over-year improvements happening so fast as to obviate any advantage of buying a “luxury” cell phone, and today most people have phones of similar quality. . Second, developed world political institutions are more responsive to their citizens and have greater state capacity to execute universal access programs—and I expect citizens to demand access to technologies that so radically improve quality of life. Of course it’s not predetermined that such demands succeed—and here is another place where we collectively have to do all we can to ensure a fair society. There is a separate problem in inequality of _wealth_ (as opposed to inequality of access to life-saving and life-enhancing technologies), which seems harder and which I discuss in Section 5.

* 选择退出问题。在发达国家和发展中国家都存在的一个担忧是人们选择退出人工智能带来的好处（类似于反疫苗运动，或更普遍的卢德运动）。最终可能会出现不良反馈循环，例如，最不善于做出正确决策的人选择退出那些能提高他们决策能力的技术，导致差距不断扩大，甚至创造一个反乌托邦的下层阶级（一些研究人员认为这将破坏民主，我将在下一节进一步讨论）。这将再次给人工智能的积极进展蒙上道德污点。这是一个难以解决的问题，因为我认为强迫人们在道德上是不对的，但我们至少可以尝试提高人们的科学理解——也许人工智能本身可以帮助我们做到这一点。一个令人鼓舞的迹象是，历史上反技术运动往往是雷声大雨点小：抨击现代技术很流行，但大多数人最终还是会采用它，至少当它是个体选择的问题时。个人倾向于采用大多数健康技术和消费技术，而真正受到阻碍的技术，如核能，往往是集体政治决策的结果。

* The opt-out problem. One concern in both developed and developing worlds alike is people _opting out_ of AI-enabled benefits (similar to the anti-vaccine movement, or Luddite movements more generally). There could end up being bad feedback cycles where, for example, the people who are least able to make good decisions opt out of the very technologies that improve their decision-making abilities, leading to an ever-increasing gap and even creating a dystopian underclass (some researchers have argued that this will undermine democracy, a topic I discuss further in the next section). This would, once again, place a moral blemish on AI’s positive advances. This is a difficult problem to solve as I don’t think it is ethically okay to coerce people, but we can at least try to increase people’s scientific understanding—and perhaps AI itself can help us with this. One hopeful sign is that historically anti-technology movements have been more bark than bite: railing against modern technology is popular, but most people adopt it in the end, at least when it’s a matter of individual choice. Individuals tend to adopt most health and consumer technologies, while technologies that are truly hampered, like nuclear power, tend to be collective political decisions.

总的来说，我对迅速将人工智能的生物医学进展带给发展中国家的人们持乐观态度。我希望——尽管不确定——人工智能也能带来前所未有的经济增长率，使发展中国家至少超过发达国家目前的水平。我担心发达国家和发展中国家的“选择退出”问题，但怀疑它会随着时间的推移而消退，并且人工智能可以帮助加速这一过程。这不会是一个完美的世界，落后的人不会完全赶上，至少在最初几年不会。但通过我们的努力，我们或许能够使事情朝着正确的方向快速前进。如果我们做到了，我们至少可以为我们欠地球上每个人的尊严和平等承诺支付首付。

Overall, I am optimistic about quickly bringing AI’s biological advances to people in the developing world. I am hopeful, though not confident, that AI can also enable unprecedented economic growth rates and allow the developing world to at least surpass where the developed world is now. I am concerned about the “opt out” problem in both the developed and developing worlds, but suspect that it will peter out over time and that AI can help accelerate this process. It won’t be a perfect world, and those who are behind won’t fully catch up, at least not in the first few years. But with strong efforts on our part, we may be able to get things moving in the right direction—and fast. If we do, we can make at least a down payment on the promises of dignity and equality that we owe to every human being on earth.

4. 和平与治理 4. Peace and governance

假设前三节中的一切都进展顺利：疾病、贫困和不平等显著减少，人类体验的基线大幅提高。但这并不意味着人类苦难的所有主要根源都已解决。人类仍然是彼此的威胁。尽管存在技术进步和经济发展导致民主与和平的趋势，但这是一个非常松散的趋势，且频繁（且近期）出现倒退。在 20 世纪初，人们认为战争已成为过去；随后爆发了两次世界大战。三十年前，弗朗西斯·福山撰写了关于“历史的终结”以及自由民主最终胜利的文章；但这尚未发生。二十年前，美国政策制定者相信与中国的自由贸易会使其随着财富增长而自由化；这完全没有发生，我们现在似乎正走向与复兴的威权集团的第二次冷战。而且，合理的理论表明，互联网技术实际上可能有利于威权主义，而不是最初认为的民主（例如在“阿拉伯之春”时期）。理解强大的人工智能将如何与这些和平、民主和自由的问题相交织，似乎很重要。

Suppose that everything in the first three sections goes well: disease, poverty, and inequality are significantly reduced and the baseline of human experience is raised substantially. It does not follow that all major causes of human suffering are solved. Humans are still a threat to each other _._ Although there is a trend of technological improvement and economic development leading to democracy and peace, it is a very loose trend, with frequent (and recent) backsliding. At the dawn of the 20th century, people thought they had put war behind them; then came the two world wars. Thirty years ago Francis Fukuyama wrote about “the End of History” and a final triumph of liberal democracy; that hasn’t happened yet. Twenty years ago US policymakers believed that free trade with China would cause it to liberalize as it became richer; that very much didn’t happen, and we now seem headed for a second Cold War with a resurgent authoritarian bloc. And plausible theories suggest that internet technology may actually advantage authoritarianism, not democracy as initially believed (e.g. in the “Arab Spring” period). It seems important to try to understand how powerful AI will intersect with these issues of peace, democracy, and freedom.

不幸的是，我认为没有强有力的理由相信人工智能会优先或结构性地推进民主与和平，就像我认为它会结构性地推进人类健康和减轻贫困一样。人类冲突是对抗性的，人工智能原则上可以帮助“好人”和“坏人”。如果说有什么不同的话，一些结构性因素似乎令人担忧：人工智能似乎可能实现更好的宣传和监控，这两者都是独裁者工具箱中的主要工具。因此，作为个体行动者，我们有责任将事情引向正确的方向：如果我们希望人工智能有利于民主和个人权利，我们就必须为此而奋斗。我对此的感受甚至比国际不平等更强烈：自由民主和政治稳定的胜利并非有保障，甚至可能不太可能，并且需要我们所有人付出巨大的牺牲和承诺，就像过去经常发生的那样。

Unfortunately, I see no strong reason to believe AI will preferentially or structurally advance democracy and peace, in the same way that I think it will structurally advance human health and alleviate poverty. Human conflict is adversarial and AI can in principle help both the “good guys” and the “bad guys”. If anything, some structural factors seem worrying: AI seems likely to enable much better propaganda and surveillance, both major tools in the autocrat’s toolkit. It’s therefore up to us as individual actors to tilt things in the right direction: if we want AI to favor democracy and individual rights, we are going to have to fight for that outcome. I feel even more strongly about this than I do about international inequality: the triumph of liberal democracy and political stability is _not_ guaranteed, perhaps not even likely, and will require great sacrifice and commitment on all of our parts, as it often has in the past.

我认为这个问题有两个部分：国际冲突和国家的内部结构。在国际方面，当强大的人工智能被创造出来时，民主国家在世界舞台上占据上风似乎非常重要。人工智能驱动的威权主义似乎太可怕了，无法想象，因此民主国家需要能够设定强大人工智能被引入世界的条件，既要避免被威权主义者压倒，也要防止威权国家内部的人权侵犯。

I think of the issue as having two parts: international conflict, and the internal structure of nations. On the international side, it seems very important that democracies have the upper hand on the world stage when powerful AI is created. AI-powered authoritarianism seems too terrible to contemplate, so democracies need to be able to set the terms by which powerful AI is brought into the world, both to avoid being overpowered by authoritarians and to prevent human rights abuses within authoritarian countries.

我目前认为实现这一目标的最佳方式是通过“协约战略”

My current guess at the best way to do this is via an “entente strategy”26

26 这是兰德公司即将发表的一篇论文的标题，该论文大致阐述了我描述的战略。

26 This is the title of a forthcoming paper from RAND, that lays out roughly the strategy I describe.

，即一个民主国家联盟寻求通过确保其供应链、快速扩展规模以及阻止或延迟对手获得芯片和半导体设备等关键资源，在强大人工智能方面获得明显优势（即使是暂时的）。这个联盟一方面将利用人工智能实现强大的军事优势（大棒），同时提供将强大人工智能的好处（胡萝卜）分配给越来越广泛的国家，以换取支持联盟促进民主的战略（这有点像“原子用于和平”）。该联盟旨在获得世界上越来越多国家的支持，孤立我们最坏的对手，并最终使他们处于一个更好的境地，即接受与世界其他国家相同的交易：放弃与民主国家竞争，以换取所有好处，并且不与更强大的敌人作战。

, in which a coalition of democracies seeks to gain a clear advantage (even just a temporary one) on powerful AI by securing its supply chain, scaling quickly, and blocking or delaying adversaries’ access to key resources like chips and semiconductor equipment. This coalition would on one hand use AI to achieve robust military superiority (the stick) while at the same time offering to distribute the benefits of powerful AI (the carrot) to a wider and wider group of countries in exchange for supporting the coalition’s strategy to promote democracy (this would be a bit analogous to “Atoms for Peace”). The coalition would aim to gain the support of more and more of the world, isolating our worst adversaries and eventually putting them in a position where they are better off taking the same bargain as the rest of the world: give up competing with democracies in order to receive all the benefits and not fight a superior foe.

如果我们能做到这一切，我们将拥有一个民主国家在世界舞台上领先的世界，并且拥有经济和军事实力来避免被威权国家削弱、征服或破坏，并可能能够将其人工智能优势转化为持久的优势。这乐观地可能导致一个“永恒的 1991”——一个民主国家占据上风、福山的梦想得以实现的世界。再次强调，这将非常难以实现，并且特别需要私营人工智能公司与民主政府之间的密切合作，以及在胡萝卜和大棒之间做出极其明智的平衡决策。

If we can do all this, we will have a world in which democracies lead on the world stage and have the economic and military strength to avoid being undermined, conquered, or sabotaged by autocracies, and may be able to parlay their AI superiority into a durable advantage. This could optimistically lead to an “eternal 1991”—a world where democracies have the upper hand and Fukuyama’s dreams are realized. Again, this will be very difficult to achieve, and will in particular require close cooperation between private AI companies and democratic governments, as well as extraordinarily wise decisions about the balance between carrot and stick.

即使一切顺利，仍然存在每个国家内部民主与威权主义之间的斗争问题。显然很难预测会发生什么，但我确实有些乐观，认为在民主国家控制最强大人工智能的全球环境下，人工智能实际上可能在结构上有利于各地的民主。特别是，在这种环境下，民主政府可以利用其优越的人工智能赢得信息战：他们可以反击威权主义者的影响和宣传行动，甚至可能通过提供威权国家缺乏技术能力来阻止或监控的信息渠道和人工智能服务，创造一个全球自由的信息环境。可能不需要传递宣传，只需要反击恶意攻击并解除信息的自由流动。尽管不是立竿见影，但像这样的公平竞争环境很有可能逐渐将全球治理引向民主，原因有几个。

Even if all that goes well, it leaves the question of the fight between democracy and autocracy _within_ each country. It is obviously hard to predict what will happen here, but I do have some optimism that _given_ a global environment in which democracies control the most powerful AI, _then_ AI may actually structurally favor democracy everywhere. In particular, in this environment democratic governments can use their superior AI to win the information war: they can counter influence and propaganda operations by autocracies and may even be able to create a globally free information environment by providing channels of information and AI services in a way that autocracies lack the technical ability to block or monitor. It probably isn’t necessary to deliver propaganda, only to counter malicious attacks and unblock the free flow of information. Although not immediate, a level playing field like this stands a good chance of gradually tilting global governance towards democracy, for several reasons.

首先，第 1-3 节中生活质量的提高，在其他条件相同的情况下，应能促进民主：历史上它们至少在某种程度上做到了。我特别期望心理健康、福祉和教育的改善会增加民主，因为这三者都与对威权领导人的支持呈负相关。一般来说，当人们的其他需求得到满足时，他们希望有更多的自我表达，而民主本身就是一种自我表达形式。相反，威权主义依赖于恐惧和怨恨。

First, the increases in quality of life in Sections 1-3 should, all things equal, promote democracy: historically they have, to at least some extent. In particular I expect improvements in mental health, well-being, and education to increase democracy, as all three are negativelycorrelated with support for authoritarian leaders. In general people want more self-expression when their other needs are met, and democracy is among other things a form of self-expression. Conversely, authoritarianism thrives on fear and resentment.

其次，只要威权主义者无法审查，自由信息确实有可能削弱威权主义。而且，未经审查的人工智能也可以为个人提供强大的工具来削弱压制性政府。压制性政府通过剥夺人们某种共同知识来生存，使他们无法意识到“皇帝没穿衣服”。例如，帮助推翻塞尔维亚米洛舍维奇政府的斯尔贾·波波维奇，撰写了大量关于从心理上剥夺威权主义者权力的技术，打破魔咒并团结起来反对独裁者。一个超级有效的、波波维奇式的人工智能版本（其技能似乎具有高智能回报）放在每个人的口袋里，而独裁者无力阻止或审查，这可以为世界各地的异见者和改革者创造一股顺风。再说一遍，这将是一场漫长而持久的斗争，一场胜利没有保证的斗争，但如果我们以正确的方式设计和构建人工智能，它至少可能是一场各地自由倡导者拥有优势的斗争。

Second, there is a good chance free information really does undermine authoritarianism, as long as the authoritarians can’t censor it. And uncensored AI can also bring individuals powerful tools for undermining repressive governments. Repressive governments survive by denying people a certain kind of common knowledge, keeping them from realizing that “the emperor has no clothes”. For example Srđa Popović), who helped to topple the Milošević government in Serbia, has written extensively about techniques for psychologically robbing authoritarians of their power, for breaking the spell and rallying support against a dictator. A superhumanly effective AI version of Popović (whose skills seem like they have high returns to intelligence) in everyone’s pocket, one that dictators are powerless to block or censor, could create a wind at the backs of dissidents and reformers across the world. To say it again, this will be a long and protracted fight, one where victory is not assured, but if we design and build AI in the right way, it may at least be a fight where the advocates of freedom everywhere have an advantage.

与神经科学和生物学一样，我们也可以问事情如何能“比正常更好”——不仅仅是避免威权主义，而是如何使民主国家比今天更好。即使在民主国家内部，不公正也时常发生。法治社会向其公民承诺，法律面前人人平等，每个人都有权享有基本人权，但显然人们在实践中并不总能获得这些权利。这一承诺即使部分实现也值得骄傲，但人工智能能帮助我们做得更好吗？

As with neuroscience and biology, we can also ask how things could be “better than normal”—not just how to avoid autocracy, but how to make democracies better than they are today. Even within democracies, injustices happen all the time. Rule-of-law societies make a promise to their citizens that everyone will be equal under the law and everyone is entitled to basic human rights, but obviously people do not always receive those rights in practice. That this promise is even partially fulfilled makes it something to be proud of, but can AI help us do better?

例如，人工智能能否通过使决策和过程更加公正来改善我们的法律和司法系统？如今，人们在法律或司法背景下主要担心人工智能系统会成为歧视的原因，这些担忧很重要，需要加以防范。与此同时，民主的活力取决于利用新技术改善民主制度，而不仅仅是应对风险。一个真正成熟和成功的人工智能实施有可能减少偏见，对每个人都更公平。

For example, could AI improve our legal and judicial system by making decisions and processes more impartial? Today people mostly worry in legal or judicial contexts that AI systems will be a _cause_ of discrimination, and these worries are important and need to be defended against. At the same time, the vitality of democracy depends on harnessing new technologies to improve democratic institutions, not just responding to risks. A truly mature and successful implementation of AI has the potential to _reduce_ bias and be fairer for everyone.

几个世纪以来，法律体系一直面临这样的困境：法律旨在公正，但本质上是主观的，因此必须由有偏见的人类来解释。试图使法律完全机械化行不通，因为现实世界是混乱的，不能总是用数学公式来捕捉。相反，法律体系依赖于众所周知的模糊标准，如“残酷和不寻常的惩罚”或“完全没有可取的社会重要性”，然后由人类解释——而且往往以显示偏见、偏袒或任意性的方式进行。加密货币中的“智能合约”并没有彻底改变法律，因为普通代码不够智能，无法裁决太多有趣的事情。但人工智能可能足够智能：它是第一个能够以可重复和机械的方式做出广泛、模糊判断的技术。

For centuries, legal systems have faced the dilemma that the law aims to be impartial, but is inherently subjective and thus must be interpreted by biased humans. Trying to make the law fully mechanical hasn’t worked because the real world is messy and can’t always be captured in mathematical formulas. Instead legal systems rely on notoriously imprecise criteria like “cruel and unusual punishment” or “utterly without redeeming social importance”, which humans then interpret—and often do so in a manner that displays bias, favoritism, or arbitrariness. “Smart contracts” in cryptocurrencies haven’t revolutionized law because ordinary code isn’t smart enough to adjudicate all that much of interest. But AI might be smart enough for this: it is the first technology capable of making broad, fuzzy judgements in a repeatable and mechanical way.

我并不是建议我们用人工智能系统直接取代法官，但公正性与理解和处理混乱现实世界的能力的结合，感觉应该在法律和正义方面有一些严肃的积极应用。至少，这样的系统可以与人一起作为决策辅助。透明度在任何这样的系统中都很重要，而成熟的人工智能科学有可能提供这一点：此类系统的训练过程可以广泛研究，先进的可解释性技术可用于查看最终模型内部并评估其隐藏的偏见，而这对于人类来说根本不可能。这样的人工智能工具也可以用于在司法或警察背景下监测基本权利的侵犯，使宪法更具自我执行性。

I am not suggesting that we literally replace judges with AI systems, but the combination of impartiality with the ability to understand and process messy, real world situations _feels_ like it should have some serious positive applications to law and justice. At the very least, such systems could work alongside humans as an aid to decision-making. Transparency would be important in any such system, and a mature science of AI could conceivably provide it: the training process for such systems could be extensively studied, and advanced interpretability techniques could be used to see inside the final model and assess it for hidden biases, in a way that is simply not possible with humans. Such AI tools could also be used to monitor for violations of fundamental rights in a judicial or police context, making constitutions more self-enforcing.

类似地，人工智能可用于聚合意见并推动公民之间的共识，解决冲突，寻找共同点，寻求妥协。计算民主项目已经在这方面进行了一些早期尝试，包括与 Anthropic 的合作。一个更知情、更有思想的公民显然会加强民主制度。

In a similar vein, AI could be used to both aggregate opinions and drive consensus among citizens, resolving conflict, finding common ground, and seeking compromise. Some early ideas in this direction have been undertaken by the computational democracy project, including collaborations with Anthropic. A more informed and thoughtful citizenry would obviously strengthen democratic institutions.

还有一个明显的机会，即利用人工智能帮助提供政府服务——如健康福利或社会服务——这些服务原则上人人可得，但实际上往往严重缺乏，且在某些地方比其他地方更差。这包括医疗服务、车管局、税务、社会保障、建筑规范执行等。拥有一个非常周到和知情的人工智能，其工作是让你以你能理解的方式获得你依法有权从政府获得的一切——并且还帮助你遵守通常令人困惑的政府规则——将是一件大事。提高国家能力既有助于实现法律面前人人平等的承诺，也有助于加强对民主治理的尊重。执行不力的服务目前是导致对政府愤世嫉俗的主要因素。

There is also a clear opportunity for AI to be used to help provision government services—such as health benefits or social services—that are in principle available to everyone but in practice often severely lacking, and worse in some places than others. This includes health services, the DMV, taxes, social security, building code enforcement, and so on. Having a very thoughtful and informed AI whose job is to give you everything you’re legally entitled to by the government in a way you can understand—and who also helps you comply with often confusing government rules—would be a big deal. Increasing state capacity both helps to deliver on the promise of equality under the law, and strengthens respect for democratic governance. Poorly implemented services are currently a major driver of cynicism about government27

所有这些都是有些模糊的想法，正如我在本节开头所说，我对它们的可行性远不如对生物学、神经科学和减贫方面的进展那么有信心。它们可能是不切实际的乌托邦。但重要的是要有一个雄心勃勃的愿景，愿意大胆梦想并尝试。人工智能作为自由、个人权利和法律面前人人平等的保障者的愿景，是一个太强大的愿景，不容不为之一战。一个 21 世纪、由人工智能驱动的政体，既可以成为个人自由的更强保护者，也可以成为帮助自由民主成为全世界都希望采用的政府形式的希望灯塔。

27 When the average person thinks of public institutions, they probably think of their experience with the DMV, IRS, medicare, or similar functions. Making these experiences more positive than they currently are seems like a powerful way to combat undue cynicism.

All of these are somewhat vague ideas, and as I said at the beginning of this section, I am not nearly as confident in their feasibility as I am in the advances in biology, neuroscience, and poverty alleviation. They may be unrealistically utopian. But the important thing is to have an ambitious vision, to be willing to dream big and try things out. The vision of AI as a guarantor of liberty, individual rights, and equality under the law is too powerful a vision not to fight for. A 21st century, AI-enabled polity could be both a stronger protector of individual freedom, and a beacon of hope that helps make liberal democracy the form of government that the whole world wants to adopt.

5. 工作与意义 5. Work and meaning

即使前四节的一切进展顺利——不仅疾病、贫困和不平等得到缓解，而且自由民主成为主导的政府形式，现有的自由民主国家也变得更美好——至少还有一个重要问题悬而未决。有人可能会反驳说：“我们生活在这样一个技术先进、公平体面的世界固然很好，但既然人工智能能做所有事情，人类如何获得意义？再者，他们如何在经济上生存？”

Even if everything in the preceding four sections goes well—not only do we alleviate disease, poverty, and inequality, but liberal democracy becomes the dominant form of government, and existing liberal democracies become better versions of themselves—at least one important question still remains. “It’s great we live in such a technologically advanced world as well as a fair and decent one”, someone might object, “but with AIs doing everything, how will humans have meaning? For that matter, how will they survive economically?”.

我认为这个问题比其他问题更难。我并不是说我对此必然比其他问题更悲观（尽管我确实看到了挑战）。我的意思是它更模糊，更难提前预测，因为它涉及社会如何组织的宏观问题，而这些问题往往只能随着时间的推移并以去中心化的方式自行解决。例如，历史上的狩猎采集社会可能会认为没有狩猎和各种与狩猎相关的宗教仪式，生活就毫无意义，并且会认为我们丰衣足食的技术社会缺乏目标。他们可能也不理解我们的经济如何养活所有人，或者在机械化社会中人们能发挥什么有用的作用。

I think this question is more difficult than the others. I don’t mean that I am necessarily more pessimistic about it than I am about the other questions (although I do see challenges). I mean that it is fuzzier and harder to predict in advance, because it relates to macroscopic questions about how society is organized that tend to resolve themselves only over time and in a decentralized manner. For example, historical hunter-gatherer societies might have imagined that life is meaningless without hunting and various kinds of hunting-related religious rituals, and would have imagined that our well-fed technological society is devoid of purpose. They might also have not understood how our economy can provide for everyone, or what function people can usefully serve in a mechanized society.

尽管如此，至少说几句话还是值得的，同时要记住，这一节的简短绝不意味着我不认真对待这些问题——恰恰相反，这表明缺乏明确的答案。

Nevertheless, it’s worth saying at least a few words, while keeping in mind that the brevity of this section is not at all to be taken as a sign that I don’t take these issues seriously—on the contrary, it is a sign of a lack of clear answers.

关于意义问题，我认为认为你承担的任务仅仅因为人工智能能做得更好就毫无意义，这很可能是一个错误。大多数人在任何事情上都不是世界最佳，但这似乎并没有特别困扰他们。当然，今天他们仍然可以通过比较优势做出贡献，并可能从他们创造的经济价值中获得意义，但人们也非常享受不产生经济价值的活动。我花大量时间玩电子游戏、游泳、在外面散步和与朋友聊天，所有这些都产生零经济价值。我可能会花一天时间试图在游戏中变得更好，或者在山地自行车上骑得更快，而对我来说，某人在某处比我擅长这些并不重要。无论如何，我认为意义主要来自人际关系和联系，而不是经济劳动。人们确实需要成就感，甚至竞争感，在人工智能后的世界里，完全有可能花数年时间尝试一些非常困难的任务，采用复杂的策略，类似于今天人们开始研究项目、试图成为好莱坞演员或创办公司时所做的事情。

On the question of meaning, I think it is very likely a mistake to believe that tasks you undertake are meaningless simply because an AI could do them better. Most people are not the best in the world at anything, and it doesn’t seem to bother them particularly much. Of course today they can still contribute through comparative advantage, and may derive meaning from the economic value they produce, but people also greatly enjoy activities that produce no economic value. I spend plenty of time playing video games, swimming, walking around outside, and talking to friends, all of which generates zero economic value. I might spend a day trying to get better at a video game, or faster at biking up a mountain, and it doesn’t really matter to me that someone somewhere is much better at those things. In any case I think meaning comes mostly from human relationships and connection, not from economic labor. People do want a sense of accomplishment, even a sense of competition, and in a post-AI world it will be perfectly possible to spend years attempting some very difficult task with a complex strategy, similar to what people do today when they embark on research projects, try to become Hollywood actors, or found companies28

28 事实上，在人工智能驱动的世界里，这类可能的挑战和项目的范围将比今天广阔得多。

28 Indeed, in an AI-powered world, the range of such possible challenges and projects will be much vaster than it is today.

（a）某处的人工智能原则上可以更好地完成这项任务，以及（b）这项任务不再是全球经济中受经济回报的元素，这些事实在我看来并不重要。

. The facts that (a) an AI somewhere could in principle do this task better, and (b) this task is no longer an economically rewarded element of a global economy, don’t seem to me to matter very much.

经济问题实际上比意义问题更难。在本节中，“经济”指的是可能的问题，即大多数或所有人类可能无法对足够先进的人工智能驱动的经济做出有意义的贡献。这是一个比不平等问题（特别是获取新技术的渠道不平等，我在第 3 节中讨论过）更宏观的问题。

The economic piece actually seems more difficult to me than the meaning piece. By “economic” in this section I mean the possible problem that _most or all_ humans may not be able to contribute meaningfully to a sufficiently advanced AI-driven economy. This is a more macro problem than the separate problem of inequality, especially inequality in access to the new technologies, which I discussed in Section 3.

首先，短期内我同意比较优势将继续使人类保持相关性，实际上提高他们的生产力，甚至可能在某种程度上为人类创造公平的竞争环境。只要人工智能只擅长某项工作的 90%，剩下的 10%就会使人类变得高度杠杆化，增加报酬，并实际上创造大量新的人类工作岗位，以补充和放大人工智能擅长的领域，使得“10%”扩大，继续雇佣几乎所有人。事实上，即使人工智能在 100%的事情上比人类做得更好，但如果它在某些任务上仍然低效或昂贵，或者如果人类和人工智能的资源投入有显著差异，那么比较优势的逻辑仍然适用。人类可能在相当长一段时间内保持相对（甚至绝对）优势的一个领域是物理世界。因此，我认为即使在达到“数据中心里的天才之国”之后，人类经济可能仍然有意义。

First of all, in the short term I agree with arguments that comparative advantage will continue to keep humans relevant and in fact increase their productivity, and may even in some ways level the playing field between humans. As long as AI is only better at 90% of a given job, the other 10% will cause humans to become highly leveraged, increasing compensation and in fact creating a bunch of new human jobs complementing and amplifying what AI is good at, such that the “10%” expands to continue to employ almost everyone. In fact, even if AI can do 100% of things better than humans, but it remains inefficient or expensive at some tasks, or if the resource _inputs_ to humans and AIs are meaningfully different, then the logic of comparative advantage continues to apply. One area humans are likely to maintain a relative (or even absolute) advantage for a significant time is the physical world. Thus, I think that the human economy may continue to make sense even a little past the point where we reach “a country of geniuses in a datacenter”.

然而，我确实认为从长远来看，人工智能将变得如此广泛有效且廉价，以至于这不再适用。到那时，我们当前的经济设置将不再有意义，需要更广泛的社会讨论如何组织经济。

However, I do think in the long run AI will become so broadly effective and so cheap that this will no longer apply. At that point our current economic setup will no longer make sense, and there will be a need for a broader societal conversation about how the economy should be organized.

虽然这听起来可能很疯狂，但事实是文明在过去成功应对了重大的经济转型：从狩猎采集到农业，从农业到封建主义，从封建主义到工业主义。我怀疑需要某种新的、更奇怪的东西，而且今天没有人能很好地设想它。它可能简单到为每个人提供大规模普遍基本收入，尽管我怀疑这只会是解决方案的一小部分。它可能是一个人工智能系统的资本主义经济，然后根据人工智能系统认为奖励人类有意义的东西（基于最终源自人类价值观的某种判断），通过某种次级经济向人类分配资源（大量资源，因为整体经济蛋糕将巨大）。也许经济靠 Whuffie 点数运行。或者，人类最终可能仍然具有经济价值，以通常经济模型未预料到的方式。所有这些解决方案都有大量可能的问题，没有大量的迭代和实验，不可能知道它们是否合理。与其他一些挑战一样，我们可能必须努力争取好的结果：剥削性或反乌托邦的方向显然也是可能的，必须加以阻止。关于这些问题可以写更多，我希望以后有机会这样做。

While that might sound crazy, the fact is that civilization has successfully navigated major economic shifts in the past: from hunting and gathering to farming, farming to feudalism, and feudalism to industrialism. I suspect that some new and stranger thing will be needed, and that it’s something no one today has done a good job of envisioning. It could be as simple as a large universal basic income for everyone, although I suspect that will only be a small part of a solution. It could be a capitalist economy of AI systems, which then give out resources (huge amounts of them, since the overall economic pie will be gigantic) to humans based on some secondary economy of what the AI systems think makes sense to reward in humans (based on some judgment ultimately derived from human values). Perhaps the economy runs on Whuffie points. Or perhaps humans will continue to be economically valuable after all, in some way not anticipated by the usual economic models. All of these solutions have tons of possible problems, and it’s not possible to know whether they will make sense without lots of iteration and experimentation. And as with some of the other challenges, we will likely have to fight to get a good outcome here: exploitative or dystopian directions are clearly also possible and have to be prevented. Much more could be written about these questions and I hope to do so at some later time.

现状评估 Taking stock

通过上述各种主题，我试图描绘一个世界的愿景：如果人工智能一切顺利，这个世界既有可能实现，又比今天的世界好得多。我不知道这个世界是否现实，即使现实，也需要许多勇敢而专注的人付出巨大努力和奋斗才能实现。每个人（包括人工智能公司！）都需要尽自己的一份力，既要防范风险，也要充分实现收益。

Through the varied topics above, I’ve tried to lay out a vision of a world that is both plausible _if_ everything goes right with AI, and much better than the world today. I don’t know if this world is realistic, and even if it is, it will not be achieved without a huge amount of effort and struggle by many brave and dedicated people. Everyone (including AI companies!) will need to do their part both to prevent risks and to fully realize the benefits.

但这是一个值得为之奋斗的世界。如果这一切真的在 5 到 10 年内发生——大多数疾病被击败，生物和认知自由增长，数十亿人摆脱贫困共享新技术，自由民主和人权的复兴——我怀疑每个目睹这一切的人都会对其产生的影响感到惊讶。我指的不是个人从所有新技术中受益的经历，尽管那当然会很惊人。我指的是目睹一套长期持有的理想在我们面前同时实现的那种体验。我认为许多人会因此感动得流泪。

But it is a world worth fighting for. If all of this really does happen over 5 to 10 years—the defeat of most diseases, the growth in biological and cognitive freedom, the lifting of billions of people out of poverty to share in the new technologies, a renaissance of liberal democracy and human rights—I suspect everyone watching it will be surprised by the effect it has on them. I don’t mean the experience of personally benefiting from all the new technologies, although that will certainly be amazing. I mean the experience of watching a long-held set of ideals materialize in front of us all at once. I think many will be literally moved to tears by it.

在撰写这篇文章的过程中，我注意到一种有趣的张力。一方面，这里描绘的愿景极其激进：几乎没有人期望在未来十年内发生，很可能被许多人视为荒谬的幻想。有些人甚至可能认为它不可取；它体现了并非所有人都认同的价值观和政治选择。但与此同时，它又有某种显而易见、甚至是被多重决定的东西，仿佛许多不同的美好世界构想都不可避免地大致指向这里。

Throughout writing this essay I noticed an interesting tension. In one sense the vision laid out here is extremely radical: it is not what almost anyone expects to happen in the next decade, and will likely strike many as an absurd fantasy. Some may not even consider it desirable; it embodies values and political choices that not everyone will agree with. But at the same time there is something blindingly obvious—something overdetermined—about it, as if many different attempts to envision a good world inevitably lead roughly here.

在伊恩·M·班克斯的《游戏玩家》中——

In Iain M. Banks’ _The Player of Games_29

29 我打破了自己不把文章写成科幻的规矩，但我发现很难完全不提及它。事实是，科幻小说是我们关于未来进行广泛思想实验的少数来源之一；我认为它如此深地与某个狭隘的亚文化纠缠在一起，这本身就说明了一些问题。

29 I am breaking my own rule not to make this about science fiction, but I’ve found it hard not to refer to it at least a bit. The truth is that science fiction is one of our only sources of expansive thought experiments about the future; I think it says something bad that it’s entangled so heavily with a particular narrow subculture.

——主角来自一个名为“文明”的社会，其原则与我在此阐述的并无不同——他前往一个压迫性的军国主义帝国，在那里领导权由一场复杂的战斗游戏中的竞争决定。然而，这个游戏足够复杂，以至于玩家的策略往往反映了他们自己的政治和哲学观点。主角在游戏中击败了皇帝，表明他的价值观（“文明”的价值观）即使在由基于残酷竞争和适者生存的社会设计的游戏中，也是一种制胜策略。斯科特·亚历山大的一篇著名文章也提出了同样的论点——竞争是自我挫败的，往往会导致一个基于同情与合作的社会。“道德宇宙的弧线”是另一个类似的概念。

, the protagonist—a member of a society called the Culture, which is based on principles not unlike those I’ve laid out here—travels to a repressive, militaristic empire in which leadership is determined by competition in an intricate battle game. The game, however, is complex enough that a player’s strategy within it tends to reflect their own political and philosophical outlook. The protagonist manages to defeat the emperor in the game, showing that his values (the Culture’s values) represent a winning strategy even in a game designed by a society based on ruthless competition and survival of the fittest. A well-known post by Scott Alexander has the same thesis—that competition is self-defeating and tends to lead to a society based on compassion and cooperation. The “arc of the moral universe” is another similar concept.

我认为“文明”的价值观是一种制胜策略，因为它们是数百万个具有明确道德力量的小决策的总和，这些决策倾向于将所有人拉到同一边。人类关于公平、合作、好奇心和自主的基本直觉是难以反驳的，并且以一种我们更具破坏性的冲动往往不具备的方式累积。很容易论证，如果我们能预防，孩子就不应该死于疾病，并且很容易由此论证每个孩子都同样有权享有这一权利。由此不难论证，我们应该团结起来，运用我们的智慧来实现这一结果。很少有人不同意人们应该因无故攻击或伤害他人而受到惩罚，并且由此不难得出惩罚应该在不同人之间一致且系统的想法。同样直观的是，人们应该对自己的生活和选择拥有自主权和责任。这些简单的直觉，如果推到逻辑结论，最终会导致法治、民主和启蒙价值观。如果不是必然，那么至少作为一种统计趋势，人类已经朝着这个方向前进。人工智能只是提供了一个机会，让我们更快地到达那里——使逻辑更清晰，目的地更明确。

I think the Culture’s values are a winning strategy because they’re the sum of a million small decisions that have clear moral force and that tend to pull everyone together onto the same side. Basic human intuitions of fairness, cooperation, curiosity, and autonomy are hard to argue with, and are cumulative in a way that our more destructive impulses often aren’t. It is easy to argue that children shouldn’t die of disease if we can prevent it, and easy from there to argue that _everyone’s_ children deserve that right equally. From there it is not hard to argue that we should all band together and apply our intellects to achieve this outcome. Few disagree that people should be punished for attacking or hurting others unnecessarily, and from there it’s not much of a leap to the idea that punishments should be consistent and systematic across people. It is similarly intuitive that people should have autonomy and responsibility over their own lives and choices. These simple intuitions, if taken to their logical conclusion, lead eventually to rule of law, democracy, and Enlightenment values. If not inevitably, then at least as a statistical tendency, this is where humanity was already headed. AI simply offers an opportunity to get us there more quickly—to make the logic starker and the destination clearer.

尽管如此，这是一件超凡之美的事物。我们有机会在其中扮演一个小角色，使之成为现实。

Nevertheless, it is a thing of transcendent beauty. We have the opportunity to play some small role in making it real.

感谢 Kevin Esvelt、Parag Mallick、Stuart Ritchie、Matt Yglesias、Erik Brynjolfsson、Jim McClave、Allan Dafoe 以及 Anthropic 的许多人对本文草稿的审阅。

_Thanks to Kevin Esvelt, Parag Mallick, Stuart Ritchie, Matt Yglesias, Erik Brynjolfsson, Jim McClave, Allan Dafoe, and many people at Anthropic for reviewing drafts of this essay._

致 2024 年诺贝尔化学奖得主，为我们指明了道路。

_To the winners of the2024 Nobel prize in Chemistry, for showing us all the way._