The impossibility of intelligence explosion
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→1. 一种源于对智能误解的有缺陷的推理。3. 我们的环境对个体智能设置了硬性限制。4. 我们的大部分智能并不在大脑中，而是外化为我们的文明。
1. A flawed reasoning that stems from a misunderstanding of intelligence 3. Our environment puts a hard limit on our individual intelligence 4. Most of our intelligence is not in our brain, it is externalized as our civilization
1. 一种源于对智能误解的有缺陷的推理
1. A flawed reasoning that stems from a misunderstanding of intelligence
3. 我们的环境对我们的个体智能设置了硬性限制
3. Our environment puts a hard limit on our individual intelligence
4. 我们的大部分智能不在大脑中，而是外化为我们的文明
4. Most of our intelligence is not in our brain, it is externalized as our civilization
5. 个体大脑无法实现递归智能增强
5. An individual brain cannot implement recursive intelligence augmentation
6. 我们对递归自我改进系统的了解
6. What we know about recursively self-improving systems
[](https://medium.com/@francois.chollet?source=post_page---byline--5be4a9eda6ec---------------------------------------)
《超验骇客》（2014 年科幻电影）
Transcendence (2014 science-fiction movie)
1965 年，I. J. Good 首次描述了与人工智能相关的“智能爆炸”概念：
In 1965, I. J. Good described for the first time the notion of “intelligence explosion”, as it relates to artificial intelligence (AI):
几十年后，“智能爆炸”的概念——导致“超级智能”突然崛起并意外终结人类——已在 AI 社区中扎根。知名商业领袖将其视为重大风险，甚至超过核战争或气候变化。普通的机器学习研究生也在支持这一观点。在 2015 年针对 AI 研究人员的电子邮件调查中，29%的受访者认为智能爆炸“可能”或“非常可能”。另有 21%的人认为这是一个严重的可能性。
Decades later, the concept of an “intelligence explosion” — leading to the sudden rise of “superintelligence” and the accidental end of the human race — has taken hold in the AI community. Famous business leaders are casting it as a major risk, greater than nuclear war or climate change. Average graduate students in machine learning are endorsing it. In a 2015 email survey targeting AI researchers, 29% of respondents answered that intelligence explosion was “likely” or “highly likely”. A further 21% considered it a serious possibility.
基本前提是，在不久的将来，第一个“种子 AI”将被创造出来，其通用问题解决能力略超人类。这个种子 AI 将开始设计更好的 AI，启动递归自我改进循环，立即将人类智能远远甩在身后，在短时间内以数量级超越。该理论的支持者还将智能视为一种超能力，赋予其拥有者近乎超自然的能力来塑造环境——例如在科幻电影《超验骇客》（2014 年）中所见。因此，超级智能意味着近乎全能，并将对人类构成生存威胁。
The basic premise is that, in the near future, a first “seed AI” will be created, with general problem-solving abilities slightly surpassing that of humans. This seed AI would start designing better AIs, initiating a recursive self-improvement loop that would immediately leave human intelligence in the dust, overtaking it by orders of magnitude in a short time. Proponents of this theory also regard intelligence as a kind of superpower, conferring its holders with almost supernatural capabilities to shape their environment — as seen in the science-fiction movie Transcendence (2014),) for instance. Superintelligence would thus imply near-omnipotence, and would pose an existential threat to humanity.
这种科幻叙事助长了当前关于 AI 风险和监管必要性的危险误导性公众辩论。在这篇文章中，我认为智能爆炸是不可能的——智能爆炸的概念源于对智能本质和递归自我增强系统行为的深刻误解。我试图基于对智能系统和递归系统的具体观察来阐述我的观点。
This science-fiction narrative contributes to the dangerously misleading public debate that is ongoing about the risks of AI and the need for AI regulation. In this post, I argue that intelligence explosion is impossible — that the notion of intelligence explosion comes from a profound misunderstanding of both the nature of intelligence and the behavior of recursively self-augmenting systems. I attempt to base my points on concrete observations about intelligent systems and recursive systems.
智能爆炸背后的推理，如同 20 世纪 60 年代和 70 年代出现的许多早期 AI 理论一样，是诡辩式的：它完全抽象地考虑“智能”，脱离其背景，并忽略了关于智能系统和递归自我改进系统的现有证据。情况不必如此。毕竟，我们生活在一个充满智能系统（包括我们）和自我改进系统的星球上，因此我们可以直接观察它们并向它们学习，以回答手头的问题，而不是提出无证据的循环推理。
The reasoning behind intelligence explosion, like many of the early theories about AI that arose in the 1960s and 1970s, is sophistic: it considers “intelligence” in a completely abstract way, disconnected from its context, and ignores available evidence about both intelligent systems and recursively self-improving systems. It doesn’t have to be that way. We are, after all, on a planet that is literally packed with intelligent systems (including us) and self-improving systems, so we can simply observe them and learn from them to answer the questions at hand, instead of coming up with evidence-free circular reasonings.
要讨论智能及其可能的自我改进特性，我们首先需要介绍必要的背景和上下文。当我们谈论智能时，我们在谈论什么？精确定义智能本身就是一个挑战。智能爆炸的叙事将智能等同于_个体智能体所展现的通用问题解决能力_——由当前的人类大脑或未来的电子大脑所展现。这并非全貌，所以让我们以此定义作为起点，并对其进行扩展。
To talk about intelligence and its possible self-improving properties, we should first introduce necessary background and context. What are we talking about when we talk about intelligence? Precisely defining intelligence is in itself a challenge. The intelligence explosion narrative equates intelligence with _the general problem-solving ability displayed by individual intelligent agents_ — by current human brains, or future electronic brains. This is not quite the full picture, so let’s use this definition as a starting point, and expand on it.
我认为智能爆炸理论的第一个问题是未能认识到智能必然是一个更广泛系统的一部分——一种将智能视为“缸中之脑”的观点，认为它可以脱离其情境而任意变得智能。大脑只是一块生物组织，本身并无内在的智能。除了你的大脑，你的身体和感官——你的感觉运动可供性——是你心智的基本组成部分。你的环境是你心智的基本组成部分。人类文化是你心智的基本组成部分。毕竟，这些都是你所有思想的来源。你不能将智能与其表达的情境分离开来。
The first issue I see with the intelligence explosion theory is a failure to recognize that intelligence is necessarily part of a broader system — a vision of intelligence as a “brain in jar” that can be made arbitrarily intelligent independently of its situation. A brain is just a piece of biological tissue, there is nothing intrinsically intelligent about it. Beyond your brain, your body and senses — your sensorimotor affordances — are a fundamental part of your mind. Your environment is a fundamental part of your mind. Human culture is a fundamental part of your mind. These are, after all, where all of your thoughts come from. You cannot dissociate intelligence from the context in which it expresses itself.
特别是，不存在所谓的“通用”智能。在抽象层面上，我们通过“没有免费午餐”定理知道这是一个事实——该定理指出，没有一种问题求解算法能在所有可能的问题上优于随机猜测。如果智能是一种问题求解算法，那么它只能相对于特定问题来理解。更具体地说，我们可以从经验上观察到，我们所知道的所有智能系统都是高度专业化的。我们今天构建的 AI 的智能在极其狭窄的任务上是高度专业化的——比如下围棋，或将图像分类为 10,000 个已知类别。章鱼的智能专门用于成为章鱼的问题。人类的智能专门用于成为人类的问题。
In particular, there is no such thing as “general” intelligence. On an abstract level, we know this for a fact via the “no free lunch” theorem— stating that no problem-solving algorithm can outperform random chance across _all_ possible problems. If intelligence is a problem-solving algorithm, then it can only be understood with respect to a _specific_ problem. In a more concrete way, we can observe this empirically in that all intelligent systems we know are highly specialized. The intelligence of the AIs we build today is hyper specialized in extremely narrow tasks — like playing Go, or classifying images into 10,000 known categories. The intelligence of an octopus is specialized in the problem of being an octopus. The intelligence of a human is specialized in the problem of being human.
如果我们把一个刚创造出来的人类大脑放进章鱼的身体里，让它生活在海底，会发生什么？它甚至能学会使用它的八条腿身体吗？它能存活几天吗？我们无法进行这个实验，但我们确实知道人类和动物的认知发展是由硬编码的、内在的动力学驱动的。人类婴儿天生就有一套高级的反射行为和内在的学习模板，这些模板驱动着他们早期的感觉运动发展，并且与人类感觉运动空间的结构根本性地交织在一起。大脑硬编码了关于拥有一个可以抓握的手、一个可以吸吮的嘴、安装在移动头部上的眼睛（前庭眼反射）等概念，这些预概念是人类智能开始控制人类身体所必需的。甚至有人令人信服地论证过，例如乔姆斯基，非常高层次的人类认知特征，比如我们发展语言的能力，是天生的。
What would happen if we were to put a freshly-created human brain in the body of an octopus, and let in live at the bottom of the ocean? Would it even learn to use its eight-legged body? Would it survive past a few days? We cannot perform this experiment, but we do know that cognitive development in humans and animals is driven by hardcoded, innate dynamics. Human babies are born with an advanced set of reflex behaviors and innate learning templates that drive their early sensorimotor development, and that are fundamentally intertwined with the structure of the human sensorimotor space. The brain has hardcoded conceptions of having a body with hands that can grab, a mouth that can suck, eyes mounted on a moving head that can be used to visually follow objects (the vestibulo-ocular reflex), and these preconceptions are required for human intelligence to start taking control of the human body. It has even been convincingly argued, for instance by Chomsky, that very high-level human cognitive features, such as our ability to develop language, are innate.
类似地，可以想象章鱼也有自己的一套硬编码的认知原语，这些原语是学习如何使用章鱼身体并在其章鱼环境中生存所必需的。人类的大脑在人类条件上是高度专业化的——一种天生的专业化可能延伸到社会行为、语言和常识——而章鱼的大脑同样会在章鱼行为上高度专业化。一个适当移植到章鱼身体里的人类婴儿大脑很可能无法充分控制其独特的感觉运动空间，并会很快死亡。现在不那么聪明了吧，优越的大脑先生。
Similarly, one can imagine that the octopus has its own set of hardcoded cognitive primitives required in order to learn how to use an octopus body and survive in its octopus environment. The brain of a human is hyper specialized in the human condition — an innate specialization extending possibly as far as social behaviors, language, and common sense — and the brain of an octopus would likewise be hyper specialized in octopus behaviors. A human baby brain properly grafted in an octopus body would most likely fail to adequately take control of its unique sensorimotor space, and would quickly die off. Not so smart now, Mr. Superior Brain.
如果我们把一个人类——包括大脑和身体——放入一个没有我们所知的人类文化的环境中，会发生什么？由狼群抚养长大的狼孩毛克利，会长大后比他的犬类兄弟姐妹更聪明吗？像我们一样聪明？如果我们把婴儿毛克利和婴儿爱因斯坦互换，他最终会自我教育，发展出关于宇宙的伟大理论吗？经验证据相对稀少，但据我们所知，在人类文化的养育环境之外长大的孩子不会发展出任何人类智能。从小在野外长大的野孩子实际上变成了动物，当回到文明社会时，再也无法获得人类行为或语言。在南非被猴子养大的 Saturday Mthiyane，五岁时被发现，直到成年都保持着像猴子一样的行为——跳跃和四肢行走，没有语言能力，拒绝吃熟食。那些在成长关键期至少有一部分时间有人类接触的野孩子，在再教育方面运气稍好一些，尽管他们很少能成为完全正常的人类。
What would happen if we were to put a human — brain and body — into an environment that does not feature human culture as we know it? Would Mowgli the man-cub, raised by a pack of wolves, grow up to outsmart his canine siblings? To be smart like us? And if we swapped baby Mowgli with baby Einstein, would he eventually educate himself into developing grand theories of the universe? Empirical evidence is relatively scarce, but from what we know, children that grow up outside of the nurturing environment of human culture don’t develop any human intelligence. Feral children raised in the wild from their earliest years become effectively animals, and can no longer acquire human behaviors or language when returning to civilization. Saturday Mthiyane, raised by monkeys in South Africa and found at five, kept behaving like a monkey into adulthood — jumping and walking on all four, incapable of language, and refusing to eat cooked food. Feral children who have human contact for at least some of their most formative years tend to have slightly better luck with reeducation, although they rarely graduate to fully-functioning humans.
如果智能从根本上与特定的感觉运动模态、特定的环境、特定的成长经历和特定的待解决问题相关联，那么你不能仅仅通过调整大脑来任意增加智能体的智能——就像你不能通过加快传送带速度来提高生产线的吞吐量一样。智能的扩展只能来自心智、其感觉运动模态及其环境的共同进化。如果你的大脑齿轮是你问题解决能力的决定性因素，那么那些智商远超人类正常范围的人就会过着远超正常范围的生活，解决以前认为无法解决的问题，并接管世界——就像一些人担心比人类更聪明的 AI 会做的那样。实际上，具有非凡认知能力的天才通常过着极其平凡的生活，很少有人取得任何显著成就。在特曼里程碑式的《天才的遗传研究》中，他指出，他大多数异常有天赋的受试者会从事“像警察、海员、打字员和档案管理员这样卑微的职业”。目前大约有 700 万人智商超过 150——比 99.9%的人类认知能力更好——而这些人大多不是你在新闻中读到的人。在那些真正试图接管世界的人中，几乎没有人似乎拥有非凡的智能；轶事上，希特勒是一个高中辍学生，两次未能考入维也纳艺术学院。
If intelligence is fundamentally linked to specific sensorimotor modalities, a specific environment, a specific upbringing, and a specific problem to solve, _then you cannot hope to arbitrarily increase the intelligence of an agent merely by tuning its brain — no more than you can increase the throughput of a factory line by speeding up the conveyor belt_. Intelligence expansion can only come from a co-evolution of the mind, its sensorimotor modalities, and its environment. If the gears of your brain were the defining factor of your problem-solving ability, then those rare humans with IQs far outside the normal range of human intelligence would live lives far outside the scope of normal lives, would solve problems previously thought unsolvable, and would take over the world — just as some people fear smarter-than-human AI will do. In practice, geniuses with exceptional cognitive abilities usually live overwhelmingly banal lives, and very few of them accomplish anything of note. In Terman’s landmark “_Genetic Studies of Genius”,_ he notes that most of his exceptionally gifted subjects would pursue occupations _“as humble as those of policeman, seaman, typist and filing clerk”_. There are currently about seven million people with IQs higher than 150 — better cognitive ability than 99.9% of humanity — and mostly, these are not the people you read about in the news.Of the people who have actually attempted to take over the world, hardly any seem to have had an exceptional intelligence; anecdotally, Hitler was a high-school dropout, who failed to get into the Vienna Academy of Art — twice.
那些最终在难题上取得突破的人是通过环境、性格、教育、智力的结合，并且他们通过在前人工作的基础上逐步改进而取得突破。成功——即表现出来的智能——是足够的能力在正确的时间遇到一个伟大的问题。大多数这些卓越的问题解决者甚至并不那么聪明——他们的技能似乎专门针对某个领域，并且通常在自己领域之外并不表现出高于平均水平的能力。有些人取得更多成就，是因为他们是更好的团队合作者，或者有更多的毅力和职业道德，或者更丰富的想象力。有些人只是恰好生活在正确的环境中，在正确的时间进行了正确的对话。智能从根本上来说是情境性的。
People who do end up making breakthroughs on hard problems do so through a combination of circumstances, character, education, intelligence, and they make their breakthroughs through incremental improvement over the work of their predecessors. _Success — expressed intelligence — is sufficient ability meeting a great problem at the right time._ Most of these remarkable problem-solvers are not even _that_ clever — their skills seem to be specialized in a given field and they typically do not display greater-than-average abilities outside of their own domain. Some people achieve more because they were better team players, or had more grit and work ethic, or greater imagination. Some just happened to have lived in the right context, to have the right conversation at the right time. Intelligence is fundamentally situational.
智能并非超能力；卓越的智能本身并不会赋予你与其成比例的超凡掌控环境的能力。然而，一个充分记录的事实是，原始认知能力——如通过智商（IQ）衡量，尽管这可能存在争议——在接近平均值的谱系段中与社会成就相关。这一点最初由特曼的研究证实，后来也被其他人确认——例如，Strenze 在 2006 年的一项广泛元分析发现，智商与社会经济成功之间存在虽弱但可见的相关性。因此，统计上，智商 130 的人比智商 70 的人更有可能成功应对生活问题——尽管在个体层面这从未得到保证——但关键在于：这种相关性在某个点之后就会失效。没有证据表明智商 170 的人比智商 130 的人更有可能在其领域取得更大影响。事实上，许多最具影响力的科学家的智商往往在 120 到 130 之间——费曼报告为 126，DNA 共同发现者詹姆斯·沃森为 124——这与大量平庸科学家的智商范围完全相同。与此同时，在当今大约 5 万名智商高达 170 或更高的人类中，有多少人能解决沃森教授所解决问题十分之一重要的问题？
Intelligence is not a superpower; exceptional intelligence does not, on its own, confer you with proportionally exceptional power over your circumstances. However, it is a well-documented fact that raw cognitive ability — as measured by IQ, which may be debatable — correlates with social attainment for slices of the spectrum that are close to the mean. This was first evidenced in Terman’s study, and later confirmed by others — for instance, an extensive 2006 metastudy by Strenze found a visible, if somewhat weak, correlation between IQ and socioeconomic success. So, a person with an IQ of 130 is statistically far more likely to succeed in navigating the problem of life than a person with an IQ of 70 — although this is never guaranteed at the individual level — but here’s the thing: this correlation breaks down after a certain point. There is no evidence that a person with an IQ of 170 is in any way more likely to achieve a greater impact in their field than a person with an IQ of 130. In fact, many of the most impactful scientists tend to have had IQs in the 120s or 130s — Feynman reported 126, James Watson, co-discoverer of DNA, 124 — which is exactly the same range as legions of mediocre scientists. At the same time, of the roughly 50,000 humans alive today who have astounding IQs of 170 or higher, how many will solve any problem a tenth as significant as Professor Watson?
为什么原始认知能力的现实效用会在超过某个阈值后停滞？这指向一个非常直观的事实：高成就需要_足够_的认知能力，但当前解决问题的_瓶颈_，即_表达出的智能_，并非潜在的认知能力本身。瓶颈在于我们的环境。我们的环境决定了智能如何显现，对我们的脑力所能达到的成就——即我们能成长为多聪明的人、我们能多有效地利用所发展的智能、我们能解决什么问题——施加了硬性限制。所有证据都表明，我们当前的环境，如同过去 20 万年人类历史和史前时期的环境一样，不允许高智能个体充分发展和利用其认知潜力。一万年前一个高潜力的人类会在低复杂度的环境中长大，可能只会说一种词汇量少于 5000 的语言，从未被教导读写，接触有限的知识和很少的认知挑战。对于大多数当代人类来说，情况稍好一些，但没有迹象表明我们的环境机会目前超过了我们的认知潜力。
Why would the real-world utility of raw cognitive ability stall past a certain threshold? This points to a very intuitive fact: that high attainment requires _sufficient_ cognitive ability, but that the current _bottleneck_ to problem-solving, to _expressed intelligence,_ is not latent cognitive ability itself. The bottleneck is our circumstances. Our environment, which determines how our intelligence manifests itself, puts a hard limit on what we can do with our brains — on how intelligent we can grow up to be, on how effectively we can leverage the intelligence that we develop, on what problems we can solve. All evidence points to the fact that our current environment, much like past environments over the previous 200,000 years of human history and prehistory, does not allow high-intelligence individuals to fully develop and utilize their cognitive potential. A high-potential human 10,000 years ago would have been raised in a low-complexity environment, likely speaking a single language with fewer than 5,000 words, would never have been taught to read or write, would have been exposed to a limited amount of knowledge and to few cognitive challenges. The situation is a bit better for most contemporary humans, but there is no indication that our environmental opportunities currently outpace our cognitive potential.
一个在丛林中长大的聪明人类不过是一只无毛猿。同样，一个拥有超人脑的人工智能，如果被放入现代世界的人类身体中，很可能不会发展出比聪明当代人类更强的能力。如果它能，那么异常高智商的人类就已经会展现出与其成比例的超凡个人成就；他们会实现对其环境的异常控制，并解决重大的未解决问题——而实际上他们并没有。
A smart human raised in the jungle is but a hairless ape. Similarly, an AI with a superhuman brain, dropped into a human body in our modern world, would likely not develop greater capabilities than a smart contemporary human. If it could, then exceptionally high-IQ humans would already be displaying proportionally exceptional levels of personal attainment; they would achieve exceptional levels of control over their environment, and solve major outstanding problems— which they don’t in practice.
不仅仅是我们的身体、感官和环境决定了我们大脑能发展出多少智能——关键在于，我们的生物大脑只是我们整体智能的一小部分。认知假肢环绕着我们，接入我们的大脑并扩展其解决问题的能力。你的智能手机。你的笔记本电脑。谷歌搜索。你在学校获得的认知工具。书籍。其他人。数学符号。编程。所有认知假肢中最基础的当然是语言本身——它本质上是认知的操作系统，没有它我们无法思考得太远。这些东西不仅仅是供大脑吸收和使用的_知识_，它们实际上是_外部认知过程_，是非生物的方式，用于运行思维线程和问题解决算法——跨越时间、空间，更重要的是，跨越个体。这些认知假肢，而非我们的大脑，承载了我们大部分的认知能力。
It’s not just that our bodies, senses, and environment determine how much intelligence our brains can develop — crucially, our biological brains are just a small part of our whole intelligence. Cognitive prosthetics surround us, plugging into our brain and extending its problem-solving capabilities. Your smartphone. Your laptop. Google search. The cognitive tools your were gifted in school. Books. Other people. Mathematical notation. Programing. The most fundamental of all cognitive prosthetics is of course language itself — essentially an operating system for cognition, without which we couldn’t think very far. These things are not merely _knowledge_ to be fed to the brain and used by it, they are literally _external cognitive processes_, non-biological ways to run threads of thought and problem-solving algorithms — across time, space, and importantly, across individuality. These cognitive prosthetics, not our brains, are where most of our cognitive abilities reside.
我们就是我们的工具。单个个体人类基本上是无用的——再说一次，人类只是双足猿。正是数千年来知识和外部系统的集体积累——我们称之为“文明”——将我们提升到了动物本性之上。当一位科学家取得突破时，他们大脑中运行的思维过程只是方程的一小部分——研究者将问题解决过程的大部分卸载到计算机、其他研究者、纸质笔记、数学符号等上。而且他们之所以能成功，是因为站在巨人的肩膀上——他们自己的工作只是跨越数十年和数千人的问题解决过程中的最后一个子程序。他们个人的认知工作对整个过程的重要性可能并不比芯片上的单个晶体管更大。
We are our tools. An individual human is pretty much useless on its own — again, humans are just bipedal apes. It’s a collective accumulation of knowledge and external systems over thousands of years — what we call “civilization” — that has elevated us above our animal nature. When a scientist makes a breakthrough, the thought processes they are running in their brain are just a small part of the equation — the researcher offloads large extents of the problem-solving process to computers, to other researchers, to paper notes, to mathematical notation, etc. And they are only able to succeed because they are standing on the shoulder of giants — their own work is but one last subroutine in a problem-solving process that spans decades and thousands of individuals. Their own individual cognitive work may not be much more significant to the whole process than the work of a single transistor on a chip.
大量证据指向一个简单事实：单个人类大脑本身无法设计出比自己更强大的智能。这纯粹是一个经验性陈述：在数十亿个来来去去的人类大脑中，没有一个做到过。显然，单个人类在单一生命周期内的智能无法设计智能，否则在数十亿次尝试中，它早已发生。
An overwhelming amount of evidence points to this simple fact: a single human brain, on its own, is not capable of designing a greater intelligence than itself. This is a purely empirical statement: out of billions of human brains that have come and gone, none has done so. Clearly, the intelligence of a single human, over a single lifetime, cannot design intelligence, or else, over billions of trials, it would have already occurred.
免费加入 Medium，即可获取这位作者的最新动态。
Join Medium for free to get updates from this writer.
然而，这数十亿个大脑，在数千年间积累知识并发展出外部智能过程，实现了一个系统——文明——它最终可能产生比单个人类更智能的人工大脑。是文明作为一个整体将创造超人类 AI，而不是你，也不是我，也不是任何个体。这是一个涉及无数人类的过程，跨越我们几乎无法理解的时间尺度。一个涉及远多于生物智能的_外部化智能_——书籍、计算机、数学、科学、互联网——的过程。在个体层面，我们只是文明的载体，在前人工作的基础上建设并传递我们的发现。我们是文明的问题解决算法运行的瞬时晶体管。
However, these billions of brains, accumulating knowledge and developing external intelligent processes over thousand of years, implement a system — civilization — which may eventually lead to artificial brains with greater intelligence than that of a single human. It is civilization as a whole that will create superhuman AI, not you, nor me, nor any individual. A process involving countless humans, over timescales we can barely comprehend. A process involving far more _externalized intelligence_ — books, computers, mathematics, science, the internet — than _biological intelligence_. On an individual level, we are but vectors of civilization, building upon previous work and passing on our findings. We are the momentary transistors on which the problem-solving algorithm of civilization runs.
未来由集体历经数世纪开发的超人类 AI，是否具备开发出比自身更强大的 AI 的能力？不，就像我们任何人都无法做到一样。回答“是”将违背我们所知的一切——再次记住，没有任何人类，也没有任何我们所知的智能实体，曾设计出比自身更聪明的东西。我们所做的是，逐渐地、集体地构建比我们自身更强大的外部问题解决系统。
Will the superhuman AIs of the future, developed collectively over centuries, have the capability to develop AI greater than themselves? No, no more than any of us can. Answering “yes” would fly in the face of everything we know — again, remember that no human, nor any intelligent entity that we know of, has ever designed anything smarter than itself. What we do is, gradually, collectively, build external problem-solving systems that are greater than ourselves.
然而，未来的 AI，就像人类和我们迄今产生的其他智能系统一样，将为我们的文明做出贡献，而我们的文明反过来将利用它们来不断扩展其所生产的 AI 的能力。从这个意义上说，AI 与计算机、书籍或语言本身并无不同：它是一种赋能我们文明的技术。因此，超人类 AI 的出现不会比计算机、书籍或语言的出现更具奇点性。文明将发展 AI，并继续前行。文明最终将超越我们现在的样子，就像它已经超越了一万年前的我们一样。这是一个渐进的过程，而非突然的转变。
However, future AIs, much like humans and the other intelligent systems we’ve produced so far, will contribute to our civilization, and our civilization, in turn, will use them to keep expanding the capabilities of the AIs it produces. AI, in this sense, is no different than computers, or books, or language itself: it’s a technology that empowers our civilization. The advent of superhuman AI will thus be no more of a singularity than the advent of computers, or books, or language. Civilization will develop AI, and just march on. Civilization will eventually transcend what we are now, much like it has transcended what we were 10,000 years ago. It’s a gradual process, not a sudden shift.
智能爆炸的基本前提——即一个“种子 AI”将出现，具有超越人类的问题解决能力，导致突然的、递归的、失控的智能改进循环——是错误的。我们的问题解决能力（特别是我们设计 AI 的能力）已经在不断改进，因为这些能力并非主要存在于我们的生物大脑中，而是存在于我们外部的、集体的工具中。递归循环已经运行了很长时间，“更好的大脑”的出现不会在性质上影响它——不会比任何以前的智能增强技术影响更大。我们的大脑本身从来就不是 AI 设计过程中的重大瓶颈。
The basic premise of intelligence explosion — that a “seed AI” will arise, with greater-than-human problem solving ability, leading to a sudden, recursive, runaway intelligence improvement loop — is false. Our problem-solving abilities (in particular, our ability to design AI) are already constantly improving, because these abilities do not reside primarily in our biological brains, but in our external, collective tools. The recursive loop has been in action for a long time, and the rise of “better brains” will not qualitatively affect it — no more than any previous intelligence-enhancing technology. Our brains themselves were never a significant bottleneck in the AI-design process.
在这种情况下，你可能会问，文明本身不就是失控的自我改进大脑吗？我们的文明智能正在爆炸吗？不。关键在于，文明层面的智能改进循环只导致了我们问题解决能力随时间_线性_的进步。而不是爆炸。但为什么呢？递归改进 X 在数学上难道不会导致 X 指数增长吗？不——简而言之，因为没有复杂的现实世界系统可以建模为 X(t + 1) = X(t) * a, a > 1。没有系统存在于真空中，智能和人类文明尤其如此。
In this case, you may ask, isn’t civilization itself the runaway self-improving brain? Is our civilizational intelligence exploding? No. Crucially, the civilization-level intelligence-improving loop has only resulted in measurably _linear_ progress in our problem-solving abilities over time. Not an explosion. But why? Wouldn’t recursively improving X mathematically result in X growing exponentially? No — in short, because no complex real-world system can be modeled as X(t + 1) = X(t) * a, a > 1. No system exists in a vacuum, and especially not intelligence, nor human civilization.
我们无需猜测，当一个智能系统开始优化自身智能时，是否会发生“爆炸”。事实上，大多数系统都是递归自我改进的。我们周围到处都是这样的系统。因此，我们确切知道这些系统在不同情境和不同时间尺度上的行为方式。你自己就是一个递归自我改进的系统：教育自己让你变得更聪明，进而让你能更有效地教育自己。同样，人类文明在更长的时间尺度上也是递归自我改进的。机电一体化是递归自我改进的——更好的制造机器人可以制造出更好的制造机器人。军事帝国是递归自我扩张的——你的帝国越大，你用来进一步扩张的军事手段就越强。个人投资是递归自我改进的——你拥有的钱越多，你能赚到的钱就越多。这样的例子比比皆是。
We don’t have to speculate about whether an “explosion” would happen the moment an intelligent system starts optimizing its own intelligence. As it happens, _most_ systems are recursively self-improving. We’re surrounded with them. So we know exactly how such systems behave — in a variety of contexts and over a variety of timescales. You are, yourself, a recursively self-improving system: educating yourself makes you smarter, in turn allowing you to educate yourself more efficiently. Likewise, human civilization is recursively self-improving, over a much longer timescale. Mechatronics is recursively self-improving — better manufacturing robots can manufacture better manufacturing robots. Military empires are recursively self-expanding — the larger your empire, the greater your military means to expand it further. Personal investing is recursively self-improving — the more money you have, the more money you can make. Examples abound.
以软件为例。编写软件显然能增强软件编写能力：首先，我们编写了编译器，它能执行“自动化编程”；然后，我们用编译器开发了实现更强大编程范式的新语言；我们用这些语言开发了先进的开发者工具——调试器、集成开发环境、代码检查工具、错误预测器。未来，软件甚至将自行编写。
Consider, for instance, software. Writing software obviously empowers software-writing: first, we programmed compilers, that could perform “automated programming”, then we used compilers to develop new languages implementing more powerful programming paradigms. We used these languages to develop advanced developer tools — debuggers, IDEs, linters, bug predictors. In the future, software will even write itself.
这个递归自我改进过程的最终结果是什么？你能用电脑上的软件比去年多做一倍的事情吗？明年你能多做一倍吗？可以说，软件的实用性一直以可测量的线性速度提升，而我们却投入了指数级的努力来生产它。几十年来，软件开发者的数量呈指数级增长，运行软件的晶体管数量也按照摩尔定律爆炸式增长。然而，与 2012 年、2002 年或 1992 年相比，我们的电脑对我们来说只是略有提升。
And what is the end result of this recursively self-improving process? Can you do 2x more with your the software on your computer than you could last year? Will you be able to do 2x more next year? Arguably, the usefulness of software has been improving at a measurably linear pace, while we have invested exponential efforts into producing it. The number of software developers has been booming exponentially for decades, and the number of transistors on which we are running our software has been exploding as well, following Moore’s law. Yet, our computers are only incrementally more useful to us than they were in 2012, or 2002, or 1992.
但为什么呢？主要是因为软件的实用性从根本上受到其应用环境的限制——就像智能既由其表达的环境定义，也受其限制一样。软件只是更大过程（我们的经济、我们的生活）中的一个齿轮，就像你的大脑只是更大过程（人类文化）中的一个齿轮。这个环境对软件的最大潜在实用性设置了硬性限制，就像我们的环境对任何个体的智能设置了硬性限制一样——即使拥有超人的大脑也是如此。
But why? Primarily, because the usefulness of software is fundamentally limited by the _context_ of its application — much like intelligence is both defined and limited by the context in which it expresses itself. Software is just one cog in a bigger process — our economies, our lives — just like your brain is just one cog in a bigger process — human culture. This context puts a hard limit on the maximum potential usefulness of software, much like our environment puts a hard limit on how intelligent any individual can be — even if gifted with a superhuman brain.
除了环境硬性限制之外，即使系统的一部分具有递归自我改进的能力，系统的其他部分也必然会开始成为瓶颈。对抗性过程会因递归自我改进而产生并压制它——在软件中，这表现为资源消耗、功能蔓延、用户体验问题。就个人投资而言，你自己的消费率就是这样一个对抗性过程——你拥有的钱越多，你花的钱就越多。就智能而言，系统间通信成为底层模块改进的制动器——一个拥有更聪明部分的大脑会更难协调它们；一个拥有更聪明个体的社会需要在网络和通信上投入更多，等等。高智商人群更容易患上某些精神疾病，这或许并非巧合。过去的军事帝国在达到一定规模后崩溃，也或许不是偶然。指数级进步，遇到指数级摩擦。
Beyond contextual hard limits, even if one part of a system has the ability to recursively self-improve, other parts of the system will inevitably start acting as bottlenecks. Antagonistic processes will arise in response to recursive self-improvement and squash it — in software, this would be resource consumption, feature creep, UX issues. When it comes to personal investing, your own rate of spending is one such antagonistic process — the more money you have, the more money you spend. When it comes to intelligence, inter-system communication arises as a brake on any improvement of underlying modules — a brain with smarter parts will have more trouble coordinating them; a society with smarter individuals will need to invest far more in networking and communication, etc. It is perhaps not a coincidence that very high-IQ people are more likely to suffer from certain mental illnesses. It is also perhaps not random happenstance that military empires of the past have ended up collapsing after surpassing a certain size. Exponential progress, meet exponential friction.
一个值得关注的具体例子是科学进步，因为它在概念上与智能本身非常接近——科学作为一个解决问题的系统，非常接近一个失控的超人人工智能。科学当然是一个递归自我改进的系统，因为科学进步导致了工具的发展，这些工具又促进了科学——无论是实验室硬件（例如，量子物理学导致了激光，而激光又促成了大量新的量子物理学实验）、概念工具（例如，新定理、新理论）、认知工具（例如，数学符号）、软件工具，还是使科学家能够更好协作的通信协议（例如，互联网）……
One specific example that is worth paying attention to is that of scientific progress, because it is conceptually very close to intelligence itself — science, as a problem-solving system, is very close to being a runaway superhuman AI. Science is, of course, a recursively self-improving system, because scientific progress results in the development of tools that empower science — whether lab hardware (e.g. quantum physics led to lasers, which enabled a wealth of new quantum physics experiments), conceptual tools (e.g. a new theorem, a new theory), cognitive tools (e.g. mathematical notation), software tools, communications protocols that enable scientists to better collaborate (e.g. the Internet)…
然而，现代科学进步是可测量的线性增长。我在 2012 年一篇题为《奇点不会到来》的文章中详细讨论了这一现象。我们在 1950-2000 年期间在物理学上取得的进步并不比 1900-1950 年期间更大——可以说，我们做得差不多。数学的进步速度今天并不比 1920 年快多少。几十年来，医学科学在所有指标上都取得了线性进步。尽管我们投入了指数级的努力——研究人员数量大约每 15 到 20 年翻一番，而且这些研究人员使用指数级更快的计算机来提高生产力。
Yet, modern scientific progress is measurably linear. I wrote about this phenomenon at length in a 2012 essay titled “_The Singularity is not coming_”. We didn’t make greater progress in physics over the 1950–2000 period than we did over 1900–1950 — we did, arguably, about as well. Mathematics is not advancing significantly faster today than it did in 1920. Medical science has been making linear progress on essentially all of its metrics, for decades. And this is despite us investing exponential efforts into science — the headcount of researchers doubles roughly once every 15 to 20 years, and these researchers are using exponentially faster computers to improve their productivity.
这是怎么回事？是什么瓶颈和对抗性反应减缓了科学中的递归自我改进？太多了，我数都数不过来。这里列举几个。重要的是，它们每一个也都适用于递归自我改进的人工智能。
How comes? What bottlenecks and adversarial counter-reactions are slowing down recursive self-improvement in science? So many, I can’t even count them. Here are a few. Importantly, every single one of them would also apply to recursively self-improving AIs.
* 在特定领域从事科学研究会随着时间的推移呈指数级困难——该领域的创始人摘取了大部分低垂的果实，后来者要达到同等影响力需要付出指数级更多的努力。没有哪个研究者能在信息论上取得香农 1948 年论文那样的进步。
* Doing science in a given field gets exponentially harder over time — the founders of the field reap most the low-hanging fruit, and achieving comparable impact later requires exponentially more effort. No researcher will ever achieve comparable progress in information theory as Shannon did in his 1948 paper.
* 随着领域规模的扩大，研究者之间的分享与合作变得指数级困难。跟上新出版物洪流变得越来越难。记住，一个有 N 个节点的网络有 N*(N-1)/2 条边。
* Sharing and cooperation between researchers gets exponentially more difficult as a field grows larger. It gets increasingly harder to keep up with the firehose of new publications. Remember that a network with N nodes has N * (N - 1) / 2edges.
* 随着科学知识的扩展，必须投入教育和培训的时间和精力也在增长，个体研究者的研究领域变得越来越狭窄。
* As scientific knowledge expands, the time and effort that have to be invested in education and training grows, and the field of inquiry of individual researchers gets increasingly narrow.
在实践中，系统瓶颈、收益递减和对抗性反应最终压制了我们周围所有递归过程中的递归自我改进。自我改进确实带来了进步，但这种进步往往是线性的，或者最多是 S 形的。你投入的第一“种子美元”通常不会导致“财富爆炸”；相反，投资回报与不断增长的支出之间的平衡通常会导致你的储蓄随时间大致线性增长。而这对于一个比自我改进的思维简单数个数量级的系统来说也是如此。
In practice, system bottlenecks, diminishing returns, and adversarial reactions end up squashing recursive self-improvement in all of the recursive processes that surround us. Self-improvement does indeed lead to progress, but that progress tends to be linear, or at best, sigmoidal. Your first “seed dollar” invested will not typically lead to a “wealth explosion”; instead, a balance between investment returns and growing spending will usually lead to a roughly linear growth of your savings over time. And that’s for a system that is orders of magnitude simpler than a self-improving mind.
同样，第一个超人人工智能只是我们很久以前就开始攀登的、明显线性的进步阶梯上的又一步。
Likewise, the first superhuman AI will just be another step on a visibly linear ladder of progress, that we started climbing long ago.
智能的扩展只能来自大脑（生物或数字）、感觉运动可供性、环境和文化的共同进化——而不是仅仅孤立地调整某个缸中大脑的齿轮。这种共同进化已经持续了亿万年，并将随着智能向日益数字化的基质迁移而继续。不会发生“智能爆炸”，因为这一过程大致以线性速度推进。
The expansion of intelligence can only come from a co-evolution of brains (biological or digital), sensorimotor affordances, environment, and culture — not from merely tuning the gears of some brain in a jar, in isolation. Such a co-evolution has already been happening for eons, and will continue as intelligence moves to an increasingly digital substrate. No “intelligence explosion” will occur, as this process advances at a roughly linear pace.
* 智能是情境性的——不存在所谓的通用智能。你的大脑是一个更广泛系统的一部分，这个系统包括你的身体、环境、其他人类以及整个文化。
* Intelligence is situational — there is no such thing as general intelligence. Your brain is one piece in a broader system which includes your body, your environment, other humans, and culture as a whole.
* 没有系统存在于真空中；任何个体智能都始终由其存在的情境、其环境所定义和限制。目前，是我们的环境，而非我们的大脑，成为了我们智能的瓶颈。
* No system exists in a vacuum; any individual intelligence will always be both defined and limited by the context of its existence, by its environment. Currently, our environment, not our brain, is acting as the bottleneck to our intelligence.
* 人类智能在很大程度上是外化的，不是存在于我们的大脑中，而是存在于我们的文明中。我们就是我们的工具——我们的大脑是一个比我们自身大得多的认知系统中的模块。这个系统已经在自我改进，并且已经持续了很长时间。
* Human intelligence is largely externalized, contained not in our brain but in our civilization. We are our tools — our brains are modules in a cognitive system much larger than ourselves. A system that is already self-improving, and has been for a long time.
* 递归自我改进系统，由于偶然的瓶颈、收益递减以及其存在的更广泛背景中产生的反作用，在实践中无法实现指数级进步。经验上，它们往往表现出线性或 S 形改进。特别是，科学进步就是如此——科学可能是我们能观察到的最接近递归自我改进 AI 的系统。
* Recursively self-improving systems, because of contingent bottlenecks, diminishing returns, and counter-reactions arising from the broader context in which they exist, cannot achieve exponential progress in practice. Empirically, they tend to display linear or sigmoidal improvement. In particular, this is the case for scientific progress — science being possibly the closest system to a recursively self-improving AI that we can observe.
* 递归智能扩展已经在发生——在我们文明的层面上。它将在 AI 时代继续发生，并且大致以线性速度推进。
* Recursive intelligence expansion is already happening — at the level of our civilization. It will keep happening in the age of AI, and it progresses at a roughly linear pace.
营销脚注：我的书《Python 深度学习》刚刚发布。如果你有 Python 技能，并且想了解深度学习能做什么和不能做什么，以及如何使用它来解决困难的实际问题，这本书就是为你写的。
Marketing footnote: my book _Deep Learning with Python_ has just been released. If you have Python skills, and you want to understand what deep learning can and cannot do, and how to use it to solve difficult real-world problems, this book was written for you.