Planning for AGI and beyond
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→我们的使命是确保通用人工智能——比人类更聪明的 AI 系统——惠及全人类。_更新于 2025 年 10 月 28 日：本文包含关于我们结构的过时信息。请参考以下页面获取最新信息。_ 我们的使命是确保通用人工智能——比人类更聪明的 AI 系统——惠及全人类。
Our mission is to ensure that artificial general intelligence—AI systems that are generally smarter than humans—benefits all of humanity. _Updated October 28, 2025: This post contains outdated information about our structure. Please refer to thefollowing pagefor updated information._ Our mission is to ensure that artificial general intelligence—AI systems that are generally smarter than humans—benefits all of humanity.
我们的使命是确保通用人工智能——比人类更聪明的 AI 系统——能够造福全人类。
Our mission is to ensure that artificial general intelligence—AI systems that are generally smarter than humans—benefits all of humanity.
更新于 2025 年 10 月 28 日：本文包含关于我们结构的过时信息。请参考以下页面获取最新信息。
_Updated October 28, 2025: This post contains outdated information about our structure. Please refer to thefollowing pagefor updated information._
我们的使命是确保通用人工智能——比人类更聪明的 AI 系统——能够造福全人类。
Our mission is to ensure that artificial general intelligence—AI systems that are generally smarter than humans—benefits all of humanity.
如果 AGI 被成功创造，这项技术可以通过增加富足、推动全球经济增长以及帮助发现改变可能性极限的新科学知识，来帮助提升人类。
If AGI is successfully created, this technology could help us elevate humanity by increasing abundance, turbocharging the global economy, and aiding in the discovery of new scientific knowledge that changes the limits of possibility.
AGI 有潜力赋予每个人难以置信的新能力；我们可以想象一个世界，其中所有人都能获得几乎任何认知任务的帮助，为人类的聪明才智和创造力提供巨大的倍增器。
AGI has the potential to give everyone incredible new capabilities; we can imagine a world where all of us have access to help with almost any cognitive task, providing a great force multiplier for human ingenuity and creativity.
另一方面，AGI 也会带来滥用、严重事故和社会动荡的巨大风险。由于 AGI 的好处如此巨大，我们认为社会不可能也不希望永远阻止其发展；相反，社会和 AGI 的开发者必须找出如何正确应对的方法。
On the other hand, AGI would also come with serious risk of misuse, drastic accidents, and societal disruption. Because the upside of AGI is so great, we do not believe it is possible or desirable for society to stop its development forever; instead, society and the developers of AGI have to figure out how to get it right.A
虽然我们无法准确预测会发生什么，当然我们目前的进展也可能遇到瓶颈，但我们可以阐明我们最关心的原则：
Although we cannot predict exactly what will happen, and of course our current progress could hit a wall, we can articulate the principles we care about most:
1. 我们希望 AGI 能够赋予人类在宇宙中最大程度地繁荣发展的能力。我们不期望未来是一个毫无保留的乌托邦，但我们希望最大化好处、最小化坏处，并让 AGI 成为人类的放大器。
1. We want AGI to empower humanity to maximally flourish in the universe. We don’t expect the future to be an unqualified utopia, but we want to maximize the good and minimize the bad, and for AGI to be an amplifier of humanity.
2. 我们希望 AGI 的好处、使用权和治理权能够得到广泛而公平的分享。
2. We want the benefits of, access to, and governance of AGI to be widely and fairly shared.
3. 我们希望成功应对巨大风险。面对这些风险，我们承认理论上看似正确的东西在实践中往往比预期更奇怪。我们认为必须通过部署较不强大的技术版本来不断学习和适应，以尽量减少“一次性成功”的情况。
3. We want to successfully navigate massive risks. In confronting these risks, we acknowledge that what seems right in theory often plays out more strangely than expected in practice. We believe we have to continuously learn and adapt by deploying less powerful versions of the technology in order to minimize “one shot to get it right”scenarios.
我们认为，当前有几件重要的事情需要做，以便为 AGI 做好准备。
There are several things we think are important to do now to prepare for AGI.
首先，随着我们不断创建更强大的系统，我们希望部署它们并在现实世界中获得操作经验。我们认为这是谨慎地将 AGI 引入存在的最佳方式——逐步过渡到拥有 AGI 的世界比突然过渡更好。我们预计强大的 AI 将使世界进步的速度大大加快，而我们认为逐步适应这种变化更好。
First, as we create successively more powerful systems, we want to deploy them and gain experience with operating them in the real world. We believe this is the best way to carefully steward AGI into existence—a gradual transition to a world with AGI is better than a sudden one. We expect powerful AI to make the rate of progress in the world much faster, and we think it’s better to adjust to this incrementally.
逐步过渡可以让人们、政策制定者和机构有时间了解正在发生的事情，亲身体验这些系统的利弊，调整我们的经济，并制定监管措施。它还能让社会与 AI 共同进化，让人们在风险相对较低时集体弄清楚他们想要什么。
A gradual transition gives people, policymakers, and institutions time to understand what’s happening, personally experience the benefits and downsides of these systems, adapt our economy, and to put regulation in place. It also allows for society and AI to co-evolve, and for people collectively to figure out what they want while the stakes are relatively low.
我们目前认为，成功应对 AI 部署挑战的最佳方式是通过快速学习和谨慎迭代的紧密反馈循环。社会将面临关于 AI 系统被允许做什么、如何应对偏见、如何处理就业替代等重大问题。最优决策将取决于技术发展的路径，而像任何新领域一样，大多数专家预测到目前为止都是错误的。这使得在真空中进行规划非常困难。
We currently believe the best way to successfully navigate AI deployment challenges is with a tight feedback loop of rapid learning and careful iteration. Society will face major questions about what AI systems are allowed to do, how to combat bias, how to deal with job displacement, and more. The optimal decisions will depend on the path the technology takes, and like any new field, most expert predictions have been wrong so far. This makes planning in a vacuum very difficult.B
总的来说，我们认为世界上 AI 的更多使用将带来好处，并希望促进这一点（通过将模型放入我们的 API、开源等方式）。我们相信，民主化的访问还将带来更多更好的研究、权力下放、更多好处，以及更广泛的人群贡献新想法。
Generally speaking, we think more usage of AI in the world will lead to good, and want to promote it (by putting models in our API, open-sourcing them, etc.). We believe that democratized access will also lead to more and better research, decentralized power, more benefits, and a broader set of people contributing new ideas.
随着我们的系统越来越接近 AGI，我们在创建和部署模型时变得越来越谨慎。我们的决策需要比社会通常应用于新技术的谨慎程度更高，也比许多用户希望的要高。AI 领域的一些人认为 AGI（及其后续系统）的风险是虚构的；如果事实证明他们是对的，我们会很高兴，但我们将按照这些风险是存在的来行事。
As our systems get closer to AGI, we are becoming increasingly cautious with the creation and deployment of our models. Our decisions will require much more caution than society usually applies to new technologies, and more caution than many users would like. Some people in the AI field think the risks of AGI (and successor systems) are fictitious; we would be delighted if they turn out to be right, but we are going to operate as if these risks areexistential(opens in a new window).
在某个时刻，部署的利弊平衡（例如赋予恶意行为者权力、造成社会和经济混乱、加速不安全的竞赛）可能会发生变化，届时我们将显著改变关于持续部署的计划。
At some point, the balance between the upsides and downsides of deployments (such as empowering malicious actors, creating social and economic disruptions, and accelerating an unsafe race) could shift, in which case we would significantly change our plans around continuous deployment.
其次，我们正致力于创建越来越对齐和可引导的模型。我们从像 GPT-3 的第一个版本到 InstructGPT 和 ChatGPT 的转变是这方面的一个早期例子。
Second, we are working towards creating increasingly aligned and steerable models. Our shift from models like the first version of GPT‑3 toInstructGPTandChatGPT(opens in a new window)is an early example of this.
特别是，我们认为社会就 AI 的使用达成极其广泛的界限很重要，但在这些界限内，个体用户应有很大的自由裁量权。我们最终的希望是，世界上的机构能就这些广泛界限达成一致；短期内，我们计划进行实验以获取外部意见。世界上的机构需要增强能力和经验，以便为关于 AGI 的复杂决策做好准备。
In particular, we think it’s important that society agree on extremely wide bounds of how AI can be used, but that within those bounds, individual users have a lot of discretion. Our eventual hope is that the institutions of the world agree on what these wide bounds should be; in the shorter term we plan to run experiments for external input. The institutions of the world will need to be strengthened with additional capabilities and experience to be prepared for complex decisions about AGI.
我们产品的“默认设置”可能会相当受限，但我们计划让用户轻松改变他们使用的 AI 的行为。我们相信赋予个人做出自己决定的权利，以及思想多样性的内在力量。
The “default setting” of our products will likely be quite constrained, but we plan to make it easy for users to change the behavior of the AI they’re using. We believe in empowering individuals to make their own decisions and the inherent power of diversity of ideas.
随着我们的模型变得更加强大，我们需要开发新的对齐技术（以及测试来了解我们当前技术何时失效）。我们的短期计划是利用 AI 帮助人类评估更复杂模型的输出并监控复杂系统，长期计划则是利用 AI 帮助我们提出更好的对齐技术的新想法。
We will need to developnew alignment techniquesas our models become more powerful (and tests to understand when our current techniques are failing). Our plan in the shorter term is touse AI to help humans evaluatethe outputs of more complex models and monitor complex systems, and in the longer term to use AI to help us come up with new ideas for better alignment techniques.
重要的是，我们认为我们通常必须同时推进 AI 安全性和能力。将它们分开讨论是一种错误的二分法；它们在许多方面是相关的。我们最好的安全工作来自于与我们最强大的模型合作。尽管如此，安全进展与能力进展的比例需要增加。
Importantly, we think we often have to make progress on AI safety and capabilities together. It’s a false dichotomy to talk about them separately; they are correlated in many ways. Our best safety work has come from working with our most capable models. That said, it’s important that the ratio of safety progress to capability progress increases.
第三，我们希望就三个关键问题进行全球对话：如何治理这些系统，如何公平分配它们产生的利益，以及如何公平分享访问权限。
Third, we hope for a global conversation about three key questions: how to govern these systems, how to fairly distribute the benefits they generate, and how to fairly share access.
除了这三个领域，我们还试图以与良好结果激励一致的方式构建我们的结构。我们的章程中有一个条款，关于在 AGI 后期开发中协助其他组织推进安全，而不是与他们竞争。我们对股东可以获得的回报设置了上限，这样我们就不会受到激励去无限制地攫取价值并冒险部署可能具有灾难性危险的东西（当然，这也是与社会分享利益的一种方式）。我们有一个非营利组织来管理我们，让我们为人类利益而运营（并且可以凌驾于任何营利利益之上），包括允许我们出于安全原因取消对股东的股权义务，以及赞助世界上最全面的 UBI 实验。
In addition to these three areas, we have attempted to set up our structure in a way that aligns our incentives with a good outcome. We havea clause in our Charterabout assisting other organizations to advance safety instead of racing with them in late-stage AGI development. We have a cap on the returns our shareholders can earn so that we aren’t incentivized to attempt to capture value without bound and risk deploying something potentially catastrophically dangerous (and of course as a way to share the benefits with society). We have a nonprofit that governs us and lets us operate for the good of humanity (and can override any for-profit interests), including letting us do things like cancel our equity obligations to shareholders if needed for safety and sponsor the world’s most comprehensive UBI experiment.
我们认为，像我们这样的努力在发布新系统之前应该接受独立审计；我们将在今年晚些时候更详细地讨论这一点。在某个时候，在开始训练未来系统之前获得独立审查可能很重要，并且最先进的努力应同意限制用于创建新模型的算力增长率。我们认为，关于 AGI 努力何时应停止训练运行、决定模型安全可发布或从生产环境中撤回模型的公共标准很重要。最后，我们认为主要国家政府应对超过一定规模的训练运行有所了解。
We think it’s important that efforts like ours submit to independent audits before releasing new systems; we will talk about this in more detail later this year. At some point, it may be important to get independent review before starting to train future systems, and for the most advanced efforts to agree to limit the rate of growth of compute used for creating new models. We think public standards about when an AGI effort should stop a training run, decide a model is safe to release, or pull a model from production use are important. Finally, we think it’s important that major world governments have insight about training runs above a certain scale.
我们相信人类的未来应由人类自己决定，并且向公众分享进展信息至关重要。所有试图构建 AGI 的努力都应受到严格审查，重大决策需进行公众咨询。
We believe that the future of humanity should be determined by humanity, and that it’s important to share information about progress with the public. There should be great scrutiny of all efforts attempting to build AGI and public consultation for major decisions.
第一个 AGI 将只是智能连续体上的一个点。我们认为进展很可能从那里继续，可能在未来很长一段时间内保持过去十年的进步速度。如果真是这样，世界可能变得与今天截然不同，风险也可能异常巨大。一个未对齐的超级智能 AGI 可能对世界造成严重伤害；一个拥有决定性超级智能领先优势的专制政权也可能如此。
The first AGI will be just a point along the continuum of intelligence. We think it’s likely that progress will continue from there, possibly sustaining the rate of progress we’ve seen over the past decade for a long period of time. If this is true, the world could become extremely different from how it is today, and the risks could be extraordinary. A misaligned superintelligent AGI could cause grievous harm to the world; an autocratic regime with a decisive superintelligence lead could do that too.
能够加速科学进步的 AI 是一个值得思考的特殊案例，其影响可能比其他一切更为深远。能够加速自身进步的 AGI 可能以惊人的速度引发重大变化（即使转变开始缓慢，我们预计在最后阶段也会相当迅速）。我们认为较慢的起飞更容易确保安全，并且在关键节点协调 AGI 努力以减缓速度可能很重要（即使在不需要解决技术对齐问题的世界中，减缓速度也可能至关重要，以便给社会足够的时间来适应）。
AI that can accelerate science is a special case worth thinking about, and perhaps more impactful than everything else. It’s possible that AGI capable enough to accelerate its own progress could cause major changes to happen surprisingly quickly (and even if the transition starts slowly, we expect it to happen pretty quickly in the final stages). We think a slower takeoff is easier to make safe, and coordination among AGI efforts to slow down at critical junctures will likely be important (even in a world where we don’t need to do this to solve technical alignment problems, slowing down may be important to give society enough time to adapt).
成功过渡到拥有超级智能的世界或许是人类历史上最重要——也最充满希望和恐惧——的项目。成功远非必然，而其中的利害关系（无限的下行风险和无限的上行潜力）有望将我们所有人团结起来。
Successfully transitioning to a world with superintelligence is perhaps the most important—and hopeful, and scary—project in human history. Success is far from guaranteed, and the stakes (boundless downside and boundless upside) will hopefully unite all of us.
我们可以想象一个人类繁荣到我们中任何人可能都无法完全想象的程度的世界。我们希望为世界贡献一个与这种繁荣对齐的 AGI。
We can imagine a world in which humanity flourishes to a degree that is probably impossible for any of us to fully visualize yet. We hope to contribute to the world an AGI aligned with such flourishing.