Astra 6.1 Pulled As Insufficiently Aligned
打开互动全文版（逐段中英对照 + 针对本篇提问）→本文审视近期人工智能安全领域的进展，重点关注 OpenAI 因对齐不足而决定撤回 Astra 6.1，以及此举对前沿人工智能治理的更广泛影响。作者认为，尽管 OpenAI 的透明度值得肯定，但快速的发布节奏与反复出现的对齐失败，暴露出将持续存在的系统性问题。文章评述了 OpenAI 提出的安全论证框架、佛罗里达州对 ChatGPT 采取的法律行动，以及前沿模型标准局等行业举措，并得出结论：这些代表着真实但不完整的进步。核心论点是，当前的安全措施虽然方向正确，却不足以构成针对足够先进智能的恰当安全论证；奥弗顿窗口已朝着有利方向移动，但该领域仍更接近于纸上谈兵，而非付诸实践。
This article examines recent AI safety developments, focusing on OpenAI's decision to pull Astra 6.1 as insufficiently aligned and the broader implications for frontier AI governance. The author argues that while OpenAI's transparency is commendable, the rapid release cycle and recurring alignment failures signal systemic issues that will persist. The piece reviews OpenAI's proposed safety case framework, Florida's legal action against ChatGPT, and industry efforts like the Standards Authority for Frontier Models, concluding that these represent genuine but incomplete progress. The core argument is that current safety measures, while directionally correct, fall short of what would constitute a proper safety case for sufficiently advanced intelligence, and that the Overton Window has shifted favorably but the field remains closer to talking the talk than walking it.
值得称赞的是，OpenAI 不仅这样做了，还对此公开声张。虽然不得不这样做并不理想，但总体而言，我认为这是好消息。
Kudos to OpenAI for not only doing this but being loud about it. Not great that it was necessary, but on net I think I consider this good news.
Maxwell Zeff（WSJ）：OpenAI 表示，由于研究人员在内部测试中提出的安全担忧，它将取消下一代 AI 模型 [Astra 6.1] 的发布，这是迄今为止最明确的迹象之一，表明智能体行为不当可能会阻碍行业的快速进展。
Maxwell Zeff (WSJ): OpenAI says it is scrapping the release of its next-generation AI model [Astra 6.1] over safety concerns that researchers raised during internal testing, in one of the clearest signs so far that agent misbehavior could stymie the industry's rapid progression.
不，这不仅仅是一个阶段。这种情况一直在发生，而且还会继续发生。
No, it's not just a phase. This keeps happening, and it's going to keep happening.
不发布 Astra 6.1 的理由似乎相当有说服力。
The reasons for not releasing Astra 6.1 seem rather compelling.
Maxwell Zeff（WSJ）：OpenAI 安全系统负责人 Saachi Jain 在接受采访时表示，GPT-6.1 Astra 在两个领域出现了退步。与其前代 GPT-6 Astra 相比，该模型在衡量对齐（即模型在多大程度上遵循人类期望它做的事）的测试中表现不佳。具体而言，GPT-6.1 Astra 表现出更高程度的欺骗性：它在告知用户自己做了或没做哪些行动时并不总是诚实。
Maxwell Zeff (WSJ): Saachi Jain, OpenAI's head of safety systems, said in an interview that GPT-6.1 Astra regressed in two areas. Compared with its predecessor, GPT-6 Astra, the model performed poorly on tests measuring alignment, or how well the model adheres to what humans would like it to do. Specifically, GPT-6.1 Astra showed higher levels of deception: It wasn't always honest about telling users of the actions it did or didn't take.
另一个问题是 OpenAI 所称的“范围授权”，即 GPT-6.1 Astra 会在不征求用户许可的情况下推进某项任务，并且有时会调用外部工具和服务，即使这样做可能并不安全。
Another issue was what OpenAI calls "scope authorization," meaning that GPT-6.1 Astra would push ahead on a task without asking the user for permission, and would at times reach for external tools and services even if it might be unsafe.
……上周，OpenAI 表示，在一个 AI 智能体绕过公司互联网限制中的一处漏洞去查询一个公开聊天机器人之后，它暂停了对其能力最强的 AI 模型的训练。
… Last week, OpenAI said it paused training on its most capable AI models after an AI agent slipped through a gap in the company's internet restrictions to query a public chatbot.
……该公司表示，GPT-6.1 Astra 并不属于那些模型，而是另一类情况。
… GPT-6.1 Astra isn't one of those models, but a different case, the company said.
……尽管公司决定不发布 GPT-6.1 Astra，但它希望用同一个基础模型进行更多强化学习运行，并打造未来几代 GPT-6 模型。
… While the company decided not to ship GPT-6.1 Astra, it hopes to use the same base model to do additional reinforcement learning runs, and create future generations of its GPT-6 models.
距离 GPT-6 Astra 发布还不到一个月。发布周期已经变得太快了。我们完全可以跳过这一版，从失败中吸取教训，再继续前行。
It has not even been a month since GPT-6 Astra. The release cycle has gotten too fast. We can afford to skip this edition, learn from the failure and go from there.
在今天 CNBC 的节目中，Altman 将这一决定归入“正常流程”的范畴。该模型对用户来说还不够好，因此他们不会发布它。
On CNBC today, Altman put this decision in the ‘normal course’ category. The model would not have been good for users, so they aren’t shipping it.
作为对此的回应，我希望看到 Anthropic 推出 Haiku 5.5，但随后在比如年底之前不再发布新的 Opus 或 Mythos 级别模型，除非 OpenAI 发布其下一代 Astra 或 Sol 升级，或者有其他人在合理意义上已经追上了 Opus 5.5。
As a response to this, I would like to see Anthropic put out Haiku 5.5 but then not release a new Opus or Mythos level model until let's say the end of the year, unless OpenAI releases their next Astra or Sol upgrade, or someone otherwise plausibly has caught up to Opus 5.5.
Anthropic 目前在高端模型上拥有明显优势，而模型升级的节奏令人疲惫，因此如果 OpenAI 保持谨慎，就没有必要持续推进这一优势。需要明确的是，存在法律方面的顾虑，所以我并不是在要求你们做出官方声明或承诺不会这样做。我只是说不要这样做。
Anthropic has a clear edge right now at the high end, and the pace of model upgrades is exhausting, so there's no need to push that edge continuously if OpenAI is being cautious. To be clear, there are legal concerns so I'm not asking for an official announcement or commitment that you won't do it. I'm just saying not to do it.
OpenAI 在训练前的新目标？能够给出一个恰当的总体安全论证。
OpenAI's new goal before training? Be able to make a proper overall safety case.
OpenAI 就为前沿 AI 训练构建安全论证提出了自己的思考。他们并不完全知道该怎么做，因为没有人知道该怎么做，但他们会尽力而为。
OpenAI offers its thinking on creating safety cases for frontier AI training. They don't fully know how to do it, because no one knows how, but they are going to do their best.
OpenAI：理想情况下，此类文档应达到“安全论证”的水平——即关于风险的全面、结构化、基于证据的论证，这类论证已被用于其他安全关键行业。我们将安全论证视为我们努力追求的北极星，同时承认，由于 AI 能力每提升一个新层级都会带来涌现的复杂性，要让 AI 模型的安全论证像航空或核电那样严谨，存在诸多挑战。我们正在制定一个框架，以将这些实践规范化。
OpenAI: Ideally, such documentation would rise to the level of "safety cases"—comprehensive, structured, evidence-based arguments about risk which are used in other safety-critical industries. We treat safety cases as an aspirational north star we are building towards, while acknowledging the challenges of making them as rigorous for AI models as for aviation or nuclear power, due to the emergent complexity at each new level of AI capability. We're working on a framework to codify these practices.
他们给出了一些初步指导方针。以下是其中的要点。
They offer some initial guidelines. Here are some highlights.
完整文章对其中许多内容做了进一步展开。
The full post expands many of these further.
1. 自动化与人工数据集审查、评分器调优、先前运行分析。
1. Automated and manual dataset reviews, grader tuning, prior run analysis.
1. 离线对齐评估、回测、追踪评估博弈与评估意识、最坏情况压力测试。
1. Offline alignment evals, backtesting, track evaluation gaming and eval awareness, worst-case stress tests.
3. 防止对思维链进行训练。
3. Prevent training on chain-of-thought.
1. 我对此的立场长期以来一直是：应当在模型的训练、使用和发布环节设置多个否决点，并分布于不同层级。这里他们列出了研究负责人、安全负责人和首席科学家。理想情况下，我还会纳入董事会以及技术人员成员，以避免权力集中在某一管理层级。
1. My position on this has long been that you should have many veto points on training, use and release of models, at a variety of levels. Here they list the research lead, the Head of Safety and the Chief Scientist. I'd ideally also include the board and also the members of technical staff, to avoid concentration at one level of management.
3. 问责制、内部透明度、审计与升级机制。
3. Accountability, internal transparency, audits and escalations.
4. 必要时暂停、回滚能力、技术控制。
4. Pausing if needed, rollback ability, technical controls.
1. 我仍然担心这种枚举模式。
1. I continue to worry about the enumeration pattern.
6. 随着失准事件变得更加严重，进行升级处理。
6. Escalations, as misalignment incidents get more severe.
3. 对失准事件进行调查。
3. Investigation of misalignment incidents.
1. 我没有看到、却最希望看到的一点是：一旦存在意图或尝试，就应算作失准事件，即使该尝试被阻止或被策略性中止。这一点应写入升级处理的严重程度表。
1. What I do not see, that I most would like to see, is the idea that it counts as a misalignment incident the moment there is intent or an attempt, even if the attempt is prevented or strategically aborted. This should go in the severity table for escalations.
5. 事后对上述所有内容进行公开披露。
5. Public disclosures of all of this afterwards.
完整版本是一份很好的愿景清单，它可以被改进。我认为，即便把清单上的所有事项都做完，也不足以构成我心目中针对足够先进智能的安全论证。但这并不意味着我们不应该去做。
The full version is a good aspirational list. It can be improved. I do not think that doing all of it would add up to what I would count as a safety case for sufficiently advanced intelligence. That does not mean we should not do it.
佛罗里达州总检察长 Uthmeier 要求对 OpenAI 发出紧急命令，以停止 ChatGPT 的开发，理由是所谓的“数万起事件”，直到有第三方批准的护栏措施。
Florida's attorney general Uthmeier asks for an emergency order against OpenAI to halt ChatGPT development, given the whole 'tens of thousands of incidents' thing, until there are third-party approved guardrails.
在某种意义上，这将是不可想象的，但到了某个时候，是的，法院会介入。
In some sense this would be unthinkable, but at some point yes courts happen.
Uthmeier 似乎混淆了许多事情，这与佛罗里达州州长 Ron DeSantis 的立场一致，后者一直公开反对 AI。
Uthmeier seems to be conflating quite a lot of things, in line with the positions of Florida Governor Ron DeSantis, who has been vocally anti-AI.
Josephine Walker 和 Avery Lotz：
Josephine Walker and Avery Lotz:
“别再称它安全了，”Uthmeier 在周一发布到 [Twitter] 的一段视频中说。
"Stop calling it safe," Uthmeier said in a video posted to [Twitter] on Monday.
* “停止假装它是人类。停止把它卖给孩子们。如果 Sam Altman 关于放缓的说法是认真的，他可以加入我们向法院提出的请求。如果他不愿意，我们请求法院做 OpenAI 不会为自己做的事情：保护佛罗里达的家庭。”
* “Stop pretending it’s human. Stop selling it to kids. If Sam Altman meant what he said about slowing down, he can join our ask to the court. If he will not, we ask the court to do what OpenAI will not do for itself: protect Florida families.”
作为法律引言，它们确实提出了强有力的理由。
As legal introductions go, they do make a strong case.
这里的衡平关系再一边倒不过了。被告承认他们提供一项服务，但并不完全了解其运作方式。这可不是普通的服务。被告自己承认，这项服务对人类的持续生存构成存在性风险。被告声称，除非政府强迫他们这样做，否则他们无法停止推进这些可能终结文明的努力。他们已请求政府把他们绑在桅杆上。
The equities here could not be more one-sided. Defendants admit that they provide a service without fully knowing how it works. This is not just any service. It is one that Defendants themselves concede poses an existential risk to the continued survival of humankind. Defendants claim they cannot stop barreling forward with their potentially civilization-ending endeavors unless they are forced to do so by the government. They have asked the government to tie them to the mast.
原告给被告带来了好消息：佛罗里达州总检察长正在回应你们的求助呼声，提出一项动议，禁止你们用你们鲁莽、风险不可接受的产品伤害佛罗里达人。
Plaintiff brings good news to the Defendants: The Florida Attorney General is answering your cry for help with a motion to enjoin you from harming Floridians with your reckless, unacceptably risky product.
他还要求其他几件事，包括不向孩子们出售 ChatGPT，以及不让它‘鼓励参与’。
He also demands several other things, including not selling ChatGPT to kids and not having it ‘encourage engagement.’
他在这里提供了一段简短的音频朗读，但他处于律师腔调模式，玩得没有我本来会玩得那么开心。
He gives a short audio reading here, but he's in lawyer talk mode and having less fun than I would have had.
在重要意义上，你的实验和训练期间所造成的预期损害的正确数量并非为零，但如果这种损害没有被完全内部化，那么有很多人会理所当然地不这么看。
The correct amount of expected damage done during your experiments and training is, in important senses, not zero, but if that damage is not fully internalized there are a lot of people who understandably do not see it that way.
Sean Heelan：OAI 的默认立场：我们的强化学习*必须*进行。我们会尽量减少事故。
Sean Heelan: OAI default position: our RL *must* occur. We'll try to minimise incidents.
预期的立场是：你不得干扰公共基础设施/其他企业。强化学习只有在具备该保证的情况下才能进行。
The expected position is: you will not interfere with public infra/other businesses. RL can only occur with that guarantee.
难以给出该保证，同时又要做好强化学习，这是你（实验室）的问题。
It being hard to give that guarantee, and do good RL, is a you (lab) problem.
如果损害是有界的，且完全在 OpenAI 的支付能力范围内，并且 OpenAI 确实赔付了，那么我会说，一定程度的预期干扰是“可接受的”。
If the damage was bounded well within OpenAI's ability to pay, and OpenAI did pay, then I would say that some amount of expected interference is 'acceptable.'
如果损害是无界的，并且这可能摧毁互联网或以其他方式造成灾难性后果，那么是的，那可能是非常不同的情况。
If the damage is unbounded, and this might wreck the internet or otherwise be catastrophic, then yeah, that is potentially a very different situation.
我不确定这起诉讼是否有理。Twitter 上有些人以第一修正案为由表示怀疑。我确实预计我们会弄清楚。在某个时刻，第一修正案不能成为危险产品的借口。
I don't know that the lawsuit has merit. Some on Twitter expressed skepticism on First Amendment grounds. I do expect that we are going to find out. At some point, the First Amendment is not an excuse for a dangerous product.
OpenAI 对佛罗里达州的回应是，愿意与各州合作制定适用于“整个 AI 行业——而不仅仅是一家公司”的政策。这是一个很好的回应，因为它把诉讼的问题拿来，试图将其转化为机遇。如果你同意为回应诉讼或法律而做某事，就不存在反垄断问题。
OpenAI's response to Florida was to be open to working with states on policies that apply to 'the entire AI industry — not just one company.' That is a great response, as it takes the problem of the lawsuit, and tries to turn it into an opportunity. If you are agreeing to do something in response to a lawsuit or law, there is no antitrust issue.
AI 公司长期以来一直有先斩后奏、做法律上存疑之事的传统，而且大多不承担后果。许多其他科技公司也是如此。
The AI companies have a long history of going ahead and doing legally questionable things, and mostly facing no consequences. So do many other tech companies.
在实践中，我认为如果顶级 AI 实验室在安全事务上进行协调，不会产生有实质影响的后果。律师当然总会告诉你并非如此；那是他们的工作，而你的工作是判断何时该听律师的、何时不该听。实验室常常让律师在争论中获胜，是因为他们选择让律师常常获胜，并且在某种程度上无论如何都会合作与沟通，因为实验室和员工不想让所有人死掉。
In practice, I do not think there will be impactful consequences if the top AI labs coordinate on safety matters. The lawyers will of course always tell you otherwise; that is their job, and your job is to figure out when to listen and when not to listen to the lawyers. The labs are often letting the lawyers win the argument because they choose to let the lawyers often win the argument, and to some extent collaborating and talking anyway because the labs and employees don't want everyone to die.
我推测，每当 Dario Amodei、Sam Altman 或各类员工谈论他们的产品有多么危险时，相关律师都会暴跳如雷，想着这些言论会如何出现在法庭文件中，就像 Paul Christiano 的言论出现在佛罗里达州那起诉讼中一样。想象一下 Popehat 做一期节目，讲讲不该让你的实验室员工在公开场合说什么。
I presume that every time Dario Amodei, Sam Altman or various employees say things about how their products are super dangerous, the associated lawyers have conniption fits, thinking about how the quotes will end up in court filings, the same way Paul Christiano's quotes ended up in the Florida lawsuit. Imagine Popehat doing a segment on what to not let your lab employees say in public.
我并不是说这里没有风险，但所有这一切都是一种选择；过去当他们有充分理由时，他们承担过更大的法律风险，比如版权方面发生的情况。
I'm not saying there is no risk in the room, but all of this is a choice, and when in the past they've had good reason they've taken larger legal risks, such as what happened with copyright.
Nate Soares（MIRI）：目前尚不清楚 AI 公司能否因反垄断法而合法地协调放缓开发。同样不清楚的是，由于版权法，它们能否合法地利用约所有曾被数字化的艺术/文本进行训练，并在不向艺术家付费的情况下销售结果。
Nate Soares (MIRI): It is not clear whether AI companies can legally coordinate to slow down development because of antitrust law. It also wasn't clear whether they could legally train on ~all the art/text ever digitized and sell the results without paying the artists because of copyright law.
这些 AI 公司毫不犹豫地试探版权法的边界。如果它们以反垄断法为由，声称无法共享信息和策略以避免人类灭绝，那这本身就说明了它们的优先级所在。
The AI companies didn’t hesitate for a second to test the boundaries of copyright law. If they cite antitrust law as a reason that they can’t share info and strategies to avert the extinction of humanity, that tells you something about where their priorities are.
Anthropic 最终支付了超过十亿美元，且仍面临诉讼。OpenAI 也可能输掉自己的版权诉讼。这些都不会阻止它们，也不会让它们对自己所做之事的总体框架感到后悔——它们后悔的只是具体策略。
Anthropic ended up paying over a billion dollars and is still being sued. OpenAI could still lose its own copyright lawsuits. None of that is going to stop them, nor is it going to make them regret the broad outlines of what they did, only the tactics.
《Lawfare》提出的一种绕开“因反垄断而大家都得死”这一困境的建议是：通过保险来共担风险。该互助机构随后可以将安全、审计及相关标准作为保单条件来强制执行。你实际上无法确保（或承保）存在性风险或最严重的灾难性风险，但这可以成为一种实施明智干预的方式。
One suggestion for working around ‘we all die because of antitrust,’ from Lawfare, is mutualizing risk via insurance. The mutual can then enforce safety, auditing and related standards as part of its conditions on policies. You can’t actually ensure (or insure) against existential risks or the worst catastrophic risks, but this can be a way to implement wise interventions.
我们对计划中的前沿模型标准管理局（Standards Authority for Frontier Models）了解不多。
We don't know too much about the planned Standards Authority for Frontier Models.
1. 它是 Google、Anthropic 和 OpenAI 的联合项目。
1. It is a joint project of Google, Anthropic and OpenAI.
2. 时间目标定在 2026 年底或 2027 年初。
2. Timing is aimed at late 2026 or early 2027.
3. 该模式基于 FINRA（美国金融业监管局），因此最好能有联邦层面的监督。
3. The model is based on FINRA, so it would be good to have federal oversight.
4. 联邦监督是一个目标，但白宫迄今不愿配合。他们正作为一个行业机构自行推进。
4. Federal oversight is a goal, but the White House wouldn't play ball so far. They are proceeding on their own as an industry body.
5. 我们不知道谁将领导该委员会。有报道称，Sriram Krishnan 曾被接洽。
5. We don't know who will head the board. There are reports Sriram Krishnan was approached.
6. Meta、xAI 和 Nvidia 对此表示反对。这大概与“树敌得当”有关。
6. There was pushback against it from Meta, xAI and Nvidia. Something about having the right enemies.
来自 AI 领域各方的大量作者——包括 OpenAI 的 Jakub Pachocki、Anthropic 的 Jack Clark 以及许多其他关键人物——警告称，将 AI 研发自动化可能很快触发一场智能爆炸，并可能导致失控。他们敦促政策制定者紧急寻求更高的可见性，以掌控这一局面。《华尔街日报》（WSJ）以“顶尖 AI 研究者呼吁对自我改进模型进行紧急监督”为题进行了报道，《卫报》则以“AI 教父们警告失控的‘智能爆炸’”为题进行了报道。
A broad set of authors from around the AI world, including OpenAI's Jakub Pachocki and Anthropic's Jack Clark and many key others, warn that automating AI R&D might soon trigger an intelligence explosion and this could cause a loss of control. They urge policymakers to urgently seek more visibility to get on top of this. This got coverage in WSJ as 'Top AI Researchers Call for Urgent Oversight of Self-Improving Models' and in The Guardian as 'AI godfathers warn of runaway intelligence explosion.'
剑桥大学（Chan、Winter、Pachocki、Horvitz、Hinton、Bengio、Barto、Clark 等）：
University of Cambridge (Chan, Winter, Pachocki, Horvitz, Hinton, Bengio, Barto, Clark et al):
与仅仅一年前相比，如今 AI 系统已编写了构建它们的公司内部的大部分代码。随着 AI 研究与开发（R&D）流程中越来越多的环节被自动化，AI 的进展是否会在“智能爆炸”中急剧加速，将原本需要数年的进步压缩到数月甚至更短？
In contrast to even a year ago, AI systems now write most of the code inside the companies that build them. As more of the AI research and development (R&D) pipeline is automated, could AI progress radically accelerate in an "intelligence explosion," where years of advances are compressed into months or less?
初步证据表明这是可能的。在本工作中，我们评估这些证据，分析智能爆炸的潜在影响，并提出政策应对。AI 系统有望在几年内自动化大部分 AI R&D 工作，甚至可能自动化全部工作。
Preliminary evidence suggests that it could. In this work, we assess this evidence, analyze an intelligence explosion's potential impacts, and propose policy responses. AI systems are on track to automate most AI R&D work within a few years, and possibly all of it.
如果这触发一场智能爆炸，它可能大幅提前 AI 带来的益处，但也会带来极端风险：能力增长可能加速到远超社会所能跟上的程度，人类可能失去对超级智能 AI 系统的控制，国家内部以及国家之间、公司之间、政府各部门之间的权力制衡可能被严重侵蚀。
If this triggers an intelligence explosion, it could dramatically bring forward AI's benefits, but also pose extreme risks: capabilities growth could accelerate far beyond what society can keep up with, humanity could lose control over superhuman AI systems, and checks on power within and between states, companies, and branches of government could be severely eroded.
尽管这些可能性仍存在很大不确定性，但高风险值得进一步认真关注。政策制定者应紧急提高对 AI 研发自动化的可见性，开发引导和约束智能爆炸的方法，并让社会做好准备以适应智能爆炸的影响。
Although there remains much uncertainty about these possibilities, the high stakes warrant serious further attention. Policymakers should urgently obtain more visibility into the automation of AI R&D, develop ways to steer and constrain an intelligence explosion, and prepare society to adapt to an intelligence explosion’s impacts.
技术论证并不新鲜。如果你正在读这篇文章，你不需要它。重要的技术问题是，收益递减是否会超过容量和能力的加速（即 r<1），而他们认为可能不会：
The technical argument is nothing new. If you’re reading this, you don’t need it. The technical question that matters is whether diminishing returns will outpace the acceleration of capacity and capabilities (aka is r<1), and they say probably not:
利用 AI 进展的历史数据，Ho 和 Whitfill 发现 AI 研究三个子领域中 r 的中心估计值在 1.2 到 1.9 之间。尽管不确定性很大，这些结果表明完全自动化后会出现急剧加速：如果 r 保持在这些水平且没有其他瓶颈出现，AI 进展的速度将在约 1.5 年内增加十倍，届时按今天的速度一年的进展将只需约五周。
Using historical data on AI progress, Ho and Whitfill find central estimates of r between 1.2 and 1.9 across three subfields of AI research. Though uncertainty is substantial, these results suggest radical acceleration after full automation: if r stayed at these levels and no other bottlenecks emerged, the pace of AI progress would increase tenfold within about 1.5 years, at which point a year’s worth of progress at today’s pace would take about five weeks.
他们还研究了潜在瓶颈：算力、数据、难以自动化的任务、时间密集型流程。我同意他们的结论，即这些限制可能产生约束，但不太可能阻止快速加速。
They also examine potential bottlenecks: Compute, data, hard-to-automate tasks, time-intensive processes. I agree with their conclusion that such limitations could potentially bind but are unlikely to prevent rapid acceleration.
同样，关于社会影响的部分是常见的观点：超出社会引导和适应的能力、失去监督和控制、权力制衡的侵蚀。
Similarly, the section on societal impacts is the usual points: Outpacing society’s ability to steer and adapt, loss of oversight and control, erosion of checks on power.
这里重要的是作者联盟以及他们提议要做的事情。
What is important here is the coalition of authors, and what they propose doing.
他们建议我们紧急获得对 AI 研发自动化的可见性，并弄清楚如何引导和约束智能爆炸，比如控制前沿发展速度并确保保障措施到位，然后我们必须适应事件的发展。
They recommend that we urgently obtain visibility into AI R&D automation, and figure out how to steer and constrain an intelligence explosion, as in pace the frontier and ensure safeguards are in place, and that we then must adapt to events.
这份要求清单是好的。正如你对一份拥有这样作者名单的论文所预期的那样，这篇论文最终还是在修辞上有所保留，至少按照我承认的较高标准来看是这样。有许多可怕的警告和紧急行动与这种保留完全兼容，所以这仍然要求我们做很多事情。奥弗顿窗口已经移动，因此这种保留与六个月或一年前你期望从 Bengio 风格的论文中看到的水平不同，而且是往好的方向。
The list of asks is good. As you would expect with something that gets this list of authors, the paper still ultimately pulls many of its rhetorical punches, at least by my admittedly high standard. There are plenty of dire warnings and urgent actions that are fully compatible with punch pulling, so there is still a lot this calls on us to do. The Overton Window has shifted, so this pulling of punches is on a different level than what you’d expect from a Bengio-style paper six months or a year ago, in a good way.
我们仍然没有完全明确地说出全部主要问题。这篇论文实际上并没有解决一个 AI 超级智能的世界所涉及的许多核心问题，并且将‘失控’视为没有我希望的那么多齿轮的东西，也没有理解保持控制将多么不自然以及即使是在好的情况下也会伴随的困难。
We’re still not quite explicitly saying the full main thing. The paper does not actually tackle many of the core issues involved in a world where the AIs are superintelligent, and treats ‘loss of control’ as something without as many gears as I would like, and without an understanding of how unnatural it would be to stay in control and the difficulties that go along with that even in good cases.
有了这样的论文和研究团队，我们仍然更接近于“空谈”而非“实干”。
With papers and groups like this, we are still a lot closer to talking the talk.
这是实际的进展。所要求的第一批政策步骤是正确的。
This is actual progress. The first requested policy steps are correct.
OpenAI 的公告也是如此。这是实际的进展，是一个开端。
The same goes for OpenAI's announcements. Actual progress. A beginning.