为了使 AI 系统能够广泛用于具有挑战性的现实任务，我们需要它们学习复杂的人类目标和偏好。一种指定复杂目标的方法是在训练过程中让人类判断哪些智能体行为是安全且有用的，但如果任务过于复杂以至于人类无法直接判断，这种方法可能会失败。为了解决这个问题，我们提出通过零和辩论游戏中的自我对弈来训练智能体。给定一个问题或提议的行动，两个智能体轮流进行简短陈述，直到达到限制，然后由人类判断哪个智能体提供了最真实、最有用的信息。与复杂性理论类比，在最优辩论下，给定多项式时间的裁判，可以回答 PSPACE 中的任何问题（直接评判只能回答 NP 问题）。在实践中，辩论是否有效涉及关于人类和我们希望 AI 执行的任务的经验问题，以及关于 AI 对齐含义的理论问题。我们报告了在初始 MNIST 实验中的结果，其中智能体竞争以说服稀疏分类器，在给定 6 个像素的情况下，分类器的准确率从 59.4%提高到 88.9%，在给定 4 个像素的情况下，从 48.2%提高到 85.2%。最后，我们讨论了辩论模型的理论和实践方面，重点关注模型扩展时的潜在弱点，并提出了未来的人类和计算机实验来测试这些特性。
To make AI systems broadly useful for challenging real-world tasks, we need them to learn complex human goals and preferences. One approach to specifying complex goals asks humans to judge during training which agent behaviors are safe and useful, but this approach can fail if the task is too complicated for a human to directly judge. To help address this concern, we propose training agents via self play on a zero sum debate game. Given a question or proposed action, two agents take turns making short statements up to a limit, then a human judges which of the agents gave the most true, useful information. In an analogy to complexity theory, debate with optimal play can answer any question in PSPACE given polynomial time judges (direct judging answers only NP questions). In practice, whether debate works involves empirical questions about humans and the tasks we want AIs to perform, plus theoretical questions about the meaning of AI alignment. We report results on an initial MNIST experiment where agents compete to convince a sparse classifier, boosting the classifier's accuracy from 59.4% to 88.9% given 6 pixels and from 48.2% to 85.2% given 4 pixels. Finally, we discuss theoretical and practical aspects of the debate model, focusing on potential weaknesses as the model scales up, and we propose future human and computer experiments to test these properties.
核心贡献 · Key contributions
提出通过零和辩论游戏中的自我对弈来训练 AI 智能体，以对齐复杂目标。 Proposes training AI agents via self-play on a zero-sum debate game to align complex goals.
证明在多项式时间评判者下，最优辩论可回答任何 PSPACE 问题。 Shows debate with optimal play can answer any PSPACE problem given polynomial-time judges.
报告 MNIST 实验，辩论将稀疏分类器准确率从 59.4%提升至 88.9%（6 像素）。 Reports MNIST experiment where debate boosts sparse classifier accuracy from 59.4% to 88.9% with 6 pixels.
讨论理论和实践方面，包括模型规模扩张时的潜在弱点。 Discusses theoretical and practical aspects, including potential weaknesses as the model scales up.
提出未来的人类和计算机实验以测试辩论特性。 Proposes future human and computer experiments to test debate properties.
局限 · Limitations
若人类评判者存在偏见或无法理解长论证，辩论可能失败。 Debate may fail if human judges are biased or cannot follow long arguments.
随机模拟和统计论证可能需要深度或不切实际的辩论协议。 Stochastic simulations and statistical arguments may require deep or impractical debate protocols.
实际中近似最优博弈可能无法保证诚实行为。 Approximate optimal play may not guarantee honest behavior in practice.
由于需要人类输入，辩论可能不如其他 ML 方法有竞争力。 Debate may be uncompetitive with other ML approaches due to human input requirements.
安全漏洞（如心理黑客或代码利用）可能破坏辩论的安全性。 Security vulnerabilities like mind hacking or code exploits could undermine debate safety.
论文章节 · Sections（共 31）
摘要Abstract
1 引言1 Introduction
2 辩论博弈2 The debate game
2.1 简短辩论的强大之处2.1 Short debates are powerful
2.2 复杂性理论类比：DEBATE=PSPACE2.2 Complexity theory analogies: \DEBATE=\PSPACE\DEBATE\PSPACE\DEBATE=\PSPACE
2.3 去除过度简化2.3 Removing oversimplifications
3 实验：提议与初步结果3 Experiments: proposed and preliminary
3.1 机器学习实验：说服稀疏分类器3.1 ML experiment: convincing a sparse classifier
3.2 人类实验：猫 vs. 狗3.2 Human experiment: cat vs. dog
4 乐观的理由4 Reasons for optimism
4.1 智能体可以承认无知4.1 Agents can admit ignorance
4.2 无人被迫说谎4.2 No one is forced to lie
4.3 辩论是自我对弈4.3 Debate is self play
5 担忧的理由5 Reasons to worry
5.1 也许人类只愿听想听的话？5.1 Maybe humans believe whatever they want to hear?
5.2 人类能理解辩论吗？5.2 Will humans understand the debates?
5.3 诚实真的是上策吗？5.3 Is honesty actually the best policy?
5.4 为辩论训练的智能体会损失性能吗？5.4 Will agents trained for debate lose performance?
5.5 纳什均衡邻域的不确定性5.5 Uncertainty about the neighborhood around Nash equilibrium
5.6 训练中均衡是否稳定？5.6 Are equilibria stable during training?
5.7 缺乏辩论收敛性5.7 Lack of debate convergence
5.8 人类或代码中的安全漏洞5.8 Security vulnerabilities in the human or code
5.9 随机模拟5.9 Stochastic simulation
5.10 训练超人类 AI 变得有说服力听起来很危险！5.10 Training superhuman AI to be convincing sounds dangerous!
6.1 对称性6.1 Symmetry
6.2 智能体间共享激活6.2 Sharing activations between agents
6.3 每一步不同智能体6.3 Different agents each step
6.4 收益结构6.4 Payoff structure
7 与放大的近似等价7 Approximate equivalence with amplification