随着 AI 系统能力增强，我们希望借助它们来监督其他 AI。我们实验了通过自我改进训练无害 AI 助手的方法，无需任何人类标签识别有害输出。唯一的人类监督是通过一系列规则或原则提供的，因此我们将该方法称为“宪法式 AI”。该过程包括监督学习和强化学习两个阶段。在监督阶段，我们从初始模型中采样，然后生成自我批评和修订，最后在修订后的响应上微调原始模型。在强化学习阶段，我们从微调后的模型中采样，使用模型评估两个样本中哪个更好，然后从该 AI 偏好数据集中训练偏好模型。接着，我们使用偏好模型作为奖励信号进行强化学习训练，即使用“来自 AI 反馈的强化学习”（RLAIF）。结果，我们能够训练出一个无害但不回避的 AI 助手，它通过解释对有害查询的反对意见来回应。监督学习和强化学习方法都可以利用思维链式推理来提高人类判断的性能和 AI 决策的透明度。这些方法使得更精确地控制 AI 行为成为可能，且所需的人类标签大大减少。
As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses. In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use 'RL from AI Feedback' (RLAIF). As a result we are able to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them. Both the SL and RL methods can leverage chain-of-thought style reasoning to improve the human-judged performance and transparency of AI decision making. These methods make it possible to control AI behavior more precisely and with far fewer human labels.
核心贡献 · Key contributions
提出宪法 AI（CAI），无需人类有害性标签即可训练无害助手。 Proposes Constitutional AI (CAI) to train harmless assistants without human harmfulness labels.
展示监督式自我批评与修订以减少有害性。 Demonstrates supervised self-critique and revision to reduce harmfulness.
引入基于 AI 反馈的强化学习（RLAIF），使用模型生成的无害性偏好标签。 Introduces RLAIF using model-generated preference labels for harmlessness.
展示思维链推理提升无害性评估与透明度。 Shows chain-of-thought reasoning improves harmlessness evaluation and transparency.
通过解释对有害查询的反对，实现非回避的无害回应。 Achieves non-evasive harmless responses by explaining objections to harmful queries.
将人类监督缩减为简短原则列表，实现可扩展监督。 Reduces human supervision to a short list of principles, enabling scalable oversight.
局限 · Limitations
依赖人类编写的原则，可能较为临时且需完善。 Relies on human-written principles which may be ad hoc and require refinement.
仍需人类帮助性标签，并非完全自监督。 Still requires human labels for helpfulness; not fully self-supervised.
思维链可能产生不准确或夸大的批评。 Chain-of-thought may produce inaccurate or overstated critiques.
RL-CAI 模型过度训练时可能出现古德哈特效应或模板化回应。 RL-CAI models can exhibit Goodharting or boilerplate responses when overtrained.
方法可能降低训练有害 AI 系统的门槛（双重用途）。 Methods may lower barriers for training pernicious AI systems (dual use).
论文章节 · Sections（共 25）
摘要Abstract
1 引言1 Introduction
1.1 动机1.1 Motivations
1.2 宪法 AI 方法1.2 The Constitutional AI Approach
1.3 贡献1.3 Contributions
1.4 模型与数据1.4 Models and Data
2 评估 AI 监督 HHH 的潜力2 Evaluating the Potential for AI Supervision of HHH
3 宪法 AI：批评、修订与监督学习3 Constitutional AI: Critiques, Revisions, and Supervised Learning
3.1 方法3.1 Method
3.2 数据集与训练3.2 Datasets and Training
3.3 主要结果3.3 Main Results
3.4 扩展趋势3.4 Scaling Trends
3.5 批评是否必要？3.5 Are Critiques Necessary?
4 宪法 AI：基于 AI 反馈的强化学习4 Constitutional AI: Reinforcement Learning from AI Feedback