We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data, efficiently improving our datasets and models. Finally, we investigate the robustness of RLHF training, and identify a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization. Alongside our main results, we perform peripheral analyses on calibration, competing objectives, and the use of OOD detection, compare our models with human writers, and provide samples from our models using prompts appearing in recent related work.
核心贡献 · Key contributions
应用基于人类反馈的强化学习（RLHF）微调语言模型，使其成为有用且无害的助手，提升了几乎所有自然语言处理评估的性能。 Applied RLHF to finetune language models as helpful and harmless assistants, improving NLP evaluations.
证明对齐训练与 Python 编程和摘要等专业技能完全兼容。 Demonstrated compatibility of alignment training with specialized skills like Python coding and summarization.
探索了迭代在线训练模式，每周更新偏好模型和强化学习策略，有效改进了数据集和模型。 Explored iterated online training with weekly updates of preference models and RL policies, improving data and models.
发现强化学习奖励与策略初始化的 KL 散度平方根之间存在近似线性关系。 Identified a linear relation between RL reward and square root of KL divergence from initial policy.
对校准、竞争目标和分布外检测进行了外围分析。 Conducted peripheral analyses on calibration, competing objectives, and OOD detection.
将模型与人类写手进行比较，并使用近期相关工作提示提供了模型样本。 Compared models with human writers and provided samples from prompts in recent related work.
局限 · Limitations
基于人类反馈的强化学习可能不是实现诚实的最佳方法；其他技术可能更有效。 RLHF may not be the best approach for honesty; other techniques may be more effective.
由于数据稀缺，偏好模型在较高分数下校准性和鲁棒性下降。 Preference models become less calibrated and robust at higher scores due to data scarcity.
在线训练需要谨慎处理分布偏移和熵减少问题。 Online training requires careful handling of distribution shift and entropy reduction.
如果未平衡，无害训练可能导致过度优化并降低有用性。 Harmlessness training can lead to over-optimization and reduced helpfulness if not balanced.
结果可能无法推广到现实任务或对抗性评估。 Results may not generalize to real-world tasks or adversarial evaluations.
论文章节 · Sections（共 55）
摘要Abstract
1 引言1 Introduction
1.1 贡献1.1 Contributions
1.2 评估与指标概述1.2 Summary of Evaluations and Metrics
1.3 相关工作1.3 Related Work
2 数据收集2 Data Collection
2.1 任务规范与众包工人2.1 Task Specification and Crowdworkers
2.2 有用性与无害性（红队）数据集2.2 Helpfulness and Harmlessness (Red Teaming) Datasets
2.3 部署到反馈接口的模型及相关数据分布2.3 Models Deployed to the Feedback Interface and Associated Data Distributions
2.4 使用 Elo 分数比较模型2.4 Comparing Models with Elo Scores
3.1 模型与训练设置3.1 Models and Training Setup
3.2 基础缩放结果3.2 Basic Scaling Results
3.3 偏好模型的校准及其对强化学习的影响3.3 Calibration of Preference Models and Implications for RL
3.4 评估有用与无害偏好模型3.4 Evaluating Helpful and Harmless Preference Models
4.1 训练设置4.1 Training Setup
4.2 鲁棒性实验4.2 Robustness Experiments
4.3 DKL 与奖励的近似线性关系4.3 An Approximately Linear Relation Between DKLsubscript𝐷KL\sqrt{D_{\rm KL}} and Reward
4.4 RLHF 训练中有用性与无害性之间的张力4.4 Tension Between Helpfulness and Harmlessness in RLHF Training
4.5 迭代在线 RLHF4.5 Iterated Online RLHF
4.6 评估：对齐奖励、诚实性与偏见4.6 Evaluations: Alignment Bonus, Honesty, and Biases
5 竞争目标、专业技能与分布外检测5 Competing Objectives, Specialized Skills, and OOD Detection
5.1 混合有用与无害目标5.1 Mixing Helpful and Harmless Objectives
5.2 作为专业技能的摘要生成5.2 Summarization as a Specialized Skill
5.3 对代码微调模型进行自然语言 RLHF5.3 Natural Language RLHF on Code-Finetuned Models
5.4 应用分布外检测拒绝奇怪或有害请求5.4 Applying Out-of-Distribution Detection to Reject Strange or Harmful Requests
6 定性示例与比较6 Qualitative Examples and Comparisons
6.1 与人类写手的比较6.1 Comparison with Human Writers
6.2 敏感问题：回避与参与6.2 Sensitive Questions and Avoidance versus Engagement
6.3 示例对话6.3 Example Dialogues
7 讨论7 Discussion
7.1 局限性7.1 Limitations
7.2 对齐数据作为公共产品7.2 Alignment Data as a Public Good
7.3 更广泛的影响7.3 Broader Impacts
致谢Acknowledgments
作者贡献Author Contributions
A.1 上下文蒸馏A.1 Context Distillation
A.2 偏好建模A.2 Preference Modeling
A.3 偏好模型随模型与数据集大小的缩放A.3 Scaling of PM with Model and Dataset Size
B.1 训练设置B.1 Training Setup
B.2 更多鲁棒性研究B.2 More on Robustness Studies
B.3 “在线”RLHF 细节B.3 Details of ‘Online’ RLHF
B.4 “在线”RLHF 的鲁棒性B.4 Robustness of ‘Online’ RLHF
B.5 众包工人比较与 Elo 分数B.5 Crowdworker Comparisons and Elo Scores
B.6 拒绝采样模型的 Elo 分数B.6 Elo Scores for Rejection Sampling Models
B.7 Stack Overflow 结果B.7 Stack Overflow Results
B.8 代码模型快照上 RLHF 的进一步分析B.8 Further Analysis of RLHF on Code-Model Snapshots
B.9 应用分布外检测拒绝奇怪或有害请求的细节B.9 Details of Applying Out-of-Distribution Detection to Reject Strange or Harmful Requests
B.10 性别偏见评估细节B.10 Gender Bias Evaluation Details
附录 C PALMS、LaMDA 和 InstructGPT 提示样本Appendix C Samples from PALMS, LaMDA, and InstructGPT Prompts
C.1 PALMS 敏感问题C.1 PALMS Sensitive Questions
C.2 InstructGPT 提示C.2 InstructGPT Prompts
D.1 概述D.1 Overview
D.2 指令与界面D.2 Instructions and Interface
D.3 数据质量测量挑战D.3 Data Quality Measurement Challenges
附录 E NLP 评估格式与提示细节Appendix E Details on NLP Evaluations Formatting and Prompts