Given the broad capabilities of large language models, it should be possible to work towards a general-purpose, text-based assistant that is aligned with human values, meaning that it is helpful, honest, and harmless. As an initial foray in this direction we study simple baseline techniques and evaluations, such as prompting. We find that the benefits from modest interventions increase with model size, generalize to a variety of alignment evaluations, and do not compromise the performance of large models. Next we investigate scaling trends for several training objectives relevant to alignment, comparing imitation learning, binary discrimination, and ranked preference modeling. We find that ranked preference modeling performs much better than imitation learning, and often scales more favorably with model size. In contrast, binary discrimination typically performs and scales very similarly to imitation learning. Finally we study a `preference model pre-training' stage of training, with the goal of improving sample efficiency when finetuning on human preferences.
核心贡献 · Key contributions
表明提示随模型规模提升对齐评估效果，且对大型模型几乎没有对齐代价。 Shows that prompting improves alignment evaluations with model size, with negligible alignment tax on large models.
发现排序偏好建模优于模仿学习，且随模型规模扩展更有利。 Finds ranked preference modeling outperforms imitation learning and scales more favorably with model size.
提出偏好模型预训练（PMP），提升基于人类偏好微调的样本效率。 Introduces preference model pre-training (PMP) to improve sample efficiency when finetuning on human preferences.
证明二元判别与模仿学习表现相似，不同于排序偏好建模。 Demonstrates that binary discrimination performs similarly to imitation learning, unlike ranked preference modeling.
展示上下文蒸馏可将提示优势转移到模型权重，且不损害大型模型性能。 Shows context distillation can transfer prompt benefits into model weights without performance loss on large models.
提供证据表明，对于下游对齐任务，二元数据的 PMP 迁移效果优于排序数据。 Provides evidence that PMP with binary data transfers better than ranked data for downstream alignment tasks.
局限 · Limitations
对齐评估仅限于有用性、诚实性和无害性标准，这些标准具有主观性。 Alignment evaluations are limited to helpfulness, honesty, and harmlessness criteria, which are subjective.
提示可能无法扩展到超越简单模仿的更复杂对齐挑战。 Prompting may not scale to more complex alignment challenges beyond simple imitation.
PMP 的优势主要在超过 10 亿参数的模型中观察到；小模型改进甚微。 PMP benefits are mainly observed for models larger than 1B parameters; small models show little improvement.
该研究未涉及具有超人能力的先进 AI 系统的对齐问题。 The study does not address alignment of advanced AI systems with superhuman capabilities.
人工评估数据可能因标注者专业性和成本而存在偏差和可扩展性有限。 Human evaluation data may have biases and limited scalability due to annotator expertise and cost.
论文章节 · Sections（共 40）
摘要Abstract
1.1 动机1.1 Motivations
1.2 研究1.2 Research
1.3 贡献1.3 Contributions
2 基于对齐行为的条件化2 Conditioning on Aligned Behavior
2.1 上下文蒸馏2.1 Context Distillation
2.2 评估与对齐代价2.2 Evaluations and Alignment Taxes
3 偏好建模与模仿学习的规模效应3 Scaling of Preference Modeling vs Imitation Learning
3.1 偏好建模与模仿学习的损失与设置3.1 Loss and Settings for Preference Modeling and Imitation Learning
3.2 排序与二元偏好数据集的性能与规模结果3.2 Performance and Scaling Results for Ranked versus Binary Preference Datasets
4 偏好模型预训练与迁移4 Preference Model Pre-Training and Transfer
4.1 PMP 与数据集4.1 PMP and Datasets
4.2 微调结果与规模趋势4.2 Finetuning Results and Scaling Trends
4.3 排序偏好建模与二元判别在 PMP 中的对比4.3 Ranked Preference Modeling vs Binary Discrimination for PMP
4.4 人-模型与人-人比较在 PMP 中的对比4.4 Human-Model vs Human-Human Comparisons for PMP
5.1 相关工作5.1 Related Work
5.2 更广泛的影响5.2 Broader Impacts
5.3 启示5.3 Implications
作者贡献Author Contributions
致谢Acknowledgments
附录 A 语言模型预训练Appendix A Language Model Pre-training
附录 B 关于提示、上下文蒸馏和评估的更多细节Appendix B More Details on Prompting, Context Distillation, and Evaluations
B.1 对齐代价评估B.1 Alignment Tax Evaluations
B.2 毒性分析细节B.2 Toxicity Analysis Details
B.3 TruthfulQA 格式B.3 TruthfulQA Formatting
B.4 关于 Lambada 格式的说明B.4 A Comment on Lambada Formatting
B.5 上下文蒸馏微调B.5 Context Distillation Finetuning
B.6 重排序样本时准确率的估计B.6 Estimator of Accuracy When Re-Ranking Samples
C.1 偏好模型预训练C.1 Preference Model Pre-training
C.2 偏好模型预训练C.2 Preference Model Pre-Training
C.3 语言建模提升 PMP 迁移C.3 Language Modeling Improves PMP Transfer