Learning to summarize from human feedback
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→随着语言模型变得更加强大，训练和评估越来越受到特定任务数据和指标的瓶颈限制。例如，摘要模型通常被训练来预测人类参考摘要，并使用 ROUGE 进行评估，但这两个指标都是我们真正关心的摘要质量的粗略代理。在这项工作中，我们展示了通过训练模型优化人类偏好，可以显著提高摘要质量。我们收集了一个大规模、高质量的人类摘要比较数据集，训练了一个模型来预测人类偏好的摘要，并使用该模型作为奖励函数，通过强化学习微调摘要策略。我们将方法应用于 Reddit 帖子的 TL;DR 数据集版本，发现我们的模型显著优于人类参考摘要和仅通过监督学习微调的更大模型。我们的模型还迁移到 CNN/DM 新闻文章，无需任何新闻特定微调就能生成几乎与人类参考一样好的摘要。我们进行了广泛的分析以理解我们的人类反馈数据集和微调模型。我们确定奖励模型能泛化到新数据集，并且根据人类判断，优化奖励模型比优化 ROUGE 能产生更好的摘要。我们希望论文中的证据能激励机器学习研究者更关注他们的训练损失如何影响他们真正想要的模型行为。
As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about -- summary quality. In this work, we show that it is possible to significantly improve summary quality by training a model to optimize for human preferences. We collect a large, high-quality dataset of human comparisons between summaries, train a model to predict the human-preferred summary, and use that model as a reward function to fine-tune a summarization policy using reinforcement learning. We apply our method to a version of the TL;DR dataset of Reddit posts and find that our models significantly outperform both human reference summaries and much larger models fine-tuned with supervised learning alone. Our models also transfer to CNN/DM news articles, producing summaries nearly as good as the human reference without any news-specific fine-tuning. We conduct extensive analyses to understand our human feedback dataset and fine-tuned models We establish that our reward model generalizes to new datasets, and that optimizing our reward model results in better summaries than optimizing ROUGE according to humans. We hope the evidence from our paper motivates machine learning researchers to pay closer attention to how their training loss affects the model behavior they actually want.