While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.
核心贡献 · Key contributions
提出直接偏好优化（DPO），一种无需强化学习的简单算法，用于从偏好中训练语言模型。 Introduces Direct Preference Optimization (DPO), a simple RL-free algorithm for training language models from preferences.
推导出奖励函数与最优策略之间的闭式映射，实现直接策略优化。 Derives a closed-form mapping between reward functions and optimal policies, enabling direct policy optimization.
证明 DPO 优化与 RLHF 相同的目标，但使用简单的二元交叉熵损失，消除了强化学习的复杂性。 Shows DPO optimizes the same objective as RLHF but with a simple binary cross-entropy loss, eliminating RL complexity.
在受控情感生成中，DPO 实现了比 PPO 更好的奖励-KL 权衡。 Demonstrates DPO achieves better reward-KL tradeoff than PPO in controlled sentiment generation.
在摘要和对话任务中，DPO 匹配或超越基于 PPO 的 RLHF，模型参数高达 6B。 Shows DPO matches or exceeds PPO-based RLHF in summarization and dialogue tasks with up to 6B parameter models.
提供理论证明，DPO 的奖励参数化不限制可表示奖励模型的类别。 Provides theoretical justification that DPO's reward parameterization does not constrain the class of representable reward models.
局限 · Limitations
与显式奖励学习相比，DPO 的分布外泛化需要进一步研究。 DPO's out-of-distribution generalization compared to explicit reward learning requires further study.
直接偏好优化中的奖励过度优化现象尚未完全理解。 Reward over-optimization in direct preference optimization setting is not fully understood.
DPO 的评估仅限于 6B 参数模型；扩展到更大模型是未来工作。 Evaluation of DPO is limited to models up to 6B parameters; scaling to larger models is future work.
GPT-4 的胜率对提示设计敏感，影响评估可靠性。 GPT-4 win rates are sensitive to prompt design, affecting evaluation reliability.
DPO 需要参考策略和离线偏好数据集，限制了在线适应能力。 DPO requires a reference policy and offline preference dataset, limiting online adaptation.
论文章节 · Sections（共 15）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Work
3 预备知识3 Preliminaries
4 直接偏好优化4 Direct Preference Optimization
5 DPO 的理论分析5 Theoretical Analysis of DPO
5.1 语言模型实为奖励模型5.1 Your Language Model Is Secretly a Reward Model
5.2 演员-评论家算法的不稳定性5.2 Instability of Actor-Critic Algorithms
6 实验6 Experiments
6.1 DPO 优化 RLHF 目标的效果6.1 How well can DPO optimize the RLHF objective?
6.2 DPO 在真实偏好数据集上的扩展性6.2 Can DPO scale to real preference datasets?
6.3 对新输入分布的泛化6.3 Generalization to a new input distribution
6.4 用人类判断验证 GPT-4 判断6.4 Validating GPT-4 judgments with human judgments