让我们逐步验证

Let's Verify Step by Step

亨特·莱特曼 Hunter Lightman · OpenAI · 2023-05-31 · arXiv:2305.20050 ↗ · 被引 3656

摘要 · Abstract

近年来，大型语言模型在执行复杂多步推理方面的能力有了很大提升。然而，即使是最先进的模型仍然经常产生逻辑错误。为了训练更可靠的模型，我们可以采用结果监督（对最终结果提供反馈）或过程监督（对每个中间推理步骤提供反馈）。考虑到训练可靠模型的重要性以及人工反馈的高昂成本，仔细比较这两种方法至关重要。近期已有研究开始进行这种比较，但仍有许多问题有待解答。我们进行了自己的调查，发现过程监督在训练模型解决具有挑战性的 MATH 数据集问题时显著优于结果监督。我们的过程监督模型在 MATH 测试集的一个代表性子集上解决了 78%的问题。此外，我们还表明主动学习显著提高了过程监督的效果。为了支持相关研究，我们还发布了 PRM800K，这是用于训练我们最佳奖励模型的完整数据集，包含 80 万个步骤级人工反馈标签。

In recent years, large language models have greatly improved in their ability to perform complex multi-step reasoning. However, even state-of-the-art models still regularly produce logical mistakes. To train more reliable models, we can turn either to outcome supervision, which provides feedback for a final result, or process supervision, which provides feedback for each intermediate reasoning step. Given the importance of training reliable models, and given the high cost of human feedback, it is important to carefully compare the both methods. Recent work has already begun this comparison, but many questions still remain. We conduct our own investigation, finding that process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset. Our process-supervised model solves 78% of problems from a representative subset of the MATH test set. Additionally, we show that active learning significantly improves the efficacy of process supervision. To support related research, we also release PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train our best reward model.

核心贡献 · Key contributions

过程监督在 MATH 数据集上训练奖励模型时显著优于结果监督。
Process supervision significantly outperforms outcome supervision for training reward models on the MATH dataset.
过程监督模型在 MATH 测试集的代表性子集上达到 78.2%的准确率。
The process-supervised model achieves 78.2% accuracy on a representative subset of the MATH test set.
主动学习将过程监督的数据效率提高了 2.6 倍。
Active learning improves data efficiency of process supervision by 2.6 times.
发布了 PRM800K 数据集，包含 80 万个步骤级人类反馈标签。
Release of PRM800K, a dataset of 800,000 step-level human feedback labels.
大规模 PRM 可监督较小模型，实现高效消融实验。
Large-scale PRM can supervise smaller models, enabling efficient ablations.
过程监督产生负对齐税，同时提升性能和对齐。
Process supervision incurs a negative alignment tax, improving both performance and alignment.

局限 · Limitations

结果仅限于数学推理；对其他领域的泛化性未知。
Results are limited to mathematical reasoning; generalization to other domains is unknown.
预训练数据中可能存在 MATH 问题的测试集污染。
Potential test set contamination from MATH problems in pretraining data.
迭代重训练的主动学习出现不稳定性且无改进。
Active learning with iterative retraining showed instability and no improvement.
大规模比较受训练集大小和选择偏差的混淆。
Comparison at large scale is confounded by different training set sizes and selection biases.
过程监督依赖昂贵的人类反馈进行数据收集。
Process supervision relies on expensive human feedback for data collection.

论文章节 · Sections（共 22）

摘要 Abstract
1 引言 1 Introduction
2 方法 2 Methods
2.1 范围 2.1 Scope
2.2 基础模型 2.2 Base Models
2.3 生成器 2.3 Generator
2.4 数据收集 2.4 Data Collection
2.5 结果监督奖励模型 2.5 Outcome-supervised Reward Models (ORMs)
2.6 过程监督奖励模型 2.6 Process-supervised Reward Models (PRMs)
3 大规模监督 3 Large-scale Supervision
4 小规模合成监督 4 Small-scale Synthetic Supervision
4.1 过程与结果监督 4.1 Process vs Outcome Supervision
4.2 主动学习 4.2 Active Learning
5 分布外泛化 5 OOD Generalization
6.1 信用分配 6.1 Credit Assignment
6.2 对齐影响 6.2 Alignment Impact
6.3 测试集污染 6.3 Test Set Contamination
7.1 结果与过程监督 7.1 Outcome vs Process Supervision
7.2 合成监督 7.2 Synthetic Supervision
7.3 自然语言推理 7.3 Natural Language Reasoning
8 结论 8 Conclusion
致谢 Acknowledgements

阅读逐段中英对照全文 →