We fine-tune GPT-3 to answer long-form questions using a text-based web-browsing environment, which allows the model to search and navigate the web. By setting up the task so that it can be performed by humans, we are able to train models on the task using imitation learning, and then optimize answer quality with human feedback. To make human evaluation of factual accuracy easier, models must collect references while browsing in support of their answers. We train and evaluate our models on ELI5, a dataset of questions asked by Reddit users. Our best model is obtained by fine-tuning GPT-3 using behavior cloning, and then performing rejection sampling against a reward model trained to predict human preferences. This model's answers are preferred by humans 56% of the time to those of our human demonstrators, and 69% of the time to the highest-voted answer from Reddit.
核心贡献 · Key contributions
微调 GPT-3 以使用基于文本的网络浏览环境回答长形式问题。 Fine-tunes GPT-3 to answer long-form questions using a text-based web-browsing environment.
收集人类示范和比较，通过模仿学习和人类反馈进行训练。 Collects human demonstrations and comparisons to train via imitation learning and human feedback.
要求模型在浏览过程中收集参考文献以帮助评估事实准确性。 Requires models to collect references during browsing to aid factual accuracy evaluation.
在 ELI5 上达到人类水平，答案比人类示范者偏好 56%，比 Reddit 最高票答案偏好 69%。 Achieves human-level performance on ELI5, with answers preferred 56% over human demonstrators and 69% over Reddit top answers.
在 TruthfulQA 上优于 GPT-3，达到 75%真实和 54%真实且信息丰富的答案。 Outperforms GPT-3 on TruthfulQA, achieving 75% truthful and 54% truthful and informative answers.
分析了数据集大小、参数数量和推理时算力对拒绝采样的缩放趋势。 Analyzes scaling trends with dataset size, parameter count, and inference-time compute for rejection sampling.
局限 · Limitations
在分布外问题上表现不佳，如在 TruthfulQA 上性能低于人类。 Struggles with out-of-distribution questions, as shown by lower performance on TruthfulQA compared to humans.
可能挑选参考文献以显得有说服力，而非反映公平的证据评估。 May cherry-pick references to appear convincing, not reflecting fair evidence assessment.
延续 GPT-3 和现有来源的偏见，可能强化确认偏误。 Perpetuates biases from GPT-3 and existing sources, potentially reinforcing confirmation bias.
由于引用显得权威，存在过度依赖风险，尽管偶尔有错误陈述。 Risk of overreliance due to authoritative appearance with citations, despite occasional falsehoods.
需要谨慎的人类监督以确保安全，尤其是更强大的模型访问网络时。 Requires careful human oversight for safety, especially with more capable models accessing the web.
论文章节 · Sections（共 20）
摘要Abstract
1 引言1 Introduction
2 环境设计2 Environment design
3.1 数据收集3.1 Data collection
3.2 训练3.2 Training
4 评估4 Evaluation
4.1 ELI54.1 ELI5
4.2 TruthfulQA4.2 TruthfulQA
4.3 TriviaQA4.3 TriviaQA
5.1 训练方法比较5.1 Comparison of training methods
5.2 规模实验5.2 Scaling experiments
6.1 WebGPT 的真实性6.1 Truthfulness of WebGPT
6.2 WebGPT 的感知真实性6.2 Perceived truthfulness of WebGPT
6.3 偏见的强化6.3 Reinforcement of bias
6.4 使用参考文献评估事实准确性6.4 Using references to evaluate factual accuracy