We study whether language models can evaluate the validity of their own claims and predict which questions they will be able to answer correctly. We first show that larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format. Thus we can approach self-evaluation on open-ended sampling tasks by asking models to first propose answers, and then to evaluate the probability "P(True)" that their answers are correct. We find encouraging performance, calibration, and scaling for P(True) on a diverse array of tasks. Performance at self-evaluation further improves when we allow models to consider many of their own samples before predicting the validity of one specific possibility. Next, we investigate whether models can be trained to predict "P(IK)", the probability that "I know" the answer to a question, without reference to any particular proposed answer. Models perform well at predicting P(IK) and partially generalize across tasks, though they struggle with calibration of P(IK) on new tasks. The predicted P(IK) probabilities also increase appropriately in the presence of relevant source materials in the context, and in the presence of hints towards the solution of mathematical word problems. We hope these observations lay the groundwork for training more honest models, and for investigating how honesty generalizes to cases where models are trained on objectives other than the imitation of human writing.
核心贡献 · Key contributions
较大的语言模型在适当格式化的多种选择题和是非题上校准良好。 Larger language models are well-calibrated on diverse multiple-choice and true/false questions when formatted appropriately.
通过 P(True)进行自我评估在各项任务上表现出令人鼓舞的性能、校准和规模扩展。 Self-evaluation via P(True) shows encouraging performance, calibration, and scaling across tasks.
向模型提供其自身的多个样本可提升自我评估性能。 Providing models with many of their own samples improves self-evaluation performance.
模型可被训练预测 P(IK)并在任务间部分泛化，但新任务上的校准具有挑战性。 Models can be trained to predict P(IK) and partially generalize across tasks, though calibration on new tasks is challenging.
当上下文中提供相关源材料或提示时，P(IK)会适当增加。 P(IK) increases appropriately when relevant source materials or hints are provided in context.
自我评估和校准随模型规模提升，表明验证比生成扩展更快。 Self-evaluation and calibration improve with model size, suggesting verification scales faster than generation.
局限 · Limitations
研究聚焦于预训练语言模型；RLHF 策略需温度调整以实现校准。 Study focuses on pretrained language models; RLHF policies require temperature adjustment for calibration.
P(IK)泛化不完全，且校准在分布外表现不佳。 P(IK) generalization is partial and calibration suffers out-of-distribution.
零样本自我评估校准不佳；需少样本提示才能获得良好校准。 Self-evaluation is zero-shot poorly calibrated; few-shot prompting is needed for good calibration.
工作未涉及有意欺骗，也未区分真理与人类共识。 Work does not address intentional deception or distinguish truth from human consensus.
仅研究了五个基于采样的数据集，限制了泛化观察的范围。 Only five sampling-based datasets were studied, limiting scope of generalization observations.
论文章节 · Sections（共 38）
摘要Abstract
1 引言1 Introduction
1.1 贡献1.1 Contributions
术语表：可观测值与度量Glossary: Observables and Metrics
1.2 模型与评估任务1.2 Models and Evaluation Tasks
1.3 相关工作1.3 Related Work
2 更大模型在多样化选择题上校准良好2 Larger Models are Calibrated on Diverse Multiple Choice Questions
3 从校准到自知之明3 From Calibration to Knowing What You Know
3.1 用“以上皆非”替换选项损害性能与校准3.1 Replacing an Option with ‘None of the Above’ Harms Performance and Calibration
3.2 模型在真/假任务上校准良好3.2 Models are Well-Calibrated on True/False Tasks
3.3 RLHF 策略误校准可通过温度调节修复3.3 RLHF Policy Miscalibration Can Be Remediated with a Temperature Tuning
4 询问 AI：你提出的答案是真还是假？4 Ask the AI: Is your proposed answer True or False?
4.1 基础自我评估4.1 Basic Self-Evaluation
4.2 展示多个 T=1 样本改进自我评估4.2 Showing Many T=1𝑇1T=1 Samples Improves Self-Evaluation
5 训练模型预测能否正确回答问题5 Training Models to Predict Whether They Can Answer Questions Correctly
5.1 评估 P(IK)训练与模型规模趋势5.1 Evaluating P(IK) Training and Model Size Trends
5.2 P(IK)的分布外泛化5.2 Out of Distribution Generalization of P(IK)
5.3 P(IK)泛化以考虑源材料5.3 P(IK) Generalizes to Account for Source Materials
5.4 P(IK)泛化以考虑 GSM8k 解题提示5.4 P(IK) Generalizes to Account for Hints Towards GSM8k Solutions
5.5 比较不同预训练分布训练的模型5.5 Comparing Models Trained with Distinct Pretraining Distributions
6 讨论6 Discussion
6.1 局限性与未来工作6.1 Limitations and Future Work
6.2 更广泛的影响6.2 Broader Impacts
7 贡献声明7 Contribution Statement
致谢Acknowledgments
A.1 校准图A.1 Calibration Charts
A.2 期望校准误差A.2 Expected Calibration Error
A.3 布里尔分数A.3 Brier Score
A.4 BIG Bench 默认格式A.4 BIG Bench Default Format
A.5 自我评估提示A.5 Prompt for Self-Evaluation
A.6 P(IK)训练A.6 Training for P(IK)
A.7 P(IK)问题格式A.7 Formatting of Questions for P(IK)
附录 B 用熵或损失区分模型所知Appendix B Discriminating What Models Know with Entropy or Loss
B.1 样本损失B.1 Loss of the Sample
B.2 答案分布的熵B.2 Entropy of the Answer Distribution
B.3 将采样答案填入单一上下文的损失B.3 Loss of Sampled Answers Stuffed into a Single Context
附录 C 更多 P(True)评估结果与细节Appendix C More P(True) Evaluation Results and Details
附录 D 混合算术与函数合成数据集描述Appendix D Mixed-Arithmetic and Function Synthesis Dataset Descriptions