我们为 o 系列模型引入了新的对齐策略，直接教授安全规范以及如何对其进行推理。我们提出了审慎对齐，这是一种训练范式，直接教导推理型大语言模型人类编写且可解释的安全规范文本，并训练它们在回答前显式地推理这些规范。我们使用审慎对齐来对齐 OpenAI 的 o 系列模型，使其能够利用思维链推理来反思用户提示，识别 OpenAI 内部政策中的相关文本，并起草更安全的响应。我们的方法实现了对 OpenAI 安全政策的高度精确遵守，且无需人工标注的思维链或答案。我们发现，o1 在内部和外部的多项安全基准测试中显著优于 GPT-4o 及其他最先进的大语言模型，并在许多具有挑战性的数据集上达到了饱和性能。我们相信这为提升安全性提供了一条令人兴奋的新路径，并且我们认为这是一个鼓舞人心的例子，展示了能力提升如何也能用于提升安全性。
Introducing our new alignment strategy for o-series models, which are directly taught safety specifications and how to reason over them. We introduce deliberative alignment, a training paradigm that directly teaches reasoning LLMs the text of human-written and interpretable safety specifications, and trains them to reason explicitly about these specifications before answering. We used deliberative alignment to align OpenAI’s o-series models, enabling them to use chain-of-thought (CoT) reasoning to reflect on user prompts, identify relevant text from OpenAI’s internal policies, and draft safer responses. Our approach achieves highly precise adherence to OpenAI’s safety policies, and without requiring human-labeled CoTs or answers. We find that o1 dramatically outperforms GPT‑4o and other state-of-the art LLMs across a range of internal and external safety benchmarks, and saturates performance on many challenging datasets.
核心贡献 · Key contributions
提出 deliberative alignment，训练 LLM 在推理时对安全规范进行推理。 Introduces deliberative alignment, training LLMs to reason over safety specifications at inference time.
在拒绝不足和过度拒绝上实现帕累托改进，性能优于 GPT-4o 等模型。 Achieves Pareto improvement on both under- and over-refusals, outperforming GPT-4o and others.
使用合成数据生成，无需人工标注的思维链或答案，实现可扩展的对齐。 Uses synthetic data generation without human-labeled CoTs or answers, enabling scalable alignment.
结合 SFT 和 RL，使用策略感知奖励模型教授安全规范。 Combines SFT and RL with a policy-aware reward model to teach safety specifications.
展示了对分布外安全场景的强大泛化能力。 Demonstrates strong generalization to out-of-distribution safety scenarios.
局限 · Limitations
依赖人工编写的安全规范，可能不完整或存在偏见。 Relies on access to human-written safety specifications, which may be incomplete or biased.
需要具备思维链能力的推理模型（o 系列），限制了适用性。 Requires a reasoning model (o-series) with chain-of-thought capability, limiting applicability.
合成数据生成可能引入伪影或无法覆盖边缘情况。 Synthetic data generation may introduce artifacts or fail to cover edge cases.
有效性取决于奖励模型的质量及其对策略的访问。 Effectiveness depends on the quality of the reward model and its access to policies.
如果规范被规避或推理被操纵，可能无法防止滥用。 May not prevent misuse if specifications are circumvented or reasoning is manipulated.