我们对对齐研究的方法 | OpenAI 我们正在提升 AI 系统从人类反馈中学习以及协助人类评估 AI 的能力。我们的目标是构建一个足够对齐的 AI 系统，能够帮助我们解决所有其他对齐问题。* 使用人类反馈训练 AI 系统
Our approach to alignment research | OpenAI We are improving our AI systems’ ability to learn from human feedback and to assist humans at evaluating AI. Our goal is to build a sufficiently aligned AI system that can help us solve all other alignment problems. * Training AI systems using human feedback
核心贡献 · Key contributions
提出三支柱方法：用人类反馈训练 AI、辅助人类评估、自动化对齐研究。 Proposes a three-pillar approach: training AI with human feedback, assisting human evaluation, and automating alignment research.
证明经 RLHF 微调的 InstructGPT 优于大 100 倍的预训练模型，算力成本不到 2%。 Demonstrates that InstructGPT, fine-tuned with RLHF, is preferred over a 100x larger pretrained model with <2% compute cost.
展示 AI 辅助评估使摘要任务中发现的缺陷增加 50%。 Shows that AI-assisted evaluation increases flaws found in model outputs by 50% on summarization tasks.
倡导迭代、经验性的对齐研究，以理解 Scaling（规模扩张）和失败模式。 Advocates for iterative, empirical alignment research to understand scaling and failure modes.
旨在构建对齐的 AI 系统以加速对齐研究，可能递归地解决问题。 Aims to build aligned AI systems that can accelerate alignment research, potentially solving the problem recursively.
强调透明和公开分享对齐研究，以惠及所有 AGI 开发者。 Emphasizes transparency and open sharing of alignment research to benefit all AGI developers.
局限 · Limitations
RLHF 假设人类能准确评估任务，这对超人类 AI 系统失效。 RLHF assumes humans can accurately evaluate tasks, which fails for superhuman AI systems.
AI 辅助评估可能放大助手中的细微偏见或漏洞。 AI-assisted evaluation may amplify subtle biases or vulnerabilities in the assistant.
若出现重大不连续或范式转变，当前模型的对齐经验可能不适用。 Lessons from aligning current models may not transfer if major discontinuities or paradigm shifts occur.
最困难的对齐问题可能无法仅通过可扩展训练信号解决。 The hardest alignment problems may not be solvable by scalable training signals alone.
能加速对齐研究的模型若未对齐，本身可能危险。 Models capable of accelerating alignment research might themselves be dangerous if misaligned.
论文章节 · Sections（共 5）
概述Overview
使用人类反馈训练 AI 系统Training AI systems using human feedback
训练模型以辅助人类评估Training models to assist human evaluation
训练 AI 系统进行对齐研究Training AI systems to do alignment research