For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than one percent of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach, we show that we can successfully train complex novel behaviors with about an hour of human time. These behaviors and environments are considerably more complex than any that have been previously learned from human feedback.
核心贡献 · Key contributions
提出一种从人类对轨迹片段的偏好中训练深度强化学习智能体的方法，降低了人类监督成本。 Proposes a method to train deep RL agents from human preferences between trajectory segments, reducing human oversight cost.
展示了在不到 1%的智能体-环境交互被标注的情况下，解决复杂强化学习任务（Atari 游戏、机器人运动）。 Demonstrates solving complex RL tasks (Atari games, robot locomotion) with less than 1% of agent-environment interactions labeled.
展示了从约一小时的人类反馈中学习新颖行为（后空翻、单腿站立）。 Shows novel behaviors (backflip, one-leg stand) can be learned from about an hour of human feedback.
引入奖励预测器集成和在线查询选择以提高学习效率。 Introduces an ensemble of reward predictors and online query selection to improve learning efficiency.
提供了人类反馈可经济地扩展到最先进深度强化学习系统的证据。 Provides evidence that human feedback can be economically scaled to state-of-the-art deep RL systems.
局限 · Limitations
方法在短片段令人困惑的任务上可能失败，例如使用真实人类反馈的 Qbert。 Method may fail on tasks where short clips are confusing, e.g., Qbert with real human feedback.
离线奖励预测器训练因占用分布的非平稳性导致性能不佳。 Offline reward predictor training leads to poor performance due to nonstationary occupancy distribution.
人类反馈质量取决于标注者一致性；错误或不一致会降低性能。 Human feedback quality depends on rater consistency; errors or inconsistency can degrade performance.
方法假设人类偏好由奖励函数生成，这在所有领域可能不成立。 Method assumes human preferences are generated by a reward function, which may not hold in all domains.
基于集成方差的查询选择是粗略近似；更好的信息论方法留待未来工作。 Query selection based on ensemble variance is a crude approximation; better information-theoretic approaches remain future work.
论文章节 · Sections（共 10）
摘要Abstract
1 引言1 Introduction
1.1 相关工作1.1 Related Work
2.1 设定与目标2.1 Setting and Goal
2.2 我们的方法2.2 Our Method
3 实验结果3 Experimental Results
3.1 具有未观测奖励的强化学习任务3.1 Reinforcement Learning Tasks with Unobserved Rewards