深度强化学习中的双 Q 学习

Deep Reinforcement Learning with Double Q-learning

哈多·范哈塞尔特 Hado van Hasselt · Google DeepMind · 2015-09-22 · arXiv:1509.06461 ↗ · 被引 9180

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

流行的 Q 学习算法在某些条件下会高估动作值。此前尚不清楚在实践中这种高估是否常见、是否损害性能，以及是否通常可以避免。在本文中，我们对所有这些问题给出了肯定答案。具体来说，我们首先展示了最近的 DQN 算法（将 Q 学习与深度神经网络相结合）在 Atari 2600 领域的某些游戏中存在严重的高估问题。然后，我们展示了在表格设置中引入的双 Q 学习算法的思想可以推广到大规模函数逼近。我们提出了对 DQN 算法的具体改编，并表明所得算法不仅如假设那样减少了观察到的高估，而且在多个游戏上带来了更好的性能。

The popular Q-learning algorithm is known to overestimate action values under certain conditions. It was not previously known whether, in practice, such overestimations are common, whether they harm performance, and whether they can generally be prevented. In this paper, we answer all these questions affirmatively. In particular, we first show that the recent DQN algorithm, which combines Q-learning with a deep neural network, suffers from substantial overestimations in some games in the Atari 2600 domain. We then show that the idea behind the Double Q-learning algorithm, which was introduced in a tabular setting, can be generalized to work with large-scale function approximation. We propose a specific adaptation to the DQN algorithm and show that the resulting algorithm not only reduces the observed overestimations, as hypothesized, but that this also leads to much better performance on several games.

核心贡献 · Key contributions

• 证明 Q-learning 在大规模确定性问题中因固有估计误差而高估动作值。
  Shows Q-learning overestimates action values in large-scale deterministic problems due to inherent estimation errors.
• 展示 DQN 在 Atari 2600 游戏中存在严重高估，损害性能。
  Demonstrates DQN suffers substantial overestimations in Atari 2600 games, harming performance.
• 将 Double Q-learning 推广到深度神经网络的大规模函数逼近。
  Generalizes Double Q-learning to large-scale function approximation with deep neural networks.
• 提出 Double DQN 算法，减少高估并提升策略质量。
  Proposes Double DQN algorithm, reducing overestimations and improving policy quality.
• Double DQN 在 Atari 2600 领域取得最先进结果。
  Achieves state-of-the-art results on Atari 2600 domain with Double DQN.

局限 · Limitations

• 评估仅限于确定性 Atari 2600 游戏，未测试随机环境。
  Evaluation limited to deterministic Atari 2600 games; stochastic environments not tested.
• 超参数为 DQN 调优，主要比较中未针对 Double DQN 优化。
  Hyperparameters tuned for DQN, not optimized for Double DQN in main comparison.
• Double DQN 可能无法完全消除高估，残留偏差仍可能存在。
  Double DQN may not eliminate overestimations entirely; residual bias possible.
• 理论分析假设无偏估计，实际偏差可能影响保证。
  Theoretical analysis assumes unbiased estimates; practical violations may affect guarantees.
• 结果可能无法推广到 Atari 2600 以外的领域。
  Results may not generalize to other domains beyond Atari 2600.

论文章节 · Sections（共 12）

• 摘要 Abstract
• 背景 Background
• 深度 Q 网络 Deep Q Networks
• 双 Q 学习 Double Q-learning
• 估计误差导致的过度乐观 Overoptimism due to estimation errors
• 双 DQN Double DQN
• 实证结果 Empirical results
• 过度乐观的结果 Results on overoptimism
• 学习策略的质量 Quality of the learned policies
• 对人类起始状态的鲁棒性 Robustness to Human starts
• 讨论 Discussion
• 致谢 Acknowledgments

阅读逐段中英对照全文 →