In this paper we argue for the fundamental importance of the value distribution: the distribution of the random return received by a reinforcement learning agent. This is in contrast to the common approach to reinforcement learning which models the expectation of this return, or value. Although there is an established body of literature studying the value distribution, thus far it has always been used for a specific purpose such as implementing risk-aware behaviour. We begin with theoretical results in both the policy evaluation and control settings, exposing a significant distributional instability in the latter. We then use the distributional perspective to design a new algorithm which applies Bellman's equation to the learning of approximate value distributions. We evaluate our algorithm using the suite of games from the Arcade Learning Environment. We obtain both state-of-the-art results and anecdotal evidence demonstrating the importance of the value distribution in approximate reinforcement learning. Finally, we combine theoretical and empirical evidence to highlight the ways in which the value distribution impacts learning in the approximate setting.
核心贡献 · Key contributions
论证了价值分布在强化学习中的根本重要性。 Argues for the fundamental importance of the value distribution in reinforcement learning.
证明了分布贝尔曼算子在策略评估的 Wasserstein 度量下是压缩映射。 Proves the distributional Bellman operator is a contraction in the Wasserstein metric for policy evaluation.
识别了控制设置中分布贝尔曼最优算子的不稳定性。 Identifies instability in the distributional Bellman optimality operator for control.
提出了 Categorical DQN，一种学习近似价值分布的新算法。 Introduces Categorical DQN, a new algorithm learning approximate value distributions.
在 Atari 2600 游戏上取得了最先进的结果，优于 DQN 和其他基线。 Achieves state-of-the-art results on Atari 2600 games, outperforming DQN and other baselines.
提供了理论和经验证据，表明价值分布改进了近似强化学习。 Provides theoretical and empirical evidence that value distributions improve approximate reinforcement learning.
局限 · Limitations
理论结果假设有界矩和有限动作空间。 Theoretical results assume bounded moments and finite action spaces.
经验评估仅限于 Atari 2600 游戏；对其他领域的泛化性未验证。 Empirical evaluation limited to Atari 2600 games; generalizability to other domains is unverified.
Categorical DQN 需要仔细调整支持范围和原子数量。 Categorical DQN requires careful tuning of support bounds and number of atoms.
分布最优算子在控制设置中可能不收敛到不动点。 The distributional optimality operator may not converge to a fixed point in control settings.
算法性能依赖于投影和损失函数的选择。 Algorithm performance depends on the choice of projection and loss function.
论文章节 · Sections（共 19）
摘要Abstract
1 引言1 Introduction
2 设定2 Setting
2.1 贝尔曼方程2.1 Bellman’s Equations
3 分布贝尔曼算子3 The Distributional Bellman Operators
3.1 分布方程3.1 Distributional Equations
3.2 瓦瑟斯坦度量3.2 The Wasserstein Metric
3.3 策略评估3.3 Policy Evaluation
3.4 控制3.4 Control
4 近似分布学习4 Approximate Distributional Learning
4.1 参数化分布4.1 Parametric Distribution
4.2 投影贝尔曼更新4.2 Projected Bellman Update
5 在 Atari 2600 游戏上的评估5 Evaluation on Atari 2600 Games
5.1 改变原子数量5.1 Varying the Number of Atoms
5.2 最先进的结果5.2 State-of-the-Art Results
6 讨论6 Discussion
6.1 为什么学习分布很重要？6.1 Why does learning a distribution matter?