We describe an iterative procedure for optimizing policies, with guaranteed monotonic improvement. By making several approximations to the theoretically-justified procedure, we develop a practical algorithm, called Trust Region Policy Optimization (TRPO). This algorithm is similar to natural policy gradient methods and is effective for optimizing large nonlinear policies such as neural networks. Our experiments demonstrate its robust performance on a wide variety of tasks: learning simulated robotic swimming, hopping, and walking gaits; and playing Atari games using images of the screen as input. Despite its approximations that deviate from the theory, TRPO tends to give monotonic improvement, with little tuning of hyperparameters.
核心贡献 · Key contributions
证明了使用 KL 散度惩罚的策略优化具有单调改进保证。 Proves a monotonic improvement guarantee for policy optimization with KL divergence penalty.
开发了 TRPO，一种使用 KL 散度约束进行稳健更新的实用算法。 Develops TRPO, a practical algorithm using a KL divergence constraint for robust updates.
展示了 TRPO 在模拟机器人运动和 Atari 游戏上的有效性。 Demonstrates TRPO's effectiveness on simulated robotic locomotion and Atari games.
在信任区域框架下统一了策略梯度和策略迭代方法。 Unifies policy gradient and policy iteration methods under a trust region framework.
在具有神经网络策略的高维控制任务上实现了最先进的性能。 Achieves state-of-the-art performance on high-dimensional control tasks with neural network policies.
局限 · Limitations
假设优势函数精确评估，这在实践中不现实。 Assumes exact advantage evaluation, which is unrealistic in practice.
Vine 方法需要状态重置，限制了其在模拟环境中的应用。 Vine method requires state resets, limiting applicability to simulation.
理论界限使用最大 KL 散度，但实际算法使用平均 KL 散度。 Theoretical bound uses maximum KL divergence, but practical algorithm uses average KL.
由于在策略采样，样本复杂度可能对真实机器人任务较高。 Sample complexity may be high for real-world robotic tasks due to on-policy sampling.
在 Atari 游戏上的性能并不始终优于 DQN 等专门方法。 Performance on Atari games is not consistently superior to specialized methods like DQN.
论文章节 · Sections（共 15）
摘要Abstract
1 引言1 Introduction
2 预备知识2 Preliminaries
3 一般随机策略的单调改进保证3 Monotonic Improvement Guarantee for General Stochastic Policies
4 参数化策略的优化4 Optimization of Parameterized Policies
5 目标和约束的基于样本的估计5 Sample-Based Estimation of the Objective and Constraint