We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.
核心贡献 · Key contributions
提出使用裁剪替代目标的 PPO，允许多次小批量更新。 Proposes PPO with clipped surrogate objective enabling multiple epochs of minibatch updates.
PPO 具有类似 TRPO 的稳定性和可靠性，但更简单、更通用。 PPO achieves stability and reliability similar to TRPO but is simpler and more general.
在连续控制和 Atari 基准上优于其他在线策略梯度方法。 Outperforms other online policy gradient methods on continuous control and Atari benchmarks.
相比先前方法，具有更好的样本复杂度和实际时间效率。 Provides better sample complexity and wall-time efficiency compared to prior methods.
支持参数共享和噪声结构的架构，与 TRPO 不同。 Supports architectures with parameter sharing and noise, unlike TRPO.
局限 · Limitations
裁剪超参数ε需要针对不同任务进行调优。 Clipping hyperparameter ε requires tuning for different tasks.
尽管有裁剪，PPO 在某些环境中仍可能不稳定。 PPO may still suffer from instability in some environments despite clipping.
在实践中没有单调改进的理论保证。 No theoretical guarantee of monotonic improvement in practice.
对于动作集非常大的离散动作空间不太适用。 Less suitable for discrete action spaces with very large action sets.
性能严重依赖于优势估计和基线设计。 Performance depends heavily on advantage estimation and baseline design.