We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.
核心贡献 · Key contributions
提出异步梯度下降用于深度强化学习，无需经验回放即可稳定训练。 Proposes asynchronous gradient descent for deep RL, enabling stable training without experience replay.
引入四种异步强化学习算法：单步 Q、Sarsa、n 步 Q 和优势行动者-评论家（A3C）。 Introduces four asynchronous RL algorithms: one-step Q, Sarsa, n-step Q, and advantage actor-critic (A3C).
A3C 在 Atari 上超越现有最优，训练时间减半且使用 CPU 而非 GPU。 A3C surpasses state-of-the-art on Atari with half training time on CPU vs GPU.
展示 A3C 在连续控制和基于像素的 3D 迷宫导航中的有效性。 Demonstrates A3C's effectiveness on continuous control and 3D maze navigation from pixels.
表明并行行动者-学习器稳定训练并实现多线程线性加速。 Shows parallel actor-learners stabilize training and enable linear speedup with multiple threads.
提供无需专用硬件的鲁棒且可扩展的深度强化学习框架。 Provides robust and scalable framework for deep RL without specialized hardware.
局限 · Limitations
实验限于模拟环境；未测试现实世界适用性。 Experiments limited to simulated environments; real-world applicability not tested.
A3C 需要仔细调整熵正则化和学习率以保持稳定性。 A3C requires careful tuning of entropy regularization and learning rate for stability.
异步更新可能导致梯度陈旧，可能减慢收敛。 Asynchronous updates may cause gradient staleness, potentially slowing convergence.
框架未纳入经验回放，而后者可提升数据效率。 Framework does not incorporate experience replay, which could improve data efficiency.
未探索扩展到超大规模分布式系统；仅测试单机 CPU。 Scalability to very large distributed systems not explored; only single-machine CPU tested.
论文章节 · Sections（共 13）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Work
3 强化学习背景3 Reinforcement Learning Background
4 异步强化学习框架4 Asynchronous RL Framework
5 实验5 Experiments
5.1 Atari 2600 游戏5.1 Atari 2600 Games
5.2 TORCS 赛车模拟器5.2 TORCS Car Racing Simulator
5.3 使用 MuJoCo 物理模拟器的连续动作控制5.3 Continuous Action Control Using the MuJoCo Physics Simulator