We adapt the ideas underlying the success of Deep Q-Learning to the continuous action domain. We present an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. Using the same learning algorithm, network architecture and hyper-parameters, our algorithm robustly solves more than 20 simulated physics tasks, including classic problems such as cartpole swing-up, dexterous manipulation, legged locomotion and car driving. Our algorithm is able to find policies whose performance is competitive with those found by a planning algorithm with full access to the dynamics of the domain and its derivatives. We further demonstrate that for many of the tasks the algorithm can learn policies end-to-end: directly from raw pixel inputs.
核心贡献 · Key contributions
将深度 Q 学习扩展到连续动作空间，基于确定性策略梯度。 Adapts Deep Q-Learning to continuous action domains via deterministic policy gradient.
提出深度确定性策略梯度（DDPG）算法，结合演员-评论家、经验回放和目标网络。 Proposes Deep DPG (DDPG) algorithm with actor-critic, replay buffer, and target networks.
使用相同超参数和网络架构解决 20 多个模拟物理任务。 Solves over 20 simulated physics tasks with same hyper-parameters and network architecture.
性能与完全访问动力学的规划算法相当。 Achieves competitive performance with planning algorithms that have full dynamics access.
展示从原始像素输入进行端到端学习的能力。 Demonstrates end-to-end learning from raw pixel inputs for many tasks.
结合批归一化，在不同单位任务间实现稳定学习。 Combines batch normalization for stable learning across tasks with different units.
局限 · Limitations
需要大量训练回合，样本效率受限。 Requires large number of training episodes, limiting sample efficiency.
非线性函数近似器无法保证收敛。 No convergence guarantees due to non-linear function approximators.
探索噪声需针对每个环境调整以实现有效探索。 Exploration noise must be tuned per environment for effective exploration.
在困难任务中 Q 值估计不准确，性能下降。 Performance degrades on harder tasks with inaccurate Q-value estimates.
仅限于模拟环境，未测试真实世界适用性。 Limited to simulated environments; real-world applicability not tested.