Policy search methods can allow robots to learn control policies for a wide range of tasks, but practical applications of policy search often require hand-engineered components for perception, state estimation, and low-level control. In this paper, we aim to answer the following question: does training the perception and control systems jointly end-to-end provide better performance than training each component separately? To this end, we develop a method that can be used to learn policies that map raw image observations directly to torques at the robot's motors. The policies are represented by deep convolutional neural networks (CNNs) with 92,000 parameters, and are trained using a partially observed guided policy search method, which transforms policy search into supervised learning, with supervision provided by a simple trajectory-centric reinforcement learning method. We evaluate our method on a range of real-world manipulation tasks that require close coordination between vision and control, such as screwing a cap onto a bottle, and present simulated comparisons to a range of prior policy search methods.
核心贡献 · Key contributions
提出一种引导式策略搜索方法，将策略搜索转化为监督学习，利用轨迹中心强化学习提供监督信号。 Develops a guided policy search method that transforms policy search into supervised learning using trajectory-centric RL.
引入一种新颖的 CNN 架构，包含空间软 argmax 层，实现从原始图像到力矩的端到端视觉运动策略学习。 Introduces a novel CNN architecture with spatial soft-argmax for end-to-end visuomotor policy learning from raw images to torques.
在真实世界操作任务上证明，感知与控制联合端到端训练优于分别训练。 Demonstrates end-to-end training of perception and control outperforms separate training on real-world manipulation tasks.
实现样本高效学习，仅需数十分钟真实交互即可完成复杂 7 自由度任务。 Achieves sample-efficient learning with only tens of minutes of real-world interaction time for complex 7-DoF tasks.
通过仿真对比表明，引导式策略搜索优于 REPS、RWR 和 PILCO 等先前方法。 Provides simulated comparisons showing guided policy search outperforms prior methods like REPS, RWR, and PILCO.
将引导式策略搜索形式化为 BADMM 算法，确保收敛到局部最优解。 Formalizes guided policy search as BADMM, ensuring convergence to a locally optimal solution.
局限 · Limitations
训练时需要完全状态可观测，限制了在非仪器化场景中的应用。 Requires full state observability during training, limiting applicability to instrumented setups.
假设低噪声动力学以进行线性高斯轨迹优化，在高度随机环境中可能不成立。 Assumes low-noise dynamics for linear-Gaussian trajectory optimization, which may not hold in highly stochastic environments.
对动作采用一阶矩约束，可能不足以处理需要高阶统计量的任务。 Uses first-moment constraints on actions, which may be insufficient for tasks requiring precise higher-order statistics.
预训练需要 ImageNet 初始化和位姿回归数据，增加了对外部数据集的依赖。 Pretraining requires ImageNet initialization and pose regression data, adding dependency on external datasets.
带有空间软 argmax 的策略架构专用于视觉运动任务，可能无法泛化到其他感知问题。 Policy architecture with spatial soft-argmax is specialized for visuomotor tasks and may not generalize to other perception problems.
论文章节 · Sections（共 31）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Work
3 背景与概述3 Background and Overview
3.1 定义与问题形式化3.1 Definitions and Problem Formulation
3.2 方法概述3.2 Approach Summary
4 基于 BADMM 的引导策略搜索4 Guided Policy Search with BADMM
4.1 算法推导4.1 Algorithm Derivation
4.2 未知动力学下的轨迹优化4.2 Trajectory Optimization under Unknown Dynamics
4.3 有监督策略优化4.3 Supervised Policy Optimization
4.4 与先前引导策略搜索方法的比较4.4 Comparison with Prior Guided Policy Search Methods
5 端到端视觉运动策略5 End-to-End Visuomotor Policies
5.1 视觉运动策略架构5.1 Visuomotor Policy Architecture
5.2 视觉运动策略训练5.2 Visuomotor Policy Training
6 实验评估6 Experimental Evaluation
6.1 与先前策略搜索方法的模拟比较6.1 Simulated Comparisons to Prior Policy Search Methods
6.2 在 PR2 机器人上学习线性高斯控制器6.2 Learning Linear-Gaussian Controllers on a PR2 Robot