扩散策略：通过动作扩散进行视觉运动策略学习

Diffusion Policy: Visuomotor Policy Learning via Action Diffusion

迟程 Cheng Chi · Stanford · 2023-03-07 · arXiv:2303.04137 ↗ · 被引 3588

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

本文介绍了扩散策略，一种通过将机器人的视觉运动策略表示为条件去噪扩散过程来生成机器人行为的新方法。我们在来自 4 个不同机器人操作基准的 12 个不同任务上对扩散策略进行了基准测试，发现它始终优于现有的最先进机器人学习方法，平均改进幅度为 46.9%。扩散策略学习动作分布得分函数的梯度，并在推理过程中通过一系列随机朗之万动力学步骤相对于该梯度场进行迭代优化。我们发现，扩散公式在用于机器人策略时产生了强大的优势，包括优雅地处理多模态动作分布、适用于高维动作空间以及表现出令人印象深刻的训练稳定性。为了在物理机器人上充分释放扩散模型在视觉运动策略学习中的潜力，本文提出了一系列关键技术贡献，包括引入滚动时域控制、视觉条件以及时间序列扩散变换器。我们希望这项工作将有助于激发新一代能够利用扩散模型强大生成建模能力的策略学习技术。代码、数据和训练细节可在 diffusion-policy.cs.columbia.edu 公开获取。

This paper introduces Diffusion Policy, a new way of generating robot behavior by representing a robot's visuomotor policy as a conditional denoising diffusion process. We benchmark Diffusion Policy across 12 different tasks from 4 different robot manipulation benchmarks and find that it consistently outperforms existing state-of-the-art robot learning methods with an average improvement of 46.9%. Diffusion Policy learns the gradient of the action-distribution score function and iteratively optimizes with respect to this gradient field during inference via a series of stochastic Langevin dynamics steps. We find that the diffusion formulation yields powerful advantages when used for robot policies, including gracefully handling multimodal action distributions, being suitable for high-dimensional action spaces, and exhibiting impressive training stability. To fully unlock the potential of diffusion models for visuomotor policy learning on physical robots, this paper presents a set of key technical contributions including the incorporation of receding horizon control, visual conditioning, and the time-series diffusion transformer. We hope this work will help motivate a new generation of policy learning techniques that are able to leverage the powerful generative modeling capabilities of diffusion models. Code, data, and training details is publicly available diffusion-policy.cs.columbia.edu

核心贡献 · Key contributions

• 提出扩散策略，将视觉运动策略表示为条件去噪扩散过程。
  Proposes Diffusion Policy, representing visuomotor policy as a conditional denoising diffusion process.
• 在 4 个基准的 12 个任务上，平均比现有最优方法提升 46.9%。
  Achieves 46.9% average improvement over state-of-the-art methods across 12 tasks from 4 benchmarks.
• 引入通过滚动时域控制的闭环动作序列，实现鲁棒执行。
  Introduces closed-loop action sequences via receding horizon control for robust execution.
• 提出时间序列扩散 Transformer，减少过度平滑并处理高频动作。
  Presents time-series diffusion transformer to reduce over-smoothing and handle high-frequency actions.
• 通过建模得分函数避免难处理的归一化常数，实现稳定训练。
  Demonstrates stable training by modeling score function, avoiding intractable normalization constant.
• 在真实世界任务中验证，包括非刚体物体和周期性运动，达到接近人类水平。
  Validates on real-world tasks including non-rigid objects and periodic motions, achieving near-human performance.

局限 · Limitations

• 继承了行为克隆的局限性，例如演示数据不足时性能欠佳。
  Inherits behavior cloning limitations, e.g., suboptimal performance with inadequate demonstration data.
• 与 LSTM-GMM 等简单方法相比，计算成本和推理延迟更高。
  Higher computational cost and inference latency compared to simpler methods like LSTM-GMM.
• Transformer 主干对超参数敏感，需要额外调参。
  Transformer backbone is sensitive to hyperparameters, requiring additional tuning.
• 尽管有动作序列预测，可能仍不足以应对需要极高频率控制的任务。
  May not suffice for tasks requiring very high-rate control despite action sequence prediction.
• 性能依赖于演示的质量和数量、机器人物理能力以及架构。
  Performance depends on quality and quantity of demonstrations, physical robot capabilities, and architecture.

论文章节 · Sections（共 28）

• 摘要 Abstract
• 一、引言 I Introduction
• 二、扩散策略公式化 II Diffusion Policy Formulation
• 二-A 去噪扩散概率模型 II-A Denoising Diffusion Probabilistic Models
• 二-B DDPM 训练 II-B DDPM Training
• 二-C 扩散用于视觉运动策略学习 II-C Diffusion for Visuomotor Policy Learning
• 三、关键设计决策 III Key Design Decisions
• 三-A 网络架构选项 III-A Network Architecture Options
• 三-B 视觉编码器 III-B Visual Encoder
• 三-C 噪声调度 III-C Noise Schedule
• 三-D 加速推理实现实时控制 III-D Accelerating Inference for Real-time Control
• 四、扩散策略的有趣特性 IV Intriguing Properties of Diffusion Policy
• 四-A 建模多模态动作分布 IV-A Model Multi-Modal Action Distributions
• 四-B 与位置控制的协同 IV-B Synergy with Position Control
• 四-C 动作序列预测的优势 IV-C Benefits of Action-Sequence Prediction
• 四-D 训练稳定性 IV-D Training Stability
• 五、评估 V Evaluation
• 五-A 仿真环境与数据集 V-A Simulation Environments and datasets
• 五-B 评估方法 V-B Evaluation Methodology
• 五-C 主要发现 V-C Key Findings
• 六、真实世界评估 VI Realworld Evaluation
• 六-A 真实世界推 T 任务 VI-A Realworld Push-T Task
• 六-B 翻转杯子任务 VI-B Mug Flipping Task
• 六-C 倒酱与涂抹 VI-C Sauce Pouring and Spreading
• 七、相关工作 VII Related Work
• 八、局限性与未来工作 VIII Limitations and Future Work
• 九、结论 IX Conclusion
• 十、致谢 X Acknowledgement

阅读逐段中英对照全文 →