We present Universal Manipulation Interface (UMI) -- a data collection and policy learning framework that allows direct skill transfer from in-the-wild human demonstrations to deployable robot policies. UMI employs hand-held grippers coupled with careful interface design to enable portable, low-cost, and information-rich data collection for challenging bimanual and dynamic manipulation demonstrations. To facilitate deployable policy learning, UMI incorporates a carefully designed policy interface with inference-time latency matching and a relative-trajectory action representation. The resulting learned policies are hardware-agnostic and deployable across multiple robot platforms. Equipped with these features, UMI framework unlocks new robot manipulation capabilities, allowing zero-shot generalizable dynamic, bimanual, precise, and long-horizon behaviors, by only changing the training data for each task. We demonstrate UMI's versatility and efficacy with comprehensive real-world experiments, where policies learned via UMI zero-shot generalize to novel environments and objects when trained on diverse human demonstrations. UMI's hardware and software system is open-sourced at https://umi-gripper.github.io.
核心贡献 · Key contributions
提出通用操作接口（UMI），一种从野外人类演示直接迁移技能到可部署机器人策略的框架。 Proposes Universal Manipulation Interface (UMI), a framework enabling direct skill transfer from in-the-wild human demonstrations to deployable robot policies.
设计手持夹爪，配备鱼眼镜头、侧镜和 IMU 感知追踪，实现便携、低成本且信息丰富的数据采集。 Designs a hand-held gripper with fisheye lens, side mirrors, and IMU-aware tracking for portable, low-cost, and information-rich data collection.
引入推理时延迟匹配和相对轨迹动作表示，实现硬件无关的策略部署。 Introduces inference-time latency matching and relative-trajectory action representation for hardware-agnostic policy deployment.
在动态、双手、精确和长时域任务中展示了对新环境和物体的零样本泛化能力。 Demonstrates zero-shot generalization to novel environments and objects across dynamic, bimanual, precise, and long-horizon tasks.
使用多样化野外数据在分布外测试中达到 71.7%成功率，优于窄域基线。 Achieves 71.7% success rate in out-of-distribution tests with diverse in-the-wild data, outperforming narrow-domain baselines.
开源硬件和软件系统，实现可复现且易获取的机器人学习。 Open-sources hardware and software system, enabling reproducible and accessible robot learning.
局限 · Limitations
依赖运动学数据过滤确保可行性，限制了从硬件不可行动作的迁移。 Relies on kinematic data filtering to ensure feasibility, limiting transfer from hardware-infeasible actions.
基于 SLAM 的动作恢复需要环境中有足够纹理，在纹理缺失场景中失效。 SLAM-based action recovery requires sufficient texture in the environment, failing in texture-deficient scenes.
UMI 夹爪的数据采集效率低于人手演示，因其重量和自由度减少。 Data collection with UMI gripper is less efficient than human hand demonstration due to weight and reduced degrees of freedom.
策略性能依赖于仔细的延迟校准，该校准因硬件平台而异。 Policy performance depends on careful latency calibration, which varies across hardware platforms.
泛化到新物体和环境需要多样化的野外数据，采集成本可能较高。 Generalization to novel objects and environments requires diverse in-the-wild data, which may be costly to collect.
论文章节 · Sections（共 20）
摘要Abstract
一、引言I Introduction
二、相关工作II Related Works
二-A 遥操作机器人数据II-A Teleoperated Robot Data
二-B 人类视频视觉演示II-B Visual Demonstrations from Human Video
二-C 手持夹爪准静态动作II-C Hand-Held Grippers for Quasi-static Actions
三、方法III Method
三-A 演示界面设计III-A Demonstration Interface Design
三-B 策略界面设计III-B Policy Interface Design
四、评估IV Evaluations
五、能力实验V Capability Experiments
五-A 杯子排列V-A Cup Arrangement
五-B 动态抛掷V-B Dynamic Tossing
五-C 双臂折布V-C Bimanual Cloth Folding
五-D 洗碗V-D Dish Washing
六、野外泛化实验VI In-the-wild Generalization Experiments
七、数据采集吞吐量与精度VII Data Collection Throughput and Accuracy