ALOHA Unleashed: A Simple Recipe for Robot Dexterity
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→近期研究在利用模仿学习训练端到端机器人策略方面取得了有前景的成果。本文探讨了在具有挑战性的灵巧操作任务中，模仿学习能推进到何种程度。我们表明，在 ALOHA 2 平台上进行大规模数据收集，结合扩散策略等表达力强的模型，这一简单配方能有效学习涉及可变形物体和复杂接触动力学的挑战性双手操作任务。我们在 5 个真实世界任务和 3 个模拟任务上验证了该配方，并展示了相较于最先进基线的性能提升。项目网站和视频可在 aloha-unleashed.github.io 查看。
Recent work has shown promising results for learning end-to-end robot policies using imitation learning. In this work we address the question of how far can we push imitation learning for challenging dexterous manipulation tasks. We show that a simple recipe of large scale data collection on the ALOHA 2 platform, combined with expressive models such as Diffusion Policies, can be effective in learning challenging bimanual manipulation tasks involving deformable objects and complex contact rich dynamics. We demonstrate our recipe on 5 challenging real-world and 3 simulated tasks and demonstrate improved performance over state-of-the-art baselines. The project website and videos can be found at aloha-unleashed.github.io.
近期工作表明，使用模仿学习来学习端到端机器人策略已取得令人鼓舞的成果。在本工作中，我们探讨了在具有挑战性的灵巧操作任务中，模仿学习能推进到何种程度。我们表明，在 ALOHA 2 平台上进行大规模数据收集，并结合诸如扩散策略等表达能力强的模型，这一简单配方能有效学习涉及可变形物体和复杂接触丰富动力学的挑战性双手操作任务。我们在 5 个具有挑战性的真实世界任务和 3 个模拟任务上验证了我们的配方，并展示了相对于最先进基线的性能提升。项目网站和视频可在 aloha-unleashed.github.io 上找到。
Recent work has shown promising results for learning end-to-end robot policies using imitation learning. In this work, we address the question of how far we can push imitation learning for challenging dexterous manipulation tasks. We show that a simple recipe of large-scale data collection on the ALOHA 2 platform, combined with expressive models such as Diffusion Policies, can be effective in learning challenging bimanual manipulation tasks involving deformable objects and complex contact-rich dynamics. We demonstrate our recipe on 5 challenging real-world and 3 simulated tasks and demonstrate improved performance over state-of-the-art baselines. The project website and videos can be found at aloha-unleashed.github.io.
关键词：模仿学习，操作
Keywords: Imitation Learning, Manipulation
诸如系鞋带或将 T 恤挂在衣架上等灵巧操作任务，传统上被认为对机器人来说非常困难。从建模的角度来看，这些任务具有挑战性，因为它们涉及具有复杂接触动力学的（可变形）物体，需要许多操作步骤才能解决任务，并且涉及高维机器人操纵器的协调，尤其是在双臂设置中，而且通常需要高精度。近年来，模仿学习已被确立为获得能够解决各种任务的策略的秘诀。然而，这些成功案例大多主要涉及非灵巧任务，如抓取放置或推挤。因此，目前尚不清楚简单地扩展模仿学习是否足以应对灵巧操作，因为为此类任务收集覆盖系统状态变化且具有所需精度的数据集似乎难以实现。
Dexterous manipulation tasks such as tying shoe laces or hanging t-shirts on a coat hanger have traditionally been seen as very difficult to achieve with robots. From a modeling perspective, these tasks are challenging since they involve (deformable) objects with complex contact dynamics, require many manipulation steps to solve the task, and involve the coordination of high-dimensional robotic manipulators, especially in bimanual setups, and generally often demand high precision. In recent years, imitation learning has been established as a recipe for obtaining policies that can solve a wide variety of tasks. However, most of these success stories predominantly concern non-dexterous tasks such as pick and place, or pushing. Therefore, it is unclear if simply scaling up imitation learning is sufficient for dexterous manipulation, since collecting a dataset that covers the state variation of the system with the required precision for such tasks seems prohibitive.
在本文中，我们证明通过选择合适的学习架构并结合适当的数据收集策略，可以推动模仿学习在灵巧操作领域的前沿。我们在 ALOHA 2 平台上展示了我们能够获得能够解决高度灵巧、长时域、双臂操作任务的策略，这些任务涉及可变形物体并需要高精度。为实现这一目标，我们开发了一种数据收集协议，其规模在双臂操作平台上前所未有，在真实机器人上为 5 个任务收集了超过 26,000 个演示，并在 3 个模拟任务上收集了超过 2,000 个演示。
In this paper, we demonstrate that by choosing the appropriate learning architecture combined with a suitable data collection strategy, it is possible to push the frontier of dexterous manipulation with imitation learning. We show on the ALOHA 2 platform that we can obtain policies that are capable of solving highly dexterous, long-horizon, bimanual manipulation tasks that involve deformable objects and require high precision. To achieve this, we develop a protocol to collect data on a scale previously unmatched by any bimanual manipulation platform, with over 26,000 demonstrations for 5 tasks on a real robot, and over 2,000 demonstrations on 3 simulated tasks.
然而，我们发现仅靠数据是不够的。我们方法的另一个关键要素是基于 Transformer 的学习架构，并使用扩散损失进行训练。该架构以多视图为条件，对动作轨迹进行去噪，并在后退时域设置中以开环方式执行。结果表明，尽管之前针对 ALOHA 平台进行了调整，但非基于扩散的架构无法解决我们的一些任务。
However, we find that the data alone is insufficient. The other key ingredient in our approach is a transformer-based learning architecture trained with a diffusion loss. Conditioned on multiple views, this architecture denoises a trajectory of actions, which is executed open-loop in a receding horizon setting. Results show that non-diffusion based architectures are incapable of solving some of our tasks, despite being previously tuned for the ALOHA platform.
我们的实验评估涉及 5 个真实世界任务，如系鞋带和将衣服挂在衣架上，以及 3 个模拟任务。我们研究了策略的数据复杂性和分布外鲁棒性。据我们所知，我们是第一个展示能够自主系鞋带或挂 T 恤的端到端策略的团队。
Our experimental evaluation involves 5 real world tasks such as tying shoe laces and hanging clothes on a hanger as well as 3 simulated tasks. We investigate the data complexity and out-of-distribution robustness of our policies. To the best of our knowledge, we are the first to demonstrate an end-to-end policy that can tie shoelaces or hang t-shirts autonomously.
**模仿学习**。模仿学习使机器人能够从专家示范中学习。早期工作通过运动原语的视角来解决这个问题。
Imitation learning. Imitation learning enables robots to learn from expert demonstrations. Early works tackle this problem through the lens of motor primitives.
随着深度学习和生成建模的发展，人们提出了不同的架构和训练目标来端到端地建模示范。这包括使用 ConvNets 或 ViT 进行图像处理，使用 RNN 或 Transformer 融合历史观测，对动作空间进行分词，以及生成建模技术如基于能量的模型、扩散模型和 VAE。在本工作中，我们追求算法的简洁性，建立在现有模仿学习算法的基础上。具体来说，我们训练一个基于 Transformer 的策略，使用扩散损失，受 Diffusion Policy 和 ACT 启发。与之前的工作不同，我们在来自非研究人员数据收集者的大量数据上进行训练，他们使用 ALOHA 2 执行既精确又多模态的任务。
With the development of deep learning and generative modeling, different architectures and training objectives are proposed to model the demonstrations end-to-end. This includes the use of ConvNets or ViT for image processing, RNN or transformers for fusing history observations, tokenization of the action space, generative modeling techniques such as energy-based models, diffusion and VAEs. In this work, we push for simplicity of the algorithm, building upon existing imitation learning algorithms. Specifically, we train a transformer-based policy with diffusion loss, inspired by Diffusion Policy and ACT. While unlike previous works, we train on large amounts of data from non-researcher data collectors, who use ALOHA 2 to perform tasks that are both precise and multi-modal.
**双手操作**。双手操作在机器人学中有着悠久的历史。早期工作从优化角度处理双手操作，假设已知环境动力学。然而，获取这样的环境动力学模型可能非常耗时，尤其是那些捕捉丰富接触或可变形物体的模型。最近，学习被纳入双手系统，包括强化学习、模仿学习或学习调节低级运动原语的关键点。之前的工作也研究了高度灵巧的双手操作任务，如解绳结、展平布料，甚至穿针引线。然而，所使用的机器人（如 Intuitive 的手术机器人）远不那么容易获得。在本工作中，我们使用一组低成本的 ALOHA 2 系统来研究仅扩大数据收集规模本身如何能带来机器人灵巧性的显著进步，而无需超精密的运动学和精细的传感。
Bimanual manipulation. Bimanual manipulation has a long history in robotics. Early works tackle bimanual manipulation from an optimization perspective, with known environment dynamics. However, obtaining such environment dynamics models can be time-consuming, especially those that capture rich contact or deformable objects. More recently, learning has been incorporated into bimanual systems, including reinforcement learning, imitating learning, or learning of key points that modulate low-level motor primitives. Previous works have also studied highly dexterous bimanual manipulation tasks, such as knot untying, cloth flattening, or even threading a needle. However, the robots used are much less accessible such as surgical robots from Intuitive. In this work, we use a fleet of low-cost ALOHA 2 systems to study how scaling up data collection itself can already bring significant advancements in robot dexterity, without ultra-precise kinematics and elaborate sensing.
**在现实世界中扩大机器人学习**。许多工作尝试使用真实世界数据收集来扩大机器人学习。遥操作是一种收集高质量数据的方式，由人在回路中控制机器人。之前的工作使用 VR 控制器或触觉设备在单臂机器人上收集了大型数据集，展示了对新场景和物体的泛化能力。或者，机器人也可以由强化学习（RL）算法编程或控制，自主收集数据，减少对人工监督的需求。另一种收集专家数据的方法是使用可穿戴或手持设备，如夹爪、外骨骼或跟踪手套。这允许在没有完整机器人的情况下扩大数据收集。还有正在进行的努力将上述所有数据集结合起来，训练一个可以控制多个机器人的单一模型。在本工作中，我们专注于扩大机器人学习的灵巧性方面，同时稳健地处理可变形和铰接物体。据我们所知，我们训练了第一个能够自主系鞋带和挂 T 恤的端到端策略。
Scale up robot learning in the real world. Many works have attempted to scale up robot learning using real-world data collection. Teleoperation is a one way to collect high-quality data, with a human in the loop controlling the robot. Previous works have collected large datasets on single-arm robots using VR controller or haptic device, demonstrating generalization to novel scenes and objects. Alternatively, the robot can also be programmed or controlled by Reinforcement Learning (RL) algorithms to collect data autonomously, reducing the need of human supervision. Another way to collect expert data is to use wearable or hand-held devices, such as grippers, exoskeleton, or tracking gloves. This allows scaling up of data collection without requiring full robots. There are also ongoing efforts to combine all the aforementioned datasets, to train a single model that can control multiple robots. In this work, we focus on scaling up the dexterity aspect of robot learning, together with robust handling of deformable and articulated objects. To the best of our knowledge, we trained the first end-to-end policy that can autonomously tie shoelaces and hang t-shirts.
我们提出了 ALOHA Unleashed，一个用于在机器人上训练灵巧策略的通用模仿学习系统。我们在 ALOHA 2 上展示了结果，该系统由一个双臂平行夹爪工作单元组成，配备两个 6 自由度手臂。ALOHA Unleashed 包含一个可扩展遥操作框架，允许用户收集数据来教机器人，并结合了一个基于 Transformer 的神经网络，该网络使用受启发于 和 的扩散策略进行训练，为模仿学习提供了一种表达力强的策略形式。凭借这一简单配方，我们在 5 个具有挑战性的现实世界任务上展示了自主策略：挂衬衫、系鞋带、更换机器人手指、插入齿轮以及堆叠随机初始化的厨房物品。我们还在 3 个模拟双臂任务上展示了结果：单钉插入、双钉插入以及将杯子放在盘子上。
We introduce ALOHA Unleashed, a general imitation learning system for training dexterous policies on robots. We demonstrate results on ALOHA 2, which consists of a bimanual parallel-jaw gripper workcell with two 6-DoF arms. ALOHA Unleashed consists of a framework for scalable teleoperation that allows users to collect data to teach robots, combined with a Transformer-based neural network trained with Diffusion Policy inspired by and , which provides an expressive policy formulation for imitation learning. With this simple recipe, we demonstrate autonomous policies on 5 challenging real world tasks: hanging a shirt, tying shoe laces, replacing a robot finger, inserting gears, and stacking randomly initialized kitchen items. We also show results on 3 simulated bimanual tasks: single peg insertion, double peg insertion, and placing a mug on a plate.
扩散策略。我们使用的数据集具有固有的多样性，因为数据由多个操作员收集，包含多种遥操作策略，并在多个机器人工作单元上长时间采集。这需要一种表达能力强的策略形式来拟合数据。我们为每个任务训练一个独立的扩散策略。扩散策略提供稳定的训练，并能表达多模态动作分布，输入为多模态（来自不同视角的 4 张图像和本体感觉状态），动作空间为 14 自由度。我们采用去噪扩散隐式模型（DDIM）形式，在测试时允许灵活使用可变数量的推理步骤。我们进行动作分块，使策略能够预测 50 个动作的块，代表跨越 1 秒的轨迹。策略输出 12 个绝对关节位置（每个 6 自由度 ViperX 机械臂 6 个），以及两个夹爪各自的连续夹爪位置值。由于我们使用长度为 50 的动作块，策略输出形状为(50, 14)的张量。训练时我们使用 50 个扩散步骤，并采用平方余弦噪声调度（来自）。
Diffusion Policy. The dataset we use has inherent diversity, given that data is collected from multiple operators, contains a variety of teleoperation strategies, and is collected over a long period of time on multiple robotic workcells. This requires an expressive policy formulation to fit the data. We train a separate Diffusion Policy for each task. Diffusion Policy provides stable training and expresses multimodal action distributions with multimodal inputs (4 images from different viewpoints and proprioceptive state) and 14-degree-of-freedom action space. We use the Denoising diffusion implicit models (DDIM) formulation, which allows flexibility at test time to use a variable number of inference steps. We perform action chunking to allow the policy to predict chunks of 50 actions, representing a trajectory spanning 1 second. The policy outputs 12 absolute joint positions, 6 for each 6-dof ViperX arm, and a continuous value for gripper position for each of the two grippers. Since we use action chunks of length 50, the policy outputs a tensor of shape (50, 14). We use 50 diffusion steps during training, with a squared cosine noise schedule from .
基于 Transformer 的架构。对于我们的基础模型，我们扩展了类似于中使用的 Transformer 编码器-解码器架构。我们使用基于 ResNet50 的视觉骨干网络，并采用 Transformer 编码器-解码器架构作为神经网络策略。4 张 RGB 图像均调整为 480x640x3，并分别输入到独立的 ResNet50 骨干网络。每个 ResNet50 从 ImageNet 预训练分类模型初始化。我们取 ResNet 的第 4 阶段输出，得到每张图像 15x20x512 的特征图。特征图被展平，产生 1200 个 512 维嵌入。我们附加另一个嵌入，即机器人本体感觉状态的投影，包括每个机械臂的关节位置和夹爪值，总共 1201 个潜在特征维度。我们向嵌入添加位置嵌入，并将其输入到 85M 参数的 Transformer 编码器中进行编码，采用双向注意力，产生观测的潜在嵌入。潜在嵌入被传递到扩散去噪器，该去噪器是一个 55M 参数的 Transformer，具有双向注意力。解码器 Transformer 的输入是一个 50x14 的张量，对应于带有学习位置嵌入的噪声动作块。这些嵌入与观测编码器的潜在嵌入以及扩散时间步（表示为 one-hot 向量）进行交叉注意力。Transformer 解码器的输出维度为 50x512，通过线性层投影为 50x14，对应于块中接下来 50 个动作的预测噪声。总体而言，基础模型包含 217M 可学习参数。对于消融实验，我们还训练了模型的小型变体，使用 17M 参数的 Transformer 编码器和 37M 参数的 Transformer 解码器，总网络大小为 150M 参数。
Transformer-based architecture. For our base model, we scale up an architecture similar to the Transformer Encoder-Decoder architecture used in . We use a ResNet50 based vision backbone, with a Transformer Encoder-Decoder architecture as the neural network policy. Each of the 4 RGB images is resized to 480x640x3 and fed into a separate ResNet50 backbone. Each ResNet50 is initialized from an ImageNet pretrained classification model. We take the stage 4 output of the ResNets, which gives a 15 x 20 x 512 feature map for each image. The feature map is flattened, resulting in 1200 512-dimensional embeddings. We append another embedding, which is a projection of the proprioceptive state of the robot, which consists of the joint positions and gripper values for each of the arms, for a total of 1201 latent feature dimensions. We add positional embeddings to the embedding and feed it into a 85M parameter Transformer encoder to encode the embeddings, with bidirectional attention, producing latent embeddings of the observations. The latents are passed into the diffusion denoiser, which is a 55M parameter transformer with bidirectional attention. The input of the decoder transformer is a 50 x 14 tensor, corresponding to a noised action chunk with a learned positional embedding. These embeddings cross-attend to the latent embeddings from the observation encoder, as well as the diffusion timestep, which is represented as a one-hot vector. The transformer decoder has an output dimension of 50 x 512, which is projected with a linear layer into 50 x 14, corresponding to the predicted noise for the next 50 actions in the chunk. In total, the Base model consists of 217M learnable parameters. For ablation experiments, we also train a Small variant of our model, which uses a 17M parameter Transformer encoder and 37M parameter Transformer decoder, with a total network size of 150M parameters.
训练细节。我们使用 JAX 在 64 个 TPUv5e 芯片上以数据并行网格训练模型。我们使用批量大小 256，训练 2M 步（约 265 小时）。我们使用 Adam 优化器，权重衰减为 0.001，线性学习率预热 5000 步，然后保持恒定速率 1e-4。
Training details. We train our models with JAX using 64 TPUv5e chips with a data parallel mesh. We use a batch size of 256 and train for 2M steps (about 265 hours of training). We use the Adam optimizer with weight decay of 0.001 and a linear learning rate warmup for 5000 steps followed by a constant rate of 1e-4.
测试时推理。在测试时，我们首先从高斯分布中采样一个噪声动作块。我们收集来自 4 个 RGB 相机的最新观测和机器人的本体感觉状态，并将其通过观测编码器。然后我们运行扩散去噪循环 50 次，输出去噪后的动作块。我们发现不需要中的时间集成，直接开环执行块中的 50 个动作。网络的一次完整前向传播和迭代去噪过程在 RTX 4090 GPU 上耗时 0.043 秒。由于我们开环运行动作块，我们能够超过 50Hz 的目标频率。
Test time inference. At test time, we first sample a noised action chunk from a gaussian distribution. We gather the latest observations from the 4 RGB cameras and the proprioceptive state of the robot, and pass these through the observation encoder. We then run the diffusion denoising loop 50 times, outputting a denoised action chunk. We find that we do not need the temporal ensembling from , and simply execute the 50 actions in the chunk open loop. A full forward pass through the network and iterative denoising process takes 0.043 seconds on a RTX 4090 GPU. Since we run the action chunk open loop, we are able to surpass our target frequency of 50Hz.
ALOHA 通过木偶式接口实现双臂遥操作，允许人类操作员反向驱动两个较小的领导臂，其关节与两个较大的从动臂同步。我们在以下 5 个任务上收集数据：
ALOHA allows bimanual teleoperation via a puppeteering interface, which allows a human teleoperator to backdrive two smaller leader arms, whose joints are synchronized with two larger follower arms. We collect data on the following 5 tasks:
挂衬衫（Shirt）：该任务要求将衬衫挂在衣架上。详细步骤包括：将衬衫展平，从架子上取下衣架，进行交接，拿起衬衫，将衣架两侧精确插入衬衫领口，然后将衬衫挂回架子上。这是一个具有挑战性的任务，需要多个步骤，涉及可变形操作、插入以及灵巧的抓取和放置行为，如将衣架从架子上钩上和取下。我们构建了该任务的两种变体：ShirtEasy 具有更受约束的初始化，衬衫展平并居中放在桌子上；ShirtMessy 允许衬衫初始化为旋转和皱褶状态，且起始位置的变化显著更大。
Shirt hanging (Shirt): This task requires hanging a shirt on a hanger. The detailed steps include flattening the shirt, picking a hanger off a rack, performing a handover, picking up the shirt, precisely inserting both sides of the hanger into the shirt collar, then hanging the shirt back on the rack. This is a challenging task that requires multiple steps with deformable manipulation, insertions, and dexterous pick and placing behaviors like hooking and unhooking the hanger from the rack. We construct two variants of this task: ShirtEasy has a more constrained initialization, with the shirt flattened and centered on the table. ShirtMessy allows the initialization of the shirt to be rotated and crumpled and has significantly more variance in starting location.
系鞋带（Lace）：该任务要求将鞋子放在桌子中央，拉直鞋带，然后执行操作将鞋带打成蝴蝶结。我们构建了该任务的两种变体：LaceEasy 具有受约束的初始化，鞋子居中且鞋带向外伸展；LaceMessy 允许鞋子角度有 ±45 度的变化，且不要求鞋带展平。
Shoelace tying (Lace): This task requires centering a shoe on the table, straightening the laces, then performing a maneuver to tie the laces in a bow. We construct two variants of this task: LaceEasy has a constrained initialization with the shoe centered on the table and laces extended outward. LaceMessy allows \(\pm\) 45 degree variance in the angle of the shoe and does not require the laces to be flattened.
更换机器人手指（FingerReplace）：该任务要求从槽式机构中取出机器人手指，拿起替换手指，重新调整手指方向，然后以毫米级公差精确插入回槽中。
Robot finger replacement (FingerReplace): This task requires removing a robot finger from a slotted mechanism, picking the replacement finger, reorienting the finger, then performing a precise insertion back into the slot with millimeter tolerance.
齿轮插入（GearInsert）：该任务要求将 3 个塑料齿轮以毫米级精度和过盈配合插入插座，同时确保齿轮完全就位且齿轮齿与相邻齿轮啮合。
Gear insertion (GearInsert): This task requires inserting 3 plastic gears onto a socket with millimeter precision with a friction fit, while ensuring that the gear is fully seated and the teeth on the gear mesh with neighboring gears.
随机厨房堆叠（RandomKitchen）：该任务要求通过堆叠碗、杯子和餐具，并将堆叠物放置在桌子中央，来清理随机初始化的桌子。
Random kitchen stack (RandomKitchen): This task requires cleaning up a randomly initialized table by stacking bowls, cups, and utensils and placing the stack at the center of the table.
为了在这些任务上扩展数据收集，我们制定了一套协议，允许非专家用户提供高质量的远程操作演示。协议文档（见附录 B.1）概述了如何使用机器人的说明，以及所执行任务的具体说明。这使得由 35 名操作员组成的团队能够在没有研究人员监督的情况下持续收集数据。利用该协议，我们在 8 个月的时间里，在 2 栋不同建筑中的 10 台不同机器人上，为 5 个真实任务收集了超过 26k 个回合。在多个机器人工作单元上长时间收集数据面临许多挑战。由于装配错误或一般性差异，机器人在硬件组装上可能存在差异，例如机器人或摄像头的安装位置。此外，硬件更换或机器人的一般磨损可能会改变机器人的动力学和行为。建筑间的差异和机器人放置位置的不同导致了 RGB 图像中背景和光照的多样性。即使每个任务都有详细的协议文档，从 35 名不同操作员那里收集数据也会导致行为上的巨大差异。
To scale data collection on these tasks, we create a protocol that allows non-expert users to provide high quality teleoperated demonstrations. Protocol documents (See Appendix B.1) outline instructions for both how to use the robots, and specific instructions for the task being performed. This allows continuous data collection by a pool of 35 operators without oversight by researchers. Using this protocol, we collect over 26k episodes for 5 real tasks, on 10 different robots in 2 different buildings over the course of 8 months. Data collection over a long period of time on multiple robotic workcells presents many challenges. Robots may have differences in hardware assembly such as mounting positions for the robots or cameras, due to either assembly mistakes or general variance. In addition, hardware changes or general wear and tear on robots may change robot dynamics and behavior. Changes across buildings and differences in robot placement contribute to diversity of backgrounds and lighting in RGB images. Collecting data from 35 different operators results in a large amount of variance in behaviors, even with detailed protocol documentation for each task.
对于核心模型的主要结果，我们在每个任务上进行 20 次试验，使用分别在 5 个数据集（Shirt、Lace、FingerReplace、GearInsert 和 RandomKitchen）上训练的模型。一个回合要么在成功时终止，要么在超时（ShirtMessy 为 120 秒，其他任务为 80 秒）时终止。对于 GearInsert，我们报告了任务进度的更详细分解，其中 GearInsert-1 表示至少成功插入 1 个齿轮，GearInsert-2 表示成功插入 2 个齿轮，GearInsert-3 表示成功插入全部 3 个齿轮。对于 KitchenStack，我们也根据任务进度报告了详细的分解。对于 GearInsert 和 KitchenStack，我们看到性能随着每个额外阶段的增加而下降，这通常是由于需要更细粒度的行为来插入较小的齿轮或拾取像叉子这样的薄物体。对于所有其他任务，只有当策略执行了所有必需步骤时，我们才标记为成功，不允许部分成功。
For our main results of our core models, we perform 20 trials on each task using models separately trained on 5 datasets (Shirt, Lace, FingerReplace, GearInsert, and RandomKitchen). An episode terminates either on success or a timeout (120 seconds for ShirtMessy and 80 seconds for other tasks). For GearInsert, we report a more detailed breakdown of task progress, where GearInsert-1 represents successful insertion of at least 1 gear, GearInsert-2 is successful insertion of 2 gears, and GearInsert-3 is successfully inserting all 3 gears. For KitchenStack, we also report a detailed breakdown based on task progress. For GearInsert and KitchenStack, we see that performance decreases with each additional stage, usually due to more fine grained behaviors needed to insert smaller gears or pick thin objects like forks. For all other tasks, we mark a success only if the policy performs all required steps, with no partial success.
在本节中，我们重点介绍策略能够从数据中学习到的灵巧行为。
In this section we highlight dexterous behaviors that the policy is able to learn from the data.
在为这些任务收集数据时，操作员会执行许多双手行为原语，例如用于重新定向的交接，以及使用腕部相机进行视角增强。例如，FingerReplace 任务要求从桌上拿起手指后重新定向，以对齐手指的插入方向。我们观察到，策略能够从机器人手指的多个起始位置学习到许多连贯的重新定向行为。尽管我们发现重新定向相当稳健，但对于数据集中未充分表示的起始位置（例如手指被翻转倒置）会出现失败，这表明可能需要显式收集更多样化的重新定向示例。在 FingerReplace 中，策略还学习了数据中存在的双手视角增强策略，即操作员使用未使用手臂的腕部相机提供增强的 RGB 输入，以便策略更好地执行精确插入，而该插入在其他视角下不太可见。
While collecting data for these tasks, operators perform many bimanual behavior primitives, like handovers for reorientation, and view augmentation with wrist cameras. For example, FingerReplace requires reorienting the finger after picking it off the table to align the finger's direction for insertion. We see that the policy learns many coherent reorientation behaviors from multiple starting positions of the robot finger. Though we find that reorienting is fairly robust, we see failures for starting positions that aren't well represented in the dataset, such as the finger being flipped upside down, suggesting that it may be necessary to explicitly collect more diverse examples of reorientation. In FingerReplace, the policy also learns a bimanual view augmentation strategy that is present in the data, where operators use the wrist camera from the unused arm to provide augmented RGB input for the policy to better perform the precise insertion, which is less visible from other views.
在所有任务中，我们都看到了许多恢复行为和重试的实例。例如，在衬衫任务中，我们看到衬衫从衣架上掉落的实例，策略会恢复并将衬衫重新放回衣架。我们还看到插入过程中的重试行为实例，例如在 GearInsert 和 FingerReplace 中，策略会重新定向并从失败的插入中恢复。
We see many instances of recovery behaviors and retries in all tasks. For example, on the shirt tasks we see instances of the shirt falling off the hanger, and the policy recovering and replacing the shirt on the hanger. We also see instances of retry behavior during insertions such as in GearInsert and FingerReplace, where the policy reorients and recovers from failed inserts.
策略执行相对夹爪控制以完成所有任务中涉及的精确拾取行为。这在 RandomKitchen 中尤为明显，该任务要求从各种初始状态从桌上拾取薄物体。我们 ALOHA 2 机器人集群中的机器人未校准，且机器人安装位置和相机安装位置可能存在差异。我们推测，尽管策略接收 RGB 和完整本体感觉状态，但策略可能正在学习从视觉反馈中执行反应式相对夹爪控制，以跨机器人泛化。
The policies perform relative gripper control to accomplish precise picking behavior involved in all tasks. This is especially apparent in RandomKitchen, which requires picking thin objects from the table from a wide variety of initial states. Robots in our ALOHA 2 fleet are uncalibrated and may have differences in robot and camera mounting positions. We speculate that though the policy receives RGB and full proprioceptive state, policies may be learning to perform reactive relative gripper control from visual feedback to generalize across robots.
在多个长时程任务中，我们观察到模式切换行为。例如，在 ShirtMessy 中，策略从在桌上抚平衬衫切换到开始伸手够衣架。类似地，在 LaceMessy 中，策略从拉直鞋子切换到剧情的系环阶段。
On several long horizon tasks, we observe mode switching behaviors. For example, on ShirtMessy, the policy changes from flattening the shirt on the table to beginning the reach for the hanger. Similarly on LaceMessy, the policy switches from straightening the shoe to the loop-tying phase of the episode.
GearInsert 和 FingerReplace 都需要毫米级精度的插入操作。尤其是 GearInsert，需要紧密的摩擦配合，才能将齿轮正确对齐并完全插入轴中。令人惊讶的是，尽管机械臂精度较低且缺乏力-力矩反馈，我们的策略仅凭视觉反馈就能完成这些任务。
GearInsert and FingerReplace both require millimeter-accuracy precision insertions. GearInsert, in particular, requires a tight friction fit to properly align and insert the gear all the way into the shaft. We find it surprising that despite low-precision robotic arms and lack of force-torque feedback, our policies are able to perform these tasks with only visual feedback.
我们进行了多项实验，以确定演示数据的数量和质量的重要性。以下所有实验均使用模型的小型 1.5 亿参数变体运行。
We perform several experiments to determine the importance of quantity and quality of demonstration data. All experiments below are run with the Small 150M parameter variant of the model.
数据数量。任务性能如何随演示数量变化？对于衬衫任务，我们在 100%、75%、50% 和 25% 的数据上训练策略。我们发现，在一定程度上，在较少数据上训练的策略在 ShirtEasy 任务上的表现与在全部数据上训练的策略相似。然而，在较少数据上训练的策略在 ShirtMessy 任务上明显更差。我们推测，ShirtMessy 需要更多的演示来学习重新整理和展平衬衫所需的动态行为。
Data quantity. How does task performance vary depending on number of demonstrations? For the Shirt tasks, we train policies on 100%, 75%, 50%, and 25% of data. We find that to an extent, performance of policies trained on less data perform similarly to policies trained on all data for the ShirtEasy task. However, policies trained on less data are clearly worse at ShirtMessy. We hypothesize that ShirtMessy requires more demonstrations to learn the dynamic behaviors required for rearranging and flattening the shirt.
数据过滤。我们观察到，操作员收集的较短片段在轨迹中往往错误较少。因此，我们针对 ShirtEasy 任务基于片段时长进行数据过滤。对于该任务，我们首先从总数据集中随机抽取 25% 的样本，共得到 2164 个片段。在这个低数据量情况下，我们在以下划分上训练模型：1) 所有片段，2) 最短的 75% 片段（短于 43 秒），3) 最短的 50% 片段（短于 29 秒），4) 最短的 25% 片段（短于 20 秒）。我们在 ShirtEasy 上看到，经过一定程度的数据过滤后，性能有所提升，从使用所有片段训练时的 30% 成功率提高到使用最短 50% 片段训练时的 55% 成功率。然而，当使用最短的 25% 片段（仅 541 个片段，虽然通常是无错误的演示）时，性能下降到 40%。我们推测，在原始演示数量和质量参差不齐的演示之间找到良好平衡非常重要。虽然干净、高质量的演示对于建模最佳行为很重要，但一定数量的次优数据也可能是必要的，因为这些数据包含恢复和重试行为，有助于策略。
Data filtering. We observe that shorter episodes collected by operators tend to have less mistakes during the trajectory. We therefore perform data filtering based on the episode duration for ShirtEasy. For this task, we first take a random sample of 25% of the total dataset, resulting in a total of 2164 episodes. In this low data regime, we then train models on the following splits: 1) all episodes, 2) shortest 75% of episodes (shorter than 43s), 3) shortest 50% of episodes (shorter than 29s), 4) shortest 25% of episodes (shorter than 20s). We see on ShirtEasy that performance improves after some amount of data filtering, improving from 30% success when trained on all episodes, to 55% success when trained on the shortest 50% of episodes. However, when using the shortest 25% of episodes (only 541 episodes, though usually mistake-free demonstrations), performance dips to 40%. We speculate that finding a good balance between number of raw demonstrations and demonstrations of varying quality is important. While clean, high quality demonstrations are important for modeling the best behaviors, some amount of suboptimal data may also be necessary, since this data contains recovery and retry behaviors that can help the policy.
扩散损失与 L1 回归损失。由于我们基于 Transformer 的架构与 [ref] 非常相似，我们将扩散损失与 L1 回归损失进行比较，后者使系统更接近 ACT。我们比较了在 ShirtEasy、ShirtMessy 和模拟环境上的性能。尽管我们有一个调优良好的动作分块 + L1 回归实现，使用 1.5 亿参数模型，但我们在 ShirtMessy 上观察到 25% 的成功率，而类似规模的扩散策略为 70%。
Diffusion vs. L1 Regression Loss. Since our Transformer-based architecture is very similar to [ref], we compare the diffusion loss to an L1 regression loss, which makes the system much closer to ACT. We compare performance on ShirtEasy, ShirtMessy, and the simulated environments. Despite having a well-tuned Action chunking + L1 regression implementation with a 150M parameter model, we observe 25% success on ShirtMessy compared to 70% for the similar sized Diffusion Policy.
模拟实验。我们使用 ALOHA 2 中的 MuJoCo Menagerie 模型，在 3 个模拟双臂任务上比较扩散损失和 L1 回归损失。我们按照 [ref] 中的描述对模拟环境进行远程操作，为每个任务收集人类演示，并在数据集上训练扩散策略和 ACT L1 回归损失基线。模拟结果在 50 次 rollout 上报告。对于扩散策略（XS-LowRes）模型，我们使用 3 个种子进行 rollout。每个片段都有不同的物体位置初始化。我们观察到，除了 DoubleInsertion 之外，扩散策略在所有任务上都优于 ACT（对于 XS-LowRes）。任务描述见图 5，更多分析见附录 A。
Simulation experiments. We compare diffusion and L1 regression loss on 3 simulated bimanual tasks using the MuJoCo Menagerie model from ALOHA 2. We teleoperate the simulated environments as described in [ref] to collect human demonstrations for each task, and train Diffusion Policies and an ACT L1 regression loss baseline on the datasets. Simulation results are reported over 50 rollouts. For Diffusion Policy (XS-LowRes) models, we run rollouts with 3 seeds. Each episode has a different initialization of object positions. We observe that Diffusion Policy outperforms ACT (for XS-LowRes) for all tasks except DoubleInsertion. See Figure 5 for descriptions of tasks and Appendix A for more analysis.
虽然我们的核心模型仅针对每个任务进行训练，但我们确实观察到模型展现出一些有希望的泛化迹象。在衬衫任务中，我们观察到模型在未见过的衬衫上成功执行，这些衬衫与训练数据中看到的衬衫截然不同。训练集中出现的衬衫仅为儿童尺寸、短袖，颜色有红、白、蓝、藏青和淡蓝，而未见的衬衫是灰色成人男款长袖。我们还观察到衬衫模型在完全不同的建筑中的一台未见过的机器人上成功执行（家庭环境，背景为白墙，而非训练集中看到的工业实验室背景）。
While our core models are only trained per-task, we do observe some promising signs of generalization from our models. In the Shirt tasks, we observe successful rollouts of the model on unseen shirts which are quite different from shirts seen in the training data. Shirts seen in the train set were only kids sizes with short sleeves and red, white, blue, navy, and baby blue colors, while the unseen shirt is a gray adult men’s size with long sleeves. We also observe successful rollouts of the Shirt model on an unseen robot in a completely different building (home environment with white wall as background instead of the industrial lab background seen in the train set).
我们通过衡量模型在 ShirtMessy 任务上的泛化能力来突破模型的边界，该任务在训练集中有 3,113 条演示。然而，该任务的状态空间仍然很大，因为可变形衬衫可能呈现多种构型。我们观察到，模型能够处理衬衫初始状态为倾斜±60 度、起皱且正面朝上的情况，并学习到在此构型下展平和居中的良好行为。我们观察到，模型通常无法从衬衫 180 度翻转或反面朝上的状态中恢复，因为训练集中没有此类实例。类似地，在鞋带任务中，我们能够学习“拉直”行为，但对于训练分布之外的状态（例如，鞋子翻倒、翻转或鞋带缠绕）则无法恢复。
We push the boundaries of our model by measuring our model’s generalization ability for the ShirtMessy task, for which we have 3,113 demonstrations in the train set. The state space of the task, however, is still large since the deformable shirts may take many configurations. We observe that the model can handle initializations of the shirt that are \(\pm\) 60 degrees tilted, wrinkled, and right-side-up on the table, with the model learning good behaviors for flattening and centering the shirt given this configuration. We observe that the model usually fails to recover from shirts being 180 degrees or face-down on the table, since there are no instances of this in the training set. Similarly on the Lace tasks, we are able to learn ”straightening” behaviors, however fail to recover for states outside of the train distribution (for example, if the shoe tips over, flips around, or the laces get tangled).
在 RandomKitchen 任务中，我们观察到一定程度的初始状态泛化，因为物体可以在机器人任务空间内的任意位置初始化。此外，我们在一台仅有 216 条演示的机器人上评估该模型，而其余 2,983 条演示是在另一栋建筑中收集的，使用的是 ALOHA 的硬件迭代版本，机器人安装位置不同。
On RandomKitchen, we observe some amount of initial state generalization given that the objects can be initialized anywhere within the robot’s task space. In addition, we evaluate this model on a robot which has 216 demonstrations, where the other 2,983 demonstrations are collected in another building with a hardware iteration of ALOHA that had different robot mounting positions.
我们提出了 ALOHA Unleashed，这是一种学习灵巧机器人行为的简单方法。我们在 ALOHA 2 平台上收集了超过 26,000 个演示，并基于这些数据训练了一个基于 Transformer 的扩散策略。我们在真实和模拟环境中都展示了灵巧的行为。
We present ALOHA Unleashed, a simple recipe for learning dexterous robot behaviors. We collect over 26k demonstrations on the ALOHA 2 platform, and train a Transformer-based Diffusion Policy on the data. We demonstrate dexterous behaviors in both real and simulated environments.
ALOHA Unleashed 表明，一种简单的方法可以推动机器人学习中双臂灵巧行为的边界。然而，这种方法在几个方面存在局限：策略一次只能训练一个任务，而其他方法使用一组模型权重，通过语言或目标图像进行条件化来执行多个任务。此外，策略每 1 秒重新规划一次，这可能不足以应对非常反应性的任务。ALOHA Unleashed 每个任务还使用大量人工演示，收集这些演示非常耗时。
ALOHA Unleashed shows that a simple recipe could push the boundaries of bimanual, dexterous behaviors in robot learning. However, this approach is limited in several aspects: policies are trained for only one task at a time, whereas other approaches use a single set of model weights that is conditioned on language or goal images to perform multiple tasks. In addition, the policy replans every 1 second, which may not be fast enough for very reactive tasks. ALOHA Unleashed also uses many human demonstrations per task, which are time consuming to collect.
我们希望扩展 ALOHA Unleashed，使用一个能够执行多个任务的单一模型来增加任务数量，添加建模改进以执行更具反应性的任务，并继续改进数据复杂性以减少学习灵巧行为所需的数据量。
We hope to extend ALOHA Unleashed to expand the number of tasks using a single model that can perform multiple tasks, add modeling improvements to perform more reactive tasks, and continue to improve data complexity to reduce the amount of data required to learn dexterous behaviors.