By transferring knowledge from large, diverse, task-agnostic datasets, modern machine learning models can solve specific downstream tasks either zero-shot or with small task-specific datasets to a high level of performance. While this capability has been demonstrated in other fields such as computer vision, natural language processing or speech recognition, it remains to be shown in robotics, where the generalization capabilities of the models are particularly critical due to the difficulty of collecting real-world robotic data. We argue that one of the keys to the success of such general robotic models lies with open-ended task-agnostic training, combined with high-capacity architectures that can absorb all of the diverse, robotic data. In this paper, we present a model class, dubbed Robotics Transformer, that exhibits promising scalable model properties. We verify our conclusions in a study of different model classes and their ability to generalize as a function of the data size, model size, and data diversity based on a large-scale data collection on real robots performing real-world tasks. The project's website and videos can be found at robotics-transformer1.github.io
核心贡献 · Key contributions
提出 RT-1，一种基于 Transformer 的模型，用于大规模真实世界机器人控制。 Proposes RT-1, a Transformer-based model for real-world robotic control at scale.
展示 RT-1 在超过 700 个训练指令上达到 97%的成功率。 Demonstrates RT-1 achieves 97% success on over 700 training instructions.
显示 RT-1 在新任务、干扰物和背景上的泛化能力显著优于基线。 Shows RT-1 generalizes to new tasks, distractors, and backgrounds significantly better than baselines.
集成 TokenLearner 和 FiLM 条件化以实现高效的实时推理。 Integrates TokenLearner and FiLM conditioning for efficient real-time inference.
展示 RT-1 吸收来自仿真和不同机器人的异构数据而不损失性能。 Demonstrates absorption of heterogeneous data from simulation and different robots without performance loss.
在 SayCan 框架中实现多达 50 步的长时域任务。 Enables long-horizon tasks with up to 50 steps in the SayCan framework.
局限 · Limitations
RT-1 是一种模仿学习方法，受限于演示者的表现。 RT-1 is an imitation learning method, limited by demonstrator performance.
对新指令的泛化仅限于已见概念的组合。 Generalization to new instructions is limited to combinations of seen concepts.
该方法在大量但非高度灵巧的操作任务上评估。 Method is evaluated on a large but not highly dexterous set of manipulation tasks.
对背景和环境的鲁棒性可通过更多样化的数据改进。 Robustness to backgrounds and environments could be improved with more diverse data.
反应速度和上下文保持可通过可扩展的注意力和记忆改进。 Reaction speed and context retention could benefit from scalable attention and memory.
6.2 RT-1 能否学习执行大量指令，并泛化到新任务、物体和环境？6.2 Can RT-1 learn to perform a large number of instructions, and to generalize to new tasks, objects and environments?
6.3 通过整合异构数据源（如仿真或不同机器人的数据），能否进一步推动模型性能？6.3 Can we push the resulting model further by incorporating heterogeneous data sources such as simulation or data from different robots?
6.4 各种方法在长时域机器人场景中的泛化表现如何？6.4 How do various methods generalize long-horizon robotic scenarios?
6.5 泛化指标如何随数据量和数据多样性的变化而变化？6.5 How do generalization metrics change with varying amounts of data quantity and data diversity?
7 结论、局限性与未来工作7 Conclusions, Limitations and Future Work