Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train generalist X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. More details can be found on the project website https://robotics-transformer-x.github.io.
核心贡献 · Key contributions
汇集了来自 21 个机构、22 种机器人的大规模多形态数据集，涵盖 527 种技能和 160,266 个任务。 Assembled a large-scale multi-embodiment dataset from 22 robots across 21 institutions, covering 527 skills and 160,266 tasks.
展示了基于 Transformer 的策略（RT-1-X 和 RT-2-X）在跨机器人训练中的正向迁移效果。 Demonstrated positive transfer across robots using Transformer-based policies (RT-1-X and RT-2-X) trained on X-embodiment data.
RT-1-X 在小规模数据集上比原始方法提升 50%，RT-2-X 在新兴技能上实现 3 倍改进。 Showed that RT-1-X outperforms original methods by 50% on small-scale datasets, and RT-2-X achieves 3x improvement in emergent skills.
提供了开源资源：统一数据集、代码和预训练的 RT-1-X 模型，以支持跨形态研究。 Provided open-source resources: unified dataset, code, and pre-trained RT-1-X model for X-embodiment research.
发现模型容量和网络规模预训练对有效的跨形态迁移至关重要。 Identified that model capacity and web-scale pretraining are critical for effective X-embodiment transfer.
建立了由 21 个机构参与的社区驱动项目，以持续扩展数据集。 Established a community-driven effort involving 21 institutions to grow the dataset over time.
局限 · Limitations
未考虑具有完全不同感知和执行模态的机器人。 Does not consider robots with very different sensing and actuation modalities.
未研究对训练中未见的新机器人的泛化能力。 Does not study generalization to new robots not seen during training.
未提供正向迁移是否发生的判定标准。 Does not provide a decision criterion for when positive transfer occurs.
不同形态间的动作和观测空间仅粗略对齐。 Action and observation spaces are only coarsely aligned across embodiments.
实验仅使用了数据集中 22 种形态中的 9 种机械臂。 Experiments limited to 9 manipulators out of 22 total embodiments in the dataset.
论文章节 · Sections（共 15）
摘要Abstract
一、引言I Introduction
二、相关工作II Related Work
三、开放 X-具身化仓库III The Open X-Embodiment Repository
三-A 开放 X-具身化数据集III-A The Open X-Embodiment Dataset
三-B 数据集分析III-B Dataset Analysis
四、RT-X 设计IV RT-X Design
四-A 数据格式统一IV-A Data format consolidation
四-B 策略架构IV-B Policy architectures
四-C 训练与推理细节IV-C Training and inference details
五、实验结果V Experimental Results
五-A 不同具身化下的分布内性能V-A In-distribution performance across different embodiments
五-B 对分布外场景的泛化改进V-B Improved generalization to out-of-distribution settings
五-C 设计决策V-C Design decisions
六、讨论、未来工作与开放问题VI Discussion, Future Work, and Open Problems