We report our recent progress towards building generalist robot policies, the development of GR-3. GR-3 is a large-scale vision-language-action (VLA) model. It showcases exceptional capabilities in generalizing to novel objects, environments, and instructions involving abstract concepts. Furthermore, it can be efficiently fine-tuned with minimal human trajectory data, enabling rapid and cost-effective adaptation to new settings. GR-3 also excels in handling long-horizon and dexterous tasks, including those requiring bi-manual manipulation and mobile movement, showcasing robust and reliable performance. These capabilities are achieved through a multi-faceted training recipe that includes co-training with web-scale vision-language data, efficient fine-tuning from human trajectory data collected via VR devices, and effective imitation learning with robot trajectory data. In addition, we introduce ByteMini, a versatile bi-manual mobile robot designed with exceptional flexibility and reliability, capable of accomplishing a wide range of tasks when integrated with GR-3. Through extensive real-world experiments, we show GR-3 surpasses the state-of-the-art baseline method, $π_0$, on a wide variety of challenging tasks. We hope GR-3 can serve as a step towards building generalist robots capable of assisting humans in daily life.
核心贡献 · Key contributions
提出 GR-3，一个大规模视觉-语言-动作（VLA）模型，用于通用机器人策略。 Introduces GR-3, a large-scale VLA model for generalist robot policies.
通过与网络规模的视觉-语言数据联合训练，实现对新颖物体、环境和指令的强泛化能力。 Achieves strong generalization to novel objects, environments, and instructions via co-training with web-scale vision-language data.
利用 VR 设备收集的少量人类轨迹数据，实现高效的少样本适应。 Enables efficient few-shot adaptation using minimal human trajectory data collected via VR devices.
在长时域和灵巧任务中表现出色，包括双臂操作和移动运动。 Excels in long-horizon and dexterous tasks, including bi-manual manipulation and mobile movement.
提出 ByteMini，一个灵活可靠的多功能双臂移动机器人。 Introduces ByteMini, a versatile bi-manual mobile robot with high flexibility and reliability.
在具有挑战性的真实世界任务中超越最先进基线π0。 Surpasses state-of-the-art baseline π0 on challenging real-world tasks.
局限 · Limitations
在遵循涉及新概念和新物体的未见指令时会出现错误。 Makes mistakes in following unseen instructions involving novel concepts and objects.
难以抓取形状未知的物体。 Struggles with grasping objects with unseen shapes.
在部署过程中可能陷入分布外状态且无法恢复。 Can be stuck in out-of-distribution states during rollouts and fails to recover.
受限于收集真实机器人轨迹的成本和时间。 Limited by the cost and time of collecting real-world robot trajectories.
未结合强化学习以进一步提升鲁棒性。 Does not incorporate reinforcement learning to further improve robustness.
论文章节 · Sections（共 16）
摘要Abstract
1 引言1 Introduction
2 GR-3 模型2 The GR-3 Model
3 训练方案3 Training Recipe
3.1 基于机器人轨迹数据的模仿学习3.1 Imitation Learning with Robot Trajectory Data
3.2 与视觉语言数据的协同训练3.2 Co-Training with Vision-Language Data
3.3 基于人类轨迹数据的少样本泛化3.3 Few-Shot Generalization with Human Trajectory Data