We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control to boost generalization and enable emergent semantic reasoning. Our goal is to enable a single end-to-end trained model to both learn to map robot observations to actions and enjoy the benefits of large-scale pretraining on language and vision-language data from the web. To this end, we propose to co-fine-tune state-of-the-art vision-language models on both robotic trajectory data and Internet-scale vision-language tasks, such as visual question answering. In contrast to other approaches, we propose a simple, general recipe to achieve this goal: in order to fit both natural language responses and robotic actions into the same format, we express the actions as text tokens and incorporate them directly into the training set of the model in the same way as natural language tokens. We refer to such category of models as vision-language-action models (VLA) and instantiate an example of such a model, which we call RT-2. Our extensive evaluation (6k evaluation trials) shows that our approach leads to performant robotic policies and enables RT-2 to obtain a range of emergent capabilities from Internet-scale training. This includes significantly improved generalization to novel objects, the ability to interpret commands not present in the robot training data (such as placing an object onto a particular number or icon), and the ability to perform rudimentary reasoning in response to user commands (such as picking up the smallest or largest object, or the one closest to another object). We further show that incorporating chain of thought reasoning allows RT-2 to perform multi-stage semantic reasoning, for example figuring out which object to pick up for use as an improvised hammer (a rock), or which type of drink is best suited for someone who is tired (an energy drink).
核心贡献 · Key contributions
提出视觉-语言-动作（VLA）模型，在机器人轨迹数据和网络规模的视觉-语言任务上共同微调视觉-语言模型。 Proposes vision-language-action (VLA) models that co-fine-tune VLMs on robotic trajectory data and web-scale vision-language tasks.
引入 RT-2，一个源自 PaLI-X 和 PaLM-E 的 VLA 模型系列，参数高达 550 亿。 Introduces RT-2, a family of VLA models derived from PaLI-X and PaLM-E, with up to 55B parameters.
证明将机器人动作表示为文本标记可直接微调视觉-语言模型用于机器人控制。 Demonstrates that representing robot actions as text tokens enables direct fine-tuning of VLMs for robotic control.
通过网络规模预训练迁移，展示了对新物体、场景和指令的显著泛化改进。 Shows significant improvements in generalization to novel objects, scenes, and instructions via web-scale pretraining transfer.
揭示了从视觉-语言预训练中涌现的能力，包括符号理解、推理和人类识别。 Reveals emergent capabilities including symbol understanding, reasoning, and human recognition from vision-language pretraining.
展示了 VLA 模型中的思维链推理，用于多阶段语义推理任务。 Demonstrates chain-of-thought reasoning in VLA models for multi-stage semantic reasoning tasks.
局限 · Limitations
物理技能局限于机器人数据中见过的技能分布；网络数据不带来新动作。 Physical skills are limited to the distribution of skills seen in robot data; no new motions from web data.
大型 VLA 模型的高计算成本可能阻碍高频实时控制。 High computational cost of large VLA models may hinder real-time control at high frequencies.
适合构建 VLA 模型的开源视觉-语言模型数量有限。 Limited availability of open-source VLM models suitable for building VLA models.
评估局限于特定机器人平台和厨房环境，限制了通用性。 Evaluation is confined to a specific robot platform and kitchen environment, limiting generality.
共同微调需要平衡机器人和网络数据，这可能依赖于数据集。 Co-fine-tuning requires balancing robot and web data, which may be dataset-dependent.
4.1 RT-2 在已知任务上的表现如何？更重要的是，它能否泛化到新物体、背景和环境？4.1 How does RT-2 perform on seen tasks and more importantly, generalize over new objects, backgrounds, and environments?
4.2 我们能观察到并衡量 RT-2 的涌现能力吗？4.2 Can we observe and measure any emergent capabilities of RT-2?
4.3 泛化能力如何随参数数量和其他设计选择而变化？4.3 How does the generalization vary with parameter count and other design decisions?
4.4 RT-2 能否像视觉-语言模型一样展现出思维链推理的迹象？4.4 Can RT-2 exhibit signs of chain-of-thought reasoning similarly to vision-language models?