OpenVLA: An Open-Source Vision-Language-Action Model
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→在互联网规模的视觉-语言数据和多样化机器人演示上预训练的大型策略，有潜力改变我们教授机器人新技能的方式：无需从头训练新行为，而是微调此类视觉-语言-动作（VLA）模型，以获得稳健、可泛化的视觉运动控制策略。然而，VLA 在机器人领域的广泛采用面临挑战：1）现有 VLA 大多封闭且不公开；2）先前工作未能探索高效微调 VLA 以适应新任务的方法，这是采用的关键。针对这些挑战，我们提出了 OpenVLA，一个 70 亿参数的开源 VLA，在包含 97 万条真实世界机器人演示的多样化数据集上训练。OpenVLA 基于 Llama 2 语言模型，结合了融合 DINOv2 和 SigLIP 预训练特征的视觉编码器。由于数据多样性和新模型组件的加入，OpenVLA 在通用操作上表现出色，在 29 个任务和多种机器人形态上，绝对任务成功率比 RT-2-X（550 亿参数）等闭源模型高出 16.5%，而参数量仅为后者的 1/7。我们进一步展示了可以有效地微调 OpenVLA 以适应新场景，在多物体多任务环境中具有特别强的泛化能力和语言接地能力，比从头开始的模仿学习方法（如 Diffusion Policy）高出 20.4%。我们还探索了计算效率；作为另一贡献，我们展示了 OpenVLA 可以通过现代低秩适应方法在消费级 GPU 上微调，并通过高效量化提供服务，而不影响下游成功率。最后，我们发布了模型检查点、微调笔记本和 PyTorch 代码库，内置支持在 Open X-Embodiment 数据集上大规模训练 VLA。
Large policies pretrained on a combination of Internet-scale vision-language data and diverse robot demonstrations have the potential to change how we teach robots new skills: rather than training new behaviors from scratch, we can fine-tune such vision-language-action (VLA) models to obtain robust, generalizable policies for visuomotor control. Yet, widespread adoption of VLAs for robotics has been challenging as 1) existing VLAs are largely closed and inaccessible to the public, and 2) prior work fails to explore methods for efficiently fine-tuning VLAs for new tasks, a key component for adoption. Addressing these challenges, we introduce OpenVLA, a 7B-parameter open-source VLA trained on a diverse collection of 970k real-world robot demonstrations. OpenVLA builds on a Llama 2 language model combined with a visual encoder that fuses pretrained features from DINOv2 and SigLIP. As a product of the added data diversity and new model components, OpenVLA demonstrates strong results for generalist manipulation, outperforming closed models such as RT-2-X (55B) by 16.5% in absolute task success rate across 29 tasks and multiple robot embodiments, with 7x fewer parameters. We further show that we can effectively fine-tune OpenVLA for new settings, with especially strong generalization results in multi-task environments involving multiple objects and strong language grounding abilities, and outperform expressive from-scratch imitation learning methods such as Diffusion Policy by 20.4%. We also explore compute efficiency; as a separate contribution, we show that OpenVLA can be fine-tuned on consumer GPUs via modern low-rank adaptation methods and served efficiently via quantization without a hit to downstream success rate. Finally, we release model checkpoints, fine-tuning notebooks, and our PyTorch codebase with built-in support for training VLAs at scale on Open X-Embodiment datasets.