Recent progress in vision language foundation models has shown their ability to understand multimodal data and resolve complicated vision language tasks, including robotics manipulation. We seek a straightforward way of making use of existing vision-language models (VLMs) with simple fine-tuning on robotics data. To this end, we derive a simple and novel vision-language manipulation framework, dubbed RoboFlamingo, built upon the open-source VLMs, OpenFlamingo. Unlike prior works, RoboFlamingo utilizes pre-trained VLMs for single-step vision-language comprehension, models sequential history information with an explicit policy head, and is slightly fine-tuned by imitation learning only on language-conditioned manipulation datasets. Such a decomposition provides RoboFlamingo the flexibility for open-loop control and deployment on low-performance platforms. By exceeding the state-of-the-art performance with a large margin on the tested benchmark, we show RoboFlamingo can be an effective and competitive alternative to adapt VLMs to robot control. Our extensive experimental results also reveal several interesting conclusions regarding the behavior of different pre-trained VLMs on manipulation tasks. We believe RoboFlamingo has the potential to be a cost-effective and easy-to-use solution for robotics manipulation, empowering everyone with the ability to fine-tune their own robotics policy.
核心贡献 · Key contributions
提出 RoboFlamingo，一个通过模仿学习将 OpenFlamingo 视觉语言模型适配到机器人操作的简单框架。 Proposes RoboFlamingo, a simple framework adapting OpenFlamingo VLM to robot manipulation via imitation learning.
通过独立的策略头解耦视觉语言理解与决策，用于历史信息建模。 Decouples vision-language understanding and decision-making with a separate policy head for history modeling.
在 CALVIN 基准上取得最优性能，比先前方法提升 2 倍。 Achieves state-of-the-art performance on CALVIN benchmark, with 2x improvement over prior methods.
展示了对未见视觉场景和语言指令的零样本泛化能力。 Demonstrates zero-shot generalization to unseen visual contexts and language instructions.
展示了数据效率：更大的视觉语言模型在有限机器人演示数据下表现更好。 Shows data efficiency: larger VLMs perform better with limited robot demonstration data.
支持开环控制和单 GPU 部署，降低计算成本。 Enables open-loop control and deployment on single GPU, reducing computational cost.
局限 · Limitations
仅在仿真环境（CALVIN）中评估，未在真实机器人上部署。 Evaluated only in simulation (CALVIN), not on real-world robots.
需要语言标注的演示数据，此类数据稀缺（仅占总数据的 1%）。 Requires language-annotated demonstration data, which is scarce (only 1% of total data).
开环控制性能下降，除非使用跳步演示重新训练。 Open-loop control performance degrades without retraining with jump step demonstrations.
对同义指令的泛化能力在序列后续任务中下降。 Generalization to synonymous instructions drops for later tasks in sequence.
依赖 OpenFlamingo 骨干网络，性能可能因其他视觉语言模型而异。 Relies on OpenFlamingo backbone; performance may vary with other VLMs.
论文章节 · Sections（共 17）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Work
3 背景3 Background
4 RoboFlamingo4 RoboFlamingo
4.1 语言条件机器人控制4.1 Language-conditioned Robot Control