Prompt-based learning has emerged as a successful paradigm in natural language processing, where a single general-purpose language model can be instructed to perform any task specified by input prompts. Yet task specification in robotics comes in various forms, such as imitating one-shot demonstrations, following language instructions, and reaching visual goals. They are often considered different tasks and tackled by specialized models. We show that a wide spectrum of robot manipulation tasks can be expressed with multimodal prompts, interleaving textual and visual tokens. Accordingly, we develop a new simulation benchmark that consists of thousands of procedurally-generated tabletop tasks with multimodal prompts, 600K+ expert trajectories for imitation learning, and a four-level evaluation protocol for systematic generalization. We design a transformer-based robot agent, VIMA, that processes these prompts and outputs motor actions autoregressively. VIMA features a recipe that achieves strong model scalability and data efficiency. It outperforms alternative designs in the hardest zero-shot generalization setting by up to $2.9\times$ task success rate given the same training data. With $10\times$ less training data, VIMA still performs $2.7\times$ better than the best competing variant. Code and video demos are available at https://vimalabs.github.io/
核心贡献 · Key contributions
提出多模态提示方法，将多种机器人操作任务统一为单一序列建模问题。 Proposes multimodal prompting to unify diverse robot manipulation tasks into a single sequence modeling problem.
开发 VIMA-Bench 仿真基准，包含 17 个任务、60 万+专家轨迹和四级评估协议。 Develops VIMA-Bench, a simulation benchmark with 17 tasks, 600K+ expert trajectories, and a four-level evaluation protocol.
设计 VIMA，一种基于 Transformer 的智能体，采用以对象为中心的标记和交叉注意力条件化处理多模态提示。 Designs VIMA, a transformer-based agent with object-centric tokens and cross-attention conditioning for multimodal prompts.
在相同训练数据下，零样本泛化任务成功率比基线高出最多 2.9 倍。 Achieves up to 2.9x task success rate over baselines in zero-shot generalization with same training data.
展示强大的数据效率：使用少 10 倍的数据，VIMA 仍比最佳竞争对手高出 2.7 倍。 Demonstrates strong data efficiency: with 10x less data, VIMA still outperforms best competitor by 2.7x.
提供开源代码、基准和预训练模型，确保可复现性。 Provides open-source code, benchmark, and pre-trained models for reproducibility.
局限 · Limitations
评估仅限于仿真桌面任务；未测试真实机器人部署。 Evaluation limited to simulated tabletop tasks; real-world robot deployment not tested.
依赖预训练的目标检测器（Mask R-CNN），在新环境中可能失败。 Relies on pre-trained object detector (Mask R-CNN) which may fail in novel environments.
从专家轨迹进行行为克隆可能无法处理次优或有噪声的演示。 Behavioral cloning from expert trajectories may not handle suboptimal or noisy demonstrations.
对新颖任务（L4）的零样本泛化仍表现出显著性能下降。 Zero-shot generalization to novel tasks (L4) still shows significant performance drop.
模型可扩展性测试至 2 亿参数；更大模型可能表现不同。 Model scalability tested up to 200M parameters; larger models may have different behaviors.
论文章节 · Sections（共 12）
摘要Abstract
1 引言1 Introduction
2 任务规范的多模态提示2 Multimodal Prompts for Task Specification
3 VIMA-Bench：多模态机器人学习基准3 VIMA-Bench: Benchmark for Multimodal Robot Learning