RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→多模态大语言模型（MLLMs）在多种多模态场景中展现出卓越能力，但在机器人场景，尤其是长程操作任务中，存在显著局限性。这些局限性源于当前 MLLMs 缺乏三种关键的机器人脑能力：规划能力（将复杂操作指令分解为可管理的子任务）、可供性感知（识别和解释交互对象的可供性）以及轨迹预测（预见成功执行所需的完整操作轨迹）。为从抽象到具体增强机器人脑的核心能力，我们引入了 ShareRobot，一个高质量异构数据集，标注了任务规划、对象可供性和末端执行器轨迹等多维信息。ShareRobot 的多样性和准确性由三位人工标注员精心优化。基于此数据集，我们开发了 RoboBrain，一个基于 MLLM 的模型，结合了机器人和通用多模态数据，采用多阶段训练策略，并利用长视频和高分辨率图像提升其机器人操作能力。大量实验表明，RoboBrain 在各种机器人任务中达到了最先进的性能，突显了其提升机器人脑能力的潜力。
Recent advancements in Multimodal Large Language Models (MLLMs) have shown remarkable capabilities across various multimodal contexts. However, their application in robotic scenarios, particularly for long-horizon manipulation tasks, reveals significant limitations. These limitations arise from the current MLLMs lacking three essential robotic brain capabilities: Planning Capability, which involves decomposing complex manipulation instructions into manageable sub-tasks; Affordance Perception, the ability to recognize and interpret the affordances of interactive objects; and Trajectory Prediction, the foresight to anticipate the complete manipulation trajectory necessary for successful execution. To enhance the robotic brain's core capabilities from abstract to concrete, we introduce ShareRobot, a high-quality heterogeneous dataset that labels multi-dimensional information such as task planning, object affordance, and end-effector trajectory. ShareRobot's diversity and accuracy have been meticulously refined by three human annotators. Building on this dataset, we developed RoboBrain, an MLLM-based model that combines robotic and general multi-modal data, utilizes a multi-stage training strategy, and incorporates long videos and high-resolution images to improve its robotic manipulation capabilities. Extensive experiments demonstrate that RoboBrain achieves state-of-the-art performance across various robotic tasks, highlighting its potential to advance robotic brain capabilities.