大型语言模型（LLM）被证明拥有丰富的可操作知识，可以以推理和规划的形式提取出来用于机器人操作。尽管取得了进展，大多数方法仍然依赖预定义的运动基元来与环境进行物理交互，这仍然是一个主要瓶颈。在这项工作中，我们旨在针对大量操作任务，根据开放式指令和开放式物体，合成机器人轨迹，即密集的 6 自由度末端执行器路径点序列。我们首先观察到，LLM 在根据自由形式的语言指令推断可供性和约束方面表现出色。更重要的是，通过利用其代码编写能力，它们可以与视觉语言模型（VLM）交互，组合 3D 价值图，将知识锚定到智能体的观察空间中。组合后的价值图随后用于基于模型的规划框架，以零样本方式合成闭环机器人轨迹，并对动态扰动具有鲁棒性。我们进一步展示了所提出的框架如何通过有效学习涉及接触丰富交互的场景的动态模型，从在线经验中受益。我们在模拟和真实机器人环境中对所提出的方法进行了大规模研究，展示了执行由自由形式自然语言指定的各种日常操作任务的能力。视频和代码见 https://voxposer.github.io。
Large language models (LLMs) are shown to possess a wealth of actionable knowledge that can be extracted for robot manipulation in the form of reasoning and planning. Despite the progress, most still rely on pre-defined motion primitives to carry out the physical interactions with the environment, which remains a major bottleneck. In this work, we aim to synthesize robot trajectories, i.e., a dense sequence of 6-DoF end-effector waypoints, for a large variety of manipulation tasks given an open-set of instructions and an open-set of objects. We achieve this by first observing that LLMs excel at inferring affordances and constraints given a free-form language instruction. More importantly, by leveraging their code-writing capabilities, they can interact with a vision-language model (VLM) to compose 3D value maps to ground the knowledge into the observation space of the agent. The composed value maps are then used in a model-based planning framework to zero-shot synthesize closed-loop robot trajectories with robustness to dynamic perturbations. We further demonstrate how the proposed framework can benefit from online experiences by efficiently learning a dynamics model for scenes that involve contact-rich interactions. We present a large-scale study of the proposed method in both simulated and real-robot environments, showcasing the ability to perform a large variety of everyday manipulation tasks specified in free-form natural language. Videos and code at https://voxposer.github.io
核心贡献 · Key contributions
提出 VoxPoser 框架，从大语言模型中提取可供性和约束，组合成 3D 价值图用于机器人操作。 Proposes VoxPoser, a framework that extracts affordances and constraints from LLMs to compose 3D value maps for robot manipulation.
利用大语言模型的代码编写能力与视觉语言模型交互，通过价值图将知识锚定到 3D 观测空间。 Leverages LLMs' code-writing to interact with VLMs, grounding knowledge into 3D observation space via value maps.
在基于模型的规划框架中使用组合的价值图，实现零样本闭环轨迹合成。 Uses composed value maps in a model-based planning framework for zero-shot closed-loop trajectory synthesis.
通过 5Hz 的快速重规划，展示了对动态扰动的鲁棒性。 Demonstrates robustness to dynamic perturbations through fast replanning at 5 Hz.
通过将零样本轨迹作为探索先验，展示了接触丰富任务中动力学模型的高效学习。 Shows efficient learning of dynamics models for contact-rich tasks by using zero-shot trajectories as exploration priors.
在仿真和真实环境中对多种日常操作任务进行了大规模验证。 Validates on a large variety of everyday manipulation tasks in both simulation and real-world settings.
局限 · Limitations
依赖外部感知模块，限制了需要整体视觉推理或细粒度几何的任务。 Relies on external perception modules, limiting tasks requiring holistic visual reasoning or fine-grained geometry.
需要通用动力学模型才能以相同泛化水平处理接触丰富任务。 Requires a general-purpose dynamics model for contact-rich tasks with same level of generalization.
运动规划器仅考虑末端执行器轨迹，未进行整臂规划。 Motion planner only considers end-effector trajectories, not whole-arm planning.
需要为大语言模型手动设计提示词。 Manual prompt engineering is required for LLMs.
感知模块对物体姿态敏感，对物体部件鲁棒性较差。 Perception module is sensitive to object poses and less robust for object parts.
论文章节 · Sections（共 14）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Works
3 方法3 Method
3.1 问题表述3.1 Problem Formulation
3.2 通过 VoxPoser 实现语言指令接地3.2 Grounding Language Instruction via VoxPoser
3.3 基于 VoxPoser 的零样本轨迹合成3.3 Zero-Shot Trajectory Synthesis with VoxPoser
3.4 利用在线经验的高效动力学学习3.4 Efficient Dynamics Learning with Online Experiences
4 实验与分析4 Experiments and Analysis
4.1 VoxPoser 在日常操作任务中的应用4.1 VoxPoser for Everyday Manipulation Tasks
4.2 对未见指令和属性的泛化4.2 Generalization to Unseen Instructions and Attributes
4.3 利用在线经验的高效动力学学习4.3 Efficient Dynamics Learning with Online Experiences
4.4 错误分析4.4 Error Breakdown
5 结论、局限性与未来工作5 Conclusion, Limitations, & Future Works