ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→将机器人操作任务表示为关联机器人与环境的约束，是编码期望机器人行为的一种有前景的方式。然而，如何制定约束使其 1) 适用于多样任务，2) 无需手动标注，3) 可由现成求解器优化以实时生成机器人动作，仍不清楚。在这项工作中，我们引入了关系关键点约束（ReKep），一种用于机器人操作中约束的视觉基础表示。具体而言，ReKep 表示为将环境中一组 3D 关键点映射到数值成本的 Python 函数。我们证明，通过将操作任务表示为一系列关系关键点约束，我们可以采用分层优化过程，以实时频率的感知-动作循环求解机器人动作（表示为 SE(3) 中的一系列末端执行器姿态）。此外，为了避免为每个新任务手动指定 ReKep，我们设计了一个自动化程序，利用大型视觉模型和视觉-语言模型从自由形式的语言指令和 RGB-D 观测中生成 ReKep。我们在轮式单臂平台和固定双臂平台上展示了系统实现，能够执行多种操作任务，包括多阶段、野外、双臂和反应性行为，所有这些都无需特定任务数据或环境模型。网站：https://rekep-robot.github.io/。
Representing robotic manipulation tasks as constraints that associate the robot and the environment is a promising way to encode desired robot behaviors. However, it remains unclear how to formulate the constraints such that they are 1) versatile to diverse tasks, 2) free of manual labeling, and 3) optimizable by off-the-shelf solvers to produce robot actions in real-time. In this work, we introduce Relational Keypoint Constraints (ReKep), a visually-grounded representation for constraints in robotic manipulation. Specifically, ReKep is expressed as Python functions mapping a set of 3D keypoints in the environment to a numerical cost. We demonstrate that by representing a manipulation task as a sequence of Relational Keypoint Constraints, we can employ a hierarchical optimization procedure to solve for robot actions (represented by a sequence of end-effector poses in SE(3)) with a perception-action loop at a real-time frequency. Furthermore, in order to circumvent the need for manual specification of ReKep for each new task, we devise an automated procedure that leverages large vision models and vision-language models to produce ReKep from free-form language instructions and RGB-D observations. We present system implementations on a wheeled single-arm platform and a stationary dual-arm platform that can perform a large variety of manipulation tasks, featuring multi-stage, in-the-wild, bimanual, and reactive behaviors, all without task-specific data or environment models. Website at https://rekep-robot.github.io/.
将机器人操作任务表示为关联机器人与环境的约束，是编码期望机器人行为的一种有前景的方式。然而，如何制定这些约束，使其 1) 适用于多种任务，2) 无需人工标注，3) 可由现成求解器优化以实时生成机器人动作，仍不清楚。在这项工作中，我们引入了关系关键点约束（ReKep），一种用于机器人操作约束的视觉基础表示。具体来说，ReKep 表示为 Python 函数，将环境中的一组 3D 关键点映射到数值代价。我们证明，通过将操作任务表示为一系列关系关键点约束，我们可以采用分层优化过程来求解机器人动作（由 \(SE(3)\) 中的一系列末端执行器姿态表示），并以实时频率进行感知-动作循环。此外，为了避免每个新任务都需要手动指定 ReKep，我们设计了一种自动化程序，利用大型视觉模型和视觉-语言模型从自由形式的语言指令和 RGB-D 观测中生成 ReKep。我们在轮式单臂平台和固定双臂平台上展示了系统实现，能够执行多种操作任务，具有多阶段、野外、双臂和反应性行为，且无需特定任务数据或环境模型。†† *表示同等贡献。通讯作者：Wenlong Huang <wenlongh@stanford.edu>。
Representing robotic manipulation tasks as constraints that associate the robot and the environment is a promising way to encode desired robot behaviors. However, it remains unclear how to formulate the constraints such that they are 1) versatile to diverse tasks, 2) free of manual labeling, and 3) optimizable by off-the-shelf solvers to produce robot actions in real-time. In this work, we introduce Relational Keypoint Constraints (ReKep), a visually-grounded representation for constraints in robotic manipulation. Specifically, ReKep is expressed as Python functions mapping a set of 3D keypoints in the environment to a numerical cost. We demonstrate that by representing a manipulation task as a sequence of Relational Keypoint Constraints, we can employ a hierarchical optimization procedure to solve for robot actions (represented by a sequence of end-effector poses in \(SE(3)\) ) with a perception-action loop at a real-time frequency. Furthermore, in order to circumvent the need for manual specification of ReKep for each new task, we devise an automated procedure that leverages large vision models and vision-language models to produce ReKep from free-form language instructions and RGB-D observations. We present system implementations on a wheeled single-arm platform and a stationary dual-arm platform that can perform a large variety of manipulation tasks, featuring multi-stage, in-the-wild, bimanual, and reactive behaviors, all without task-specific data or environment models. †† *Denotes equal contribution. Correspondence to Wenlong Huang <wenlongh@stanford.edu>.
机器人操作涉及与环境中的物体进行复杂的交互，这些交互通常可以表示为空间和时间域上的约束。考虑图 1 中倒茶的任务：机器人必须抓住手柄，在搬运过程中保持杯子直立，将壶嘴对准目标容器，然后以正确的角度倾斜杯子进行倒水。在这里，约束不仅编码了中间子目标（例如，对准壶嘴），还编码了过渡行为（例如，搬运过程中保持杯子直立），这些共同决定了机器人动作相对于环境的空间、时间和其他组合要求。
Robotic manipulation involves intricate interactions with objects in the environment, which can often be expressed as constraints in both spatial and temporal domains. Consider the task of pouring tea into a cup in Fig. 1: the robot must grasp at the handle, keep the cup upright while transporting it, align the spout with the target container, and then tilt the cup at the correct angle to pour. Here, the constraints encode not only the intermediate sub-goals (e.g., align the spout) but also the transitioning behaviors (e.g., keep the cup upright in transportation), which collectively dictate the spatial, timing, and other combinatorial requirements of the robot’s actions in relation to the environment.
然而，为大量真实世界任务有效地制定这些约束面临着重大挑战。虽然使用机器人与物体之间的相对位姿来表示约束是一种直接且广泛使用的方法，但刚体变换不能描述几何细节，需要先验地获取物体模型，并且无法处理可变形物体。另一方面，数据驱动的方法使得直接在视觉空间中学习约束成为可能。虽然更灵活，但随着约束在物体和任务方面呈组合式增长，如何有效收集训练数据仍不清楚。因此，我们提出一个问题：我们如何表示操作中的约束，使其 1）广泛适用：适用于需要多阶段、野外、双臂和反应性行为的任务；2）可扩展获取：有可能通过基础模型的进步实现完全自动化；3）实时优化：能够通过现成的求解器高效求解，以产生复杂的操作行为？
However, effectively formulating these constraints for a large variety of real-world tasks presents significant challenges. While representing constraints using relative poses between robots and objects is a direct and widely-used approach, rigid-body transformations do not depict geometric details, require obtaining object models a priori, and cannot work on deformable objects. On the other hand, data-driven approaches enable learning constraints directly in visual space. While more flexible, it remains unclear how to effectively collect training data as the number of constraints grows combinatorially in terms of objects and tasks. Therefore, we ask the question: how can we represent constraints in manipulation that are 1) widely applicable: adaptable to tasks that require multi-stage, in-the-wild, bimanual, and reactive behaviors, 2) scalably obtainable: have the potential to be fully automated through the advances in foundation models, and 3) real-time optimizable: can be efficiently solved by off-the-shelf solvers to produce complex manipulation behaviors?
在这项工作中，我们提出了关系关键点约束（ReKep）。具体来说，ReKep 将约束表示为 Python 函数，这些函数将一组关键点映射到数值成本，其中每个关键点是场景中任务特定且语义上有意义的 3D 点。每个函数由对关键点的（可能非线性的）算术运算组成，并编码它们之间期望的“关系”，其中关键点可能属于环境中的不同实体，例如机器人手臂、物体部件和其他智能体。虽然每个关键点仅包含其世界坐标系中的 3D 笛卡尔坐标，但如果强制关键点之间的刚性，多个关键点可以共同指定直线、表面和/或 3D 旋转。我们在顺序操作问题的背景下研究 ReKep，其中每个任务涉及多个具有时空依赖性的阶段（例如，前述示例中的“抓取”、“对准”和“倒水”）。
In this work, we propose Relational Keypoint Constraints (ReKep). Specifically, ReKep represents constraints as Python functions that map a set of keypoints to a numerical cost, where each keypoint is a task-specific and semantically meaningful 3D point in the scene. Each function is composed of (potentially nonlinear) arithmetic operations on the keypoints and encodes a desired “relation” between them, where the keypoints may belong to different entities in the environment, such as the robot arms, object parts, and other agents. While each keypoint only consists of its 3D Cartesian coordinates in the world frame, multiple keypoints can collectively specify lines, surfaces, and/or 3D rotations if rigidity between keypoints is enforced. We study ReKep in the context of the sequential manipulation problem, where each task involves multiple stages that have spatio-temporal dependencies (e.g., “grasping”, “aligning”, and “pouring” in the aforementioned example).
虽然约束通常为每个任务手动定义，但我们证明了 ReKep 的特定形式具有独特优势，即它们可以通过预训练的大型视觉模型（LVM）和视觉语言模型（VLM）实现自动化，从而能够从 RGB-D 观测和自由形式的语言指令中实现 ReKep 的野外规范。具体来说，我们利用 LVM 来提出场景中细粒度且语义上有意义的关键点，并利用 VLM 从叠加了所提出关键点的视觉输入中编写约束作为 Python 函数。这个过程可以解释为将细粒度的空间关系（通常是那些不易用自然语言指定的关系）通过视觉引用表达式在 VLM 支持的输出模态（代码）中进行基础化。
While constraints are typically defined manually per task, we demonstrate the specific form of ReKep possesses a unique advantage in that they can be automated by pre-trained large vision models (LVM) and vision-language models (VLM), enabling in-the-wild specification of ReKep from RGB-D observations and free-form language instructions. Specifically, we leverage LVM to propose fine-grained and semantically meaningful keypoints in the scene and VLM to write the constraints as Python functions from visual input overlaid with proposed keypoints. This process can be interpreted as grounding fine-grained spatial relations, often those not easily specified with natural language, in an output modality supported by VLM (code) using visual referral expressions.
利用生成的约束，可以使用现成的求解器通过基于跟踪的关键点重新评估约束来产生机器人动作。受启发，我们采用分层优化过程，首先求解一组路点作为子目标（表示为\(SE(3)\)末端执行器位姿），然后求解滚动时域控制问题以获得密集的动作序列来实现每个子目标。通过适当的问题实例化，我们证明了对于本工作中考虑的任务，它可以以约 10 Hz 的频率可靠地求解。
With the generated constraints, off-the-shelf solvers can be used to produce robot actions by re-evaluating the constraints based on tracked keypoints. Inspired by, we employ a hierarchical optimization procedure to first solve a set of waypoints as sub-goals (represented as \(SE(3)\) end-effector poses) and then solve the receding-horizon control problem to obtain a dense sequence of actions to achieve each sub-goal. With appropriate instantiation of the problem, we demonstrate that it can be reliably solved at approximately 10 Hz for the tasks considered in this work.
我们的贡献总结如下：1) 我们将操作任务建模为带有关系关键点约束的层次优化问题；2) 我们设计了一个利用大型视觉模型和视觉-语言模型自动指定关键点和约束的流程；3) 我们在两个真实机器人平台上实现了系统，该系统以语言指令和 RGB-D 观测为输入，为各种操作任务生成多阶段、野外、双臂和反应式行为，且无需任务特定数据或环境模型。
Our contributions are summarized as follows: 1) We formulate manipulation tasks as a hierarchical optimization problem with Relational Keypoint Constraints; 2) We devise a pipeline to automatically specify keypoints and constraints using large vision models and vision-language models; 3) We present system implementations on two real-robot platforms that take as input a language instruction and RGB-D observations, and produce multi-stage, in-the-wild, bimanual, and reactive behaviors for a large variety of manipulation tasks, all without task-specific data or environment models.
操作的结构化表示。结构化表示决定了操作系统中不同模块的编排方式，并对系统的能力、假设、效率和有效性产生不同的影响。刚体姿态最为常用，因为自由空间中的刚体运动已被充分理解，且其在建模物体的长距离依赖方面效率较高。然而，由于这通常需要预先对环境的几何和动力学进行建模，许多工作研究了使用数据驱动方法的结构化表示，例如学习以对象为中心的表示、基于粒子的动力学以及关键点或描述符。其中，关键点因其可解释性、效率、对实例变化的泛化能力以及同时建模刚体和可变形物体的能力而显示出巨大潜力。然而，每个任务都需要手动标注，因此在开放世界环境中缺乏可扩展性，这正是我们旨在解决的工作。
Structural Representations for Manipulation. Structural representations determine the orchestration of different modules in a manipulation system and yield different implications on the capabilities, assumptions, efficiency, and effectiveness of the system. Rigid-body poses are most commonly used given well-understood rigid-body motions in free space and their efficiency at modeling long-range dependencies of objects. However, since it often requires both geometry and dynamics of environment to be modeled beforehand, various works have studied structural representations using data-driven methods, such as learning object-centric representation, particle-based dynamics, and keypoints or descriptors. Among them, keypoints have shown great promises given their interpretability, efficiency, generalization to instance variations, and ability to model both rigid bodies and deformable objects. However, manual annotation is required per task, thus lacking scalability in open-world settings, which we aim to address in this work.
操作中的约束优化。约束通常用于给机器人施加期望的行为。运动规划算法使用几何约束来计算避免障碍并实现目标的可行轨迹。接触约束可用于规划有力或接触丰富的行为。对于顺序操作任务，任务和运动规划（TAMP）是一个广泛使用的框架，通常被表述为约束满足问题，并以连续几何问题作为子程序。逻辑几何规划（Logic-Geometric Programming）则在整个状态轨迹上制定一个非线性约束规划，同时考虑逻辑和几何约束。约束可以是手动编写的，也可以以流形、可行性模型或符号距离场的形式从数据中学习。受启发，我们将顺序操作任务表述为一个集成的连续数学规划，并以滚动时域的方式反复求解，关键区别在于约束由基础模型合成。
Constrained Optimization in Manipulation. Constraints are often used to impose desired behaviors on robots. Motion planning algorithms use geometric constraints to compute feasible trajectories that avoid obstacles and achieve goals. Contact constraints can be used to plan forceful or contact-rich behaviors. For sequential manipulation tasks, task and motion planning (TAMP) is a widely used framework, often formulated as constraint satisfaction problems with continuous geometric problems as subroutines. Logic-Geometric Programming alternatively formulates a nonlinear constrained program over the entire state trajectory, taking into account both logical and geometric constraints. Constraints can either be manually written or learned from data in the form of manifolds, feasibility model, or signed-distance fields. Inspired by, we formulate sequential manipulation tasks as an integrated continuous mathematical program that is repeatedly solved in a receding-horizon fashion, with the key difference being that the constraints are synthesized by foundation models.
机器人学中的基础模型。利用基础模型进行机器人学是一个活跃的研究领域。我们请读者参阅相关综述和最新应用。这里，我们专注于能够结合视觉输入进行操作的视觉语言模型（VLM）。然而，尽管视觉语言模型在开放世界规划和目标指定方面显示出潜力，但其基于标题引导的预训练方案往往限制了图像中可保留的视觉细节。另一方面，自监督视觉模型（如 DINO）提供了细粒度的像素级特征，对多种视觉和机器人任务有用，但缺乏解释开放世界语义的有效方法，而这对于跨任务泛化至关重要。在这项工作中，我们利用它们的互补优势：使用 DINOv2 进行细粒度关键点提议，并使用 GPT-4o 在支持的输出模态（代码）中发挥其视觉推理能力。类似形式的视觉提示技术也在并行工作中被探索。在这项工作中，我们展示了 ReKep 具有执行具有挑战性的 6-12 自由度任务、集成高层推理以实现反应式重规划、高频闭环执行以及通过视觉提示生成黑盒约束的独特优势。更多讨论见附录 A.10。
Foundation Models for Robotics. Leveraging foundation models for robotics is an active area of research. We refer readers to for overview and recent applications. Here, we focus on VLMs that are capable of incorporating visual inputs for manipulation. However, despite showing promises for open-world planning and goal specification, the caption-guided pretraining scheme of VLMs often limits visual details that can be retained about the image. Self-supervised vision models (e.g., DINO), on the other hand, provide fine-grained pixel-level features useful for various vision and robotic tasks, but there lack effective ways for interpreting open-world semantics pivotal for cross-task generalization. In this work, we leverage their complementary strengths by using DINOv2 for fine-grained keypoint proposal and using GPT-4o for its visual reasoning capability in a supported output modality (code). Similar forms of visual prompting techniques are also explored in concurrent works. In this work, we demonstrate ReKep possesses the unique advantages of performing challenging 6-12 DoF tasks, integrated high-level reasoning for reactive replanning, high-frequency closed-loop execution, and generating black-box constraints via visual prompting. More discussion in Appendix A.10.
在此我们讨论：(1) 什么是关系关键点约束（第 3.1 节）？(2) 如何利用 ReKep 将操作问题形式化为带约束的优化问题（第 3.2 节）？(3) 我们提出的算法实例如何实时高效地求解该优化问题（第 3.3 节）？(4) 如何从 RGB-D 观测和语言指令中自动获取 ReKep（第 3.4 节）？
Herein we discuss: (1) What are Relational Keypoint Constraints (Sec. 3.1)? (2) How to formulate manipulation as a constrained optimization problem with ReKep (Sec. 3.2)? (3) What is our algorithmic instantiation that can efficiently solve the optimization in real-time (Sec. 3.3)? (4) How to automatically obtain ReKep from RGB-D observations and language instructions (Sec. 3.4)?
在此，我们定义 ReKep 的单个实例。为清晰起见，我们假设已指定一组 \(K\) 个关键点（详见第 3.4 节）。具体而言，每个关键点 \(k_{i}\in\mathbb{R}^{3}\) 指场景表面上具有笛卡尔坐标的三维点，其依赖于任务语义和环境（例如，手柄上的抓取点、壶嘴）。
Herein we define a single instance of ReKep. For clarity, we assume that a set of \(K\) keypoints have been specified (discussed later in Sec. 3.4). Concretely, each keypoint \(k_{i}\in\mathbb{R}^{3}\) refers to a 3D point on the scene surface with Cartesian coordinates, which is dependent on the task semantics and the environment (e.g., grasp point on the handle, spout).
ReKep 的单个实例是一个函数 \(f:\mathbb{R}^{K\times 3}\rightarrow\mathbb{R}\)，它将关键点数组（记为 \(\bm{k}\)）映射到一个无界代价，其中 \(f(\bm{k})\leq 0\) 表示约束满足。函数 \(f\) 实现为无状态 Python 函数，包含对关键点的 NumPy 操作，这些操作可能是非线性和非凸的。本质上，ReKep 的一个实例编码了关键点之间的一种期望空间关系，这些关键点可能属于机械臂、物体部件和其他智能体。
A single instance of ReKep is a function \(f:\mathbb{R}^{K\times 3}\rightarrow\mathbb{R}\) that maps an array of keypoints, denoted as \(\bm{k}\), to an unbounded cost, where \(f(\bm{k})\leq 0\) indicates the constraint is satisfied. The function \(f\) is implemented as a stateless Python function, containing NumPy operations on keypoints, which may be nonlinear and nonconvex. In essence, one instance of ReKep encodes one desired spatial relation between keypoints, which may belong to robot arm(s), object parts, and other agents.
然而，操作任务通常涉及多个空间关系，并且可能有多个时间上依赖的阶段，每个阶段需要不同的空间关系。为此，我们将任务分解为 \(N\) 个阶段，并使用 ReKep 为每个阶段 \(i\in\{1,\ldots,N\}\) 指定两类约束：一组子目标约束 \(\mathcal{C}_{\text{sub-goal}}^{(i)}=\{f_{\text{sub-goal},1}^{(i)}(\bm{k}),\ldots,f_{\text{sub-goal},n}^{(i)}(\bm{k})\}\) 和一组路径约束 \(\mathcal{C}_{\text{path}}^{(i)}=\{f_{\text{path},1}^{(i)}(\bm{k}),\ldots,f_{\text{path},m}^{(i)}(\bm{k})\}\)，其中 \(f_{\text{sub-goal}}^{(i)}\) 编码在阶段 \(i\) 结束时需达到的关键点关系，\(f_{\text{path}}^{(i)}\) 编码在阶段 \(i\) 内每个状态都需满足的关键点关系。考虑图 2 中的倒水任务，它包含三个阶段：抓取、对齐和倒水。阶段 1 的子目标约束将末端执行器拉向茶壶手柄。然后阶段 2 的子目标约束指定茶壶嘴需要在杯口上方。此外，阶段 2 的路径约束确保茶壶在运输过程中保持直立以避免溢出。最后，阶段 3 的子目标约束指定期望的倒水角度。
However, a manipulation task typically involves multiple spatial relations and may have multiple temporally dependent stages where each stage entails different spatial relations. To this end, we decompose a task into \(N\) stages and use ReKep to specify two kinds of constraints for each stage \(i\in\{1,\ldots,N\}\): a set of sub-goal constraints \(\mathcal{C}_{\text{sub-goal}}^{(i)}=\{f_{\text{sub-goal},1}^{(i)}(\bm{k}),\ldots,f_{\text{sub-goal},n}^{(i)}(\bm{k})\}\) and a set of path constraints \(\mathcal{C}_{\text{path}}^{(i)}=\{f_{\text{path},1}^{(i)}(\bm{k}),\ldots,f_{\text{path},m}^{(i)}(\bm{k})\}\), where \(f_{\text{sub-goal}}^{(i)}\) encodes one keypoint relation to be achieved at the end of stage \(i\), and \(f_{\text{path}}^{(i)}\) encodes one keypoint relation to be satisfied for every state within stage \(i\). Consider the pouring task in Fig. 2, which consists of three stages: grasp, align, and pour. The stage-1 sub-goal constraint pulls the end-effector towards the teapot handle. Then stage-2 sub-goal constraint specifies that the teapot spout needs to be on top of the cup opening. Additionally, stage-2 path constraint ensures the teapot stays upright to avoid spillage when transported. Finally, the stage-3 sub-goal constraint specifies the desired pouring angle.
利用 ReKep 作为表示约束的通用工具，我们采用文献[引用]中的公式，并展示如何将操作任务表述为涉及 \(\mathcal{C}_{\text{sub-goal}}^{(i)}\) 和 \(\mathcal{C}_{\text{path}}^{(i)}\) 的约束优化问题。我们将末端执行器位姿记为 \(\mathbf{e}\in SE(3)\)。为了执行操作任务，我们旨在通过将控制问题表述如下来获得整体离散时间轨迹 \(\mathbf{e}_{1:T}\)：
Using ReKep as a general tool to represent constraints, we adopt the formulation in [reference] and show how a manipulation task can be formulated as a constrained optimization problem involving \(\mathcal{C}_{\text{sub-goal}}^{(i)}\) and \(\mathcal{C}_{\text{path}}^{(i)}\). We denote the end-effector pose as \(\mathbf{e}\in SE(3)\). To perform the manipulation task, we aim to obtain the overall discrete-time trajectory \(\mathbf{e}_{1:T}\) by formulating the control problem as follows:
其中 \(\mathbf{e}_{t}\) 表示时间 \(t\) 时的末端执行器位姿，\(g_{i}\in\{1,\ldots,T\}\) 是从阶段 \(i\) 到 \(i+1\) 的过渡时间，也是辅助决策变量，\(\bm{k}_{t}\) 是时间 \(t\) 时关键点位置的数组，\(h\) 是关键点的前向模型，\(\lambda_{\text{sub-goal}}^{(i)}\) 和 \(\lambda_{\text{path}}^{(i)}\) 分别是子目标和路径问题的辅助成本函数（例如，碰撞避免）。即，对于每个阶段 \(i\)，优化应找到一个末端执行器位姿作为下一个子目标，以及其时间，并找到一个位姿序列 \(\mathbf{e}_{g_{i-1}:g_{i}}\) 来实现该子目标，同时满足给定的 ReKep 约束和辅助成本。该公式可视为轨迹优化中的直接打靶法。
where \(\mathbf{e}_{t}\) denotes the end-effector pose at time \(t\), \(g_{i}\in\{1,\ldots,T\}\) are the timings of the transition from stage \(i\) to \(i+1\) which are also auxiliary decision variables, \(\bm{k}_{t}\) is the array of keypoint positions at time \(t\), \(h\) is a forward model of keypoints, and \(\lambda_{\text{sub-goal}}^{(i)}\) and \(\lambda_{\text{path}}^{(i)}\) are auxiliary cost functions (e.g., collision avoidance) for the sub-goal and path problems respectively. Namely, for each stage \(i\), the optimization shall find an end-effector pose as next sub-goal, along with its timing, and a sequence of poses \(\mathbf{e}_{g_{i-1}:g_{i}}\) that achieves the sub-goal, subject to the given set of ReKep constraints and auxiliary costs. This formulation can be considered as direct shooting in trajectory optimization.
为了实时求解公式 1，我们采用对完整问题的分解，仅优化紧邻的下一个子目标以及到达该子目标的对应路径（伪代码见算法 1）。所有优化问题均使用 SciPy 实现和求解，决策变量归一化到 \([0,1]\)。初始使用 Dual Annealing 并以 SLSQP 作为局部优化器求解（约 1 秒），随后基于先前解仅使用局部优化器以约 10 Hz 的频率求解。
To solve Eq. 1 in real-time, we employ a decomposition of the full problem and only optimize for the immediate next sub-goal and the corresponding path to reach the sub-goal (pseudo-code in Algorithm 1). All optimization problems are implemented and solved using SciPy with decision variables normalized to \([0,1]\). They are initially solved with Dual Annealing with SLSQP as local optimizer (around 1 second) and subsequently solved with only local optimizer based on the previous solution at approximately 10 Hz.
子目标问题：我们首先求解子目标问题，以获得当前阶段 \(i\) 的 \(\mathbf{e}_{g_{i}}\)：
The Sub-Goal Problem: We first solve the sub-goal problem to obtain \(\mathbf{e}_{g_{i}}\) for the current stage \(i\):
其中 \(\lambda_{\text{sub-goal}}\) 包含辅助控制成本：场景碰撞避免、可达性、位姿正则化、解一致性以及双臂设置下的自碰撞（详见 A.8）。即，公式 2 试图在最小化辅助成本的同时，找到一个满足 \(\mathcal{C}_{\text{sub-goal}}^{i}\) 的子目标。如果某个阶段涉及抓取，则还会包含抓取度量。在本工作中，我们使用 AnyGrasp 11 1 由于 AnyGrasp 是抓取检测器而非度量，且在优化循环中计算成本高昂，我们总是利用与抓取相关的 ReKep 总是将末端执行器上的虚拟关键点与一个实际关键点关联的事实，返回最接近指定“抓取关键点”的抓取。
where \(\lambda_{\text{sub-goal}}\) subsumes auxiliary control costs: scene collision avoidance, reachability, pose regularization, solution consistency, and self-collision for bimanual setup (details in A.8). Namely, Eq. 2 attempts to find a sub-goal that satisfies \(\mathcal{C}_{\text{sub-goal}}^{i}\) while minimizing the auxiliary costs. If a stage is concerned with grasping, a grasp metric is also included. In this work, we use AnyGrasp 11 1 Since AnyGrasp is a grasp detector instead of a metric and is computationally expensive to run in optimization loops, we always return the grasp closest to a specified “grasp keypoint” by exploiting the fact that ReKep related to grasping always associates a dummy keypoint on the end-effector and one actual keypoint..
路径问题：获得子目标 \(\mathbf{e}_{g_{i}}\) 后，我们求解从当前末端执行器位姿 \(\mathbf{e}_{t}\) 到子目标 \(\mathbf{e}_{g_{i}}\) 的轨迹 \(\mathbf{e}_{t:g_{i}}\)：
The Path Problem: After obtaining sub-goal \(\mathbf{e}_{g_{i}}\), we solve for a trajectory \(\mathbf{e}_{t:g_{i}}\) starting from current end-effector pose \(\mathbf{e}_{t}\) to the sub-goal \(\mathbf{e}_{g_{i}}\):
其中 \(\lambda_{\text{path}}\) 包含以下辅助控制成本：场景碰撞避免、可达性、路径长度、解一致性以及双臂设置下的自碰撞（详见 A.9）。如果到子目标 \(\mathbf{e}_{g_{i}}\) 的距离在较小容差 \(\epsilon\) 内，则进入下一阶段 \(i+1\)。
where \(\lambda_{\text{path}}\) subsumes the following auxiliary control costs: scene collision avoidance, reachability, path length, solution consistency, and self-collision in the case of bimanual setup (details in A.9). If the distance to the sub-goal \(\mathbf{e}_{g_{i}}\) is within a small tolerance \(\epsilon\), we progress to the next stage \(i+1\).
回溯：尽管子问题可以以实时频率求解，以在阶段内响应外部干扰，但如果上一阶段的任何子目标约束不再成立（例如，倒水任务中杯子从夹爪中取出），系统必须能够在阶段之间重新规划。具体来说，在每个控制循环中，我们检查 \(\mathcal{C}_{\text{path}}^{(i)}\) 是否被违反。如果发现违反，我们迭代地回溯到前一个阶段 \(j\)，使得 \(\mathcal{C}_{\text{path}}^{(j)}\) 得到满足。
Backtracking: Although the sub-problems can be solved at a real-time frequency to react to external disturbances within a stage, it is imperative that the system can replan across stages if any sub-goal constraint from the last stage no longer holds (e.g., cup taken out of the gripper in the pouring task). Specifically, in every control loop, we check for violation of \(\mathcal{C}_{\text{path}}^{(i)}\) . If one is found, we iteratively backtrack to a previous stage \(j\) such that \(\mathcal{C}_{\text{path}}^{(j)}\) is satisfied.
关键点的前向模型：为了求解式(2)和式(3)，必须在优化过程中利用前向模型 \(h\)，从 \(\Delta\mathbf{e}\) 估计 \(\Delta\bm{k}\)。与先前工作一样，我们在末端执行器和“被抓取关键点”（属于同一物体或部件的刚性关键点组；由第 3.4 节所述的分割模型获得）之间做出刚性假设。即，给定末端执行器位姿变化 \(\Delta\mathbf{e}\)，我们可以通过施加相同的相对刚体变换 \(\bm{k}^{\prime}[\text{grasped}]=\mathbf{T}_{\Delta\mathbf{e}}\cdot\bm{k}[\text{grasped}]\) 来计算关键点位置的变化，同时假设其他关键点保持静止。我们注意到这是一个“局部”假设，即仅假设在问题求解的短时间内（0.1 秒）成立。实际关键点位置通过视觉输入以 20 Hz 的频率跟踪，并在每个新问题中使用。对于更具挑战性的场景（例如动态或接触丰富的任务），可以使用学习模型或基于物理的模型。
Forward Models for Keypoints: To solve Eq. 2 and Eq. 3, one must utilize a forward model \(h\) that estimates \(\Delta\bm{k}\) from \(\Delta\mathbf{e}\) in the optimization process. As in prior work , we make the rigidity assumption between the end-effector and the “grasped keypoints” (a rigid group of keypoints that belong to the same object or part; obtained from the segmentation model as described in Sec. 3.4.). Namely, given a change in the end-effector pose \(\Delta\mathbf{e}\) , we can calculate the change in keypoint positions by applying the same relative rigid transformation \(\bm{k}^{\prime}[\text{grasped}]=\mathbf{T}_{\Delta\mathbf{e}}\cdot\bm{k}[\text{grasped}]\) , while assuming other keypoints stay static. We note that this is a “local” assumption in that it is only assumed to hold for the short duration (0.1s) that the problem is solved. Actual keypoint positions are tracked using visual input at 20 Hz and used in every new problem. For more challenging scenarios (e.g., dynamic or contact-rich tasks), a learned or physics-based model may be used.
为了使系统能够根据自由形式的任务指令在真实场景中执行任务，我们设计了一个利用大型视觉模型和视觉-语言模型进行关键点提议和 ReKep 生成的流程，下面分别进行讨论：
To enable the system to perform tasks in-the-wild given a free-form task instruction, we devise a pipeline using large vision models and vision-language models for keypoint proposal and ReKep generation, which are respectively discussed as follows:
关键点提议：给定一张 RGB 图像 \(\mathbb{R}^{h\times w\times 3}\) ，我们首先从 DINOv2 中提取分块特征 \(\mathbf{F}_{\text{patch}}\in\mathbb{R}^{h^{\prime}\times w^{\prime}\times d}\) 。然后进行双线性插值，将特征上采样到原始图像尺寸，得到 \(\mathbf{F}_{\text{interp}}\in\mathbb{R}^{h\times w\times d}\) 。为了确保提议覆盖场景中所有相关物体，我们使用 Segment Anything (SAM)提取场景中的所有掩码 \(\mathbf{M}=\{\mathbf{m}_{1},\mathbf{m}_{2},\ldots,\mathbf{m}_{n}\}\) 。对于每个掩码 \(j\) ，我们使用 \(k\) -均值（\(k=5\)）和余弦相似度度量对掩码特征 \(\mathbf{F}_{\text{interp}}[\mathbf{m}_{j}]\) 进行聚类。聚类中心作为关键点候选，并通过标定的 RGB-D 相机投影到世界坐标 \(\mathbb{R}^{3}\) 。距离其他候选点小于 \(8\) 厘米的候选点被过滤掉。总的来说，我们发现该过程能够很好地识别出大部分细粒度且语义上有意义的物体区域。
Keypoint Proposal: Given an RGB image \(\mathbb{R}^{h\times w\times 3}\) , we first extract the patch-wise features \(\mathbf{F}_{\text{patch}}\in\mathbb{R}^{h^{\prime}\times w^{\prime}\times d}\) from DINOv2 . Then we perform bilinear interpolation to upsample the features to the original image size, \(\mathbf{F}_{\text{interp}}\in\mathbb{R}^{h\times w\times d}\) . To ensure the proposal covers all relevant objects in the scene, we extract all masks \(\mathbf{M}=\{\mathbf{m}_{1},\mathbf{m}_{2},\ldots,\mathbf{m}_{n}\}\) in the scene using Segment Anything (SAM) . For each mask \(j\) , we cluster the masked features \(\mathbf{F}_{\text{interp}}[\mathbf{m}_{j}]\) using \(k\) -means with \(k=5\) with a cosine-similarity metric. The centroids of the clusters are used as keypoint candidates, which are projected to a world coordinate \(\mathbb{R}^{3}\) using a calibrated RGB-D camera. Candidates that are within \(8\) cm of others are filtered out. Overall, we find that this procedure is adept at identifying a large percentage of fine-grained and semantically meaningful regions of objects.
ReKep 生成：获得关键点候选后，我们将其以数字标记叠加在原始 RGB 图像上。结合任务的语言指令，我们使用视觉提示查询 GPT-4o，以生成所需的阶段数以及每个阶段 \(i\) 对应的子目标约束 \(\mathcal{C}_{\text{sub-goal}}^{(i)}\) 和路径约束 \(\mathcal{C}_{\text{path}}^{(i)}\) （提示见 A.6）。值得注意的是，这些函数并不直接操作关键点位置的数值。相反，我们利用 VLM 的优势，将空间关系指定为算术运算，例如关键点之间的 L2 距离或点积，这些运算仅在调用由专门 3D 跟踪器跟踪的实际关键点位置时才被实例化。此外，对一组关键点位置使用算术运算的一个重要优势是，当提供足够的点并强制相关点之间的刚性时，它可以指定完整的 \(SO(3)\) 三维旋转，但这仅在任务语义需要时才进行。这使得 VLM 能够在 3D 笛卡尔空间中通过算术运算推理 3D 旋转，从而有效避免了处理替代 3D 旋转表示以及进行数值计算的需要。
ReKep Generation: After obtaining the keypoint candidates, we overlay them on the original RGB image with numerical marks. Coupled with the language instruction of the task, we then use visual prompting to query GPT-4o to generate the number of required stages and the corresponding sub-goal constraints \(\mathcal{C}_{\text{sub-goal}}^{(i)}\) and path constraints \(\mathcal{C}_{\text{path}}^{(i)}\) for each stage \(i\) (prompts are in A.6). Notably, the functions do not directly manipulate the numerical values of the keypoint positions. Rather, we exploit the strength of VLM to specify spatial relations as arithmetic operations, such as L2 distance or dot product between keypoints, that are only instantiated when invoked with actual keypoint positions tracked by a specialized 3D tracker. Furthermore, an important advantage of using arithmetic operations on a set of keypoint positions is that it can specify 3D rotations in full \(SO(3)\) when sufficient points are provided and rigidity between relevant points is enforced, but this is done only when needed depending on task semantic. This enables VLM to reason about 3D rotations with arithmetic operations in 3D Cartesian space, effectively circumventing the need for dealing with alternative 3D rotation representation and the need for performing numerical computation.
我们旨在回答以下研究问题：（1）我们的框架在自动制定和合成操作行为方面表现如何（第 4.1 节）？（2）我们的系统能否泛化到新物体和操作策略（第 4.2 节）？（3）各个组件如何导致系统的失败案例（第 4.3 节）？我们在两个真实机器人平台上验证了 ReKep：一个轮式单臂平台和一个固定式双臂平台（图 3）。更多实现细节见附录，包括关键点提议（A.5）、VLM 查询（A.6）、点跟踪器（A.7）、子目标求解器（A.8）和路径求解器（A.9）。
We aim to answer the following research questions: (1) How well does our framework automatically formulate and synthesize manipulation behaviors (Sec. 4.1)? (2) Can our system generalize to novel objects and manipulation strategies (Sec. 4.2)? (3) How do the individual components contribute to the failure cases of the system (Sec. 4.3)? We validate ReKep on two real robot platforms: a wheeled single-arm platform, and a stationary dual-arm platform (Figure. 3). Additional implementation details can be found in Appendix, including keypoint proposal (A.5), VLM querying (A.6), point trackers (A.7), sub-goal solver (A.8), and path solver (A.9).
任务。我们特意选择了一组任务（如图 3 所示），旨在检验系统的多阶段（m）、野外（w）、双臂（b）和反应式（r）行为。任务及其特征包括：倒茶（m, w, r）、收纳书籍（w）、回收易拉罐（w）、封箱（w, r）、折叠衣物（b）、打包鞋子（b）以及协作折叠（b, r）。我们进一步在外部干扰下评估其中三个任务（记为“Dist.”），方法是在执行过程中改变任务物体的姿态。
Tasks. We purposefully select a set of tasks (shown in Fig. 3) with the goal of examining the multi-stage (m), in-the-wild (w), bimanual (b), and reactive (r) behaviors of the system. The tasks and their features are Pour Tea (m, w, r), Stow Book (w), Recycle Can (w), Tape Box (w, r), Fold Garment (b), Pack Shoes (b), and Collaborative Folding (b, r). We further evaluate three of the tasks under external disturbances (denoted as “Dist.”) by changing poses of task objects during execution.
指标与基线。每种设置进行 10 次试验，其中物体姿态随机化。成功率报告在表 1 中。我们以 VoxPoser 作为基线进行比较。我们评估了系统的两种变体：“自动”使用基础模型自动生成 ReKep，“标注”使用人工标注的 ReKep。
Metric and Baselines. Each setting has 10 trials, in which object poses are randomized. Success rate is reported in Tab. 1. We compare to VoxPoser as a baseline. We evaluate two variants of the system: “Auto” uses foundation models to automatically generate ReKep, and “Annotated (Annot.)” uses human-annotated ReKep.
结果。与基线相比，ReKep 能够有效处理每个任务的核心挑战。例如，它能在多阶段任务中制定正确的时间依赖关系（例如，倒水前壶嘴需与杯子对齐），利用常识知识（例如，可乐罐应回收），并在双臂设置（例如，同时折叠左右袖子）和人机协作设置（例如，与人一起对齐四个角来折叠大毯子）中构建协调行为。结合优化框架，它还能在收纳书籍任务中生成受限空间内具有运动学挑战性的行为，并在打包鞋子任务中找到在狭小体积内紧密放置两只鞋的可行方案。由于关键点以高频跟踪，系统能够对外部干扰做出反应，并在阶段内和跨阶段进行重新规划。尽管结果令人鼓舞，我们也发现了若干局限性，将在第 5 节中讨论。
Results. Compared to baselines, ReKep can effectively handle core challenges of each task. For example, it can formulate correct temporal dependency in multi-stage tasks (e.g., spout needs to be aligned with the cup before pouring), leverage commonsense knowledge (e.g., coke cans should be recycled), and construct coordination behaviors in both bimanual settings (e.g., folding left sleeve and right sleeve simultaneously) and human-robot collaboration setting (e.g., folding a large blanket by aligning the four corners together with human). Coupled with an optimization framework, it can also generate kinematically challenging behaviors in confined spaces in the Stow Book task and find a feasible solution that densely fits two shoes within a small volume in the Pack Shoes task. Since the keypoints are tracked at a high frequency, the system can react to external disturbances and replan both within stage and across stages. Despite promising results, we also identify several limitations which are discussed in Sec. 5.
任务。我们系统性地评估了如何通过聚焦于单一任务——衣物折叠——来制定新颖的操作策略，但涉及 8 个独特的衣物类别，每个类别都需要独特的折叠方式，并同时要求几何推理和常识推理。评估在双臂平台上进行，这给双臂协调带来了额外的挑战。
Tasks. We systematically evaluate how novel manipulation strategies can be formulated by focusing on a single task, garment folding, but with 8 unique categories of garments, each demanding a unique way of folding and requiring both geometrical and commonsense reasoning. Evaluation is done on the bimanual platform, presenting additional challenges in bimanual coordination.
指标。我们使用 GPT-4o，其提示仅包含通用指令，没有上下文示例。“策略成功率”衡量生成的 ReKep 是否可行，这同时测试了关键点提议模块和 VLM；“执行成功率”衡量在给定可行策略下系统对每件衣物的成功率。每项指标均通过 10 次试验测量。
Metric. We use GPT-4o with a prompt containing only generic instructions with no in-context examples. “Strategy Success” measures whether generated ReKep is feasible, which tests both the keypoint proposal module and the VLM, and “Execution Success” measures system success rate given feasible strategies for each clothing. Each is measured with 10 trials.
结果。有趣的是，我们观察到不同类别之间的策略差异很大，其中许多策略与人类折叠衣物的方式一致。例如，系统能够识别出在完全折叠衣物之前，通常先将两个袖子折叠在一起。在不需要使用双臂的情况下，类似于人类折叠衣物的方式，系统仅使用一只手臂。然而，我们确实观察到 VLM 可能会遗漏操作者期望的某些完成折叠的步骤，但我们认识到这本质上是一个开放性问题，往往取决于个人偏好。
Results. Interestingly, we observe drastically different strategies across categories, many of which are aligned with how humans might fold each garment. For example, it can recognize that two sleeves often are folded together, prior to fully folding the clothes. In cases where using two arms is unnecessary, akin to how humans fold clothes, only one arm is being used. However, we do observe that the VLM may miss certain steps to complete the folding as the operator expected, but we recognize that this is inherently an open-ended problem often based on one’s preferences.
该框架的模块化设计因其可解释性而具有分析系统错误的优势。在本节中，我们通过手动检查表 1 中报告的实验失败案例进行实证研究，并据此计算每个模块导致错误的可能性，同时考虑它们在流水线中的时间依赖性。结果如图 4 所示。在不同模块中，点跟踪器产生的错误占比最大，因为频繁且间歇性的遮挡对准确跟踪构成了重大挑战。关键点提议和 VLM 也产生了相当比例的错误，常见情况包括提议模块遗漏某些关键点以及 VLM 引用错误的关键点。相比之下，优化模块尽管时间预算有限，但对失败的贡献并不大，因为每个问题通常存在许多可能的解决方案。其他模块，如分割、3D 重建和低级控制器，也对某些失败案例有所贡献，但与其他模块相比相对不显著。
The modular design of the framework entails an advantage for analyzing system errors due to its interpretability. In this section, we perform an empirical investigation by manually inspecting the failure cases of the experiments reported in Tab. 1, which is then used to calculate the likelihood of a module causing an error while accounting for their temporal dependencies in the pipeline. Results are reported in Fig. 4. Among the different modules, the point tracker produces the largest portion of errors, as frequent and intermittent occlusion poses significant challenges for accurate tracking. Keypoint proposal and VLM also produce considerable portions of errors, where common cases include the proposal module missing certain keypoints and the VLM referring to incorrect keypoints. The optimization module, on the other hand, does not contribute as much to the failures despite given limited time budget, since there often exist many possible solutions for each problem. Other modules, such as segmentation, 3D reconstruction, and low-level controller, also contribute to some failure cases, but they are relatively insignificant compared to other modules.
在本工作中，我们提出了关系关键点约束（ReKep），一种基于约束的结构化任务表示，它作用于语义关键点，以指定机器人手臂、物体（部件）以及环境中其他智能体之间的期望关系。结合点跟踪器，我们证明了 ReKep 约束可以在分层优化框架中被反复且高效地求解，从而充当以实时频率运行的闭环策略。我们还展示了 ReKep 的独特优势：它可以由大型视觉模型和视觉语言模型自动合成。我们在两个机器人平台以及多种任务上展示了结果，这些任务具有多阶段、野外、双臂和反应性行为的特点，且均无需特定任务数据、额外训练或环境模型。尽管前景广阔，但仍存在若干局限性。首先，优化框架依赖于基于刚性假设的关键点前向模型，尽管高频反馈回路放宽了模型的精度要求。其次，ReKep 依赖精确的点跟踪来在闭环中正确优化动作，而由于严重的间歇性遮挡，这本身就是一个具有挑战性的 3D 视觉任务。最后，当前的公式假设每个任务具有固定的阶段序列（即骨架）。使用不同骨架进行重新规划需要高频运行关键点提议和 VLM，这带来了相当大的计算挑战。关于局限性的更详细讨论见附录 A.11。
In this work, we presented Relational Keypoint Constraints (ReKep), a structural task representation using constraints that operates on semantic keypoints to specify desired relations between robot arms, object (parts), and other agents in the environment. Coupled with point trackers, we demonstrate that ReKep constraints can be repeatedly and efficiently solved in a hierarchical optimization framework to act as a closed-loop policy that runs at a real-time frequency. We also demonstrate the unique advantage of ReKep in that it can be automatically synthesized by large vision models and vision-language models. Results are shown on two robot platforms and on a variety of tasks featuring multi-stage, in-the-wild, bimanual, and reactive behaviors, all without task-specific data, additional training, or environment models. Despite the promises, several limitations remained. First, the optimization framework relies on a forward model of keypoints based on rigidity assumption, albeit a high-frequency feedback loop that relaxes the accuracy requirement of the model. Second, ReKep relies on accurate point tracking to correctly optimize actions in closed-loop, which is itself a challenging 3D vision task due to heavy intermittent occlusions. Lastly, the current formulation assumes a fixed sequence of stages (i.e., skeletons) for each task. Replanning with different skeletons requires running keypoint proposal and VLM at a high-frequency, which poses considerable computational challenges. An extended discussion of limitations can be found in Appendix A.11.