对于 3D 物体操作，构建显式 3D 表示的方法比仅依赖相机图像的方法表现更好。但使用体素等显式 3D 表示会带来巨大的计算成本，影响可扩展性。在这项工作中，我们提出了 RVT，一种用于 3D 操作的多视角变换器，兼具可扩展性和准确性。RVT 的关键特性包括一种跨视角聚合信息的注意力机制，以及从机器人工作空间周围的虚拟视角重新渲染相机输入。在仿真中，我们发现单个 RVT 模型在 18 个 RLBench 任务（含 249 个任务变体）上表现良好，相对成功率比现有最先进方法（PerAct）高出 26%。在达到相同性能的情况下，其训练速度比 PerAct 快 36 倍，推理速度是 PerAct 的 2.3 倍。此外，RVT 在现实世界中仅需每个任务少量（约 10 个）演示即可执行多种操作任务。视觉结果、代码和训练模型可在 https://robotic-view-transformer.github.io/获取。
For 3D object manipulation, methods that build an explicit 3D representation perform better than those relying only on camera images. But using explicit 3D representations like voxels comes at large computing cost, adversely affecting scalability. In this work, we propose RVT, a multi-view transformer for 3D manipulation that is both scalable and accurate. Some key features of RVT are an attention mechanism to aggregate information across views and re-rendering of the camera input from virtual views around the robot workspace. In simulations, we find that a single RVT model works well across 18 RLBench tasks with 249 task variations, achieving 26% higher relative success than the existing state-of-the-art method (PerAct). It also trains 36X faster than PerAct for achieving the same performance and achieves 2.3X the inference speed of PerAct. Further, RVT can perform a variety of manipulation tasks in the real world with just a few ($\sim$10) demonstrations per task. Visual results, code, and trained model are provided at https://robotic-view-transformer.github.io/.
核心贡献 · Key contributions
提出 RVT，一种用于 3D 物体操作的多视角 Transformer，兼具可扩展性和准确性。 Proposes RVT, a multi-view transformer for 3D object manipulation that is both scalable and accurate.
在 18 个 RLBench 任务（249 种变体）上，相对成功率比 PerAct 高 26%。 Achieves 26% higher relative success than PerAct on 18 RLBench tasks with 249 variations.
训练速度比 PerAct 快 36 倍，推理速度快 2.3 倍。 Trains 36X faster and achieves 2.3X inference speed compared to PerAct.
在真实世界中，每个任务仅需约 10 次演示即可完成操作。 Demonstrates real-world manipulation with only ~10 demonstrations per task.
引入从虚拟视角重新渲染，将传感器图像与 Transformer 输入解耦。 Introduces re-rendering from virtual views to decouple sensor images from transformer input.
研究了设计选择，例如在联合注意力之前单独处理图像令牌。 Investigates design choices like separate processing of image tokens before joint attention.
局限 · Limitations
需要标定相机到机器人的外参，限制了可移植性。 Requires calibration of camera-to-robot extrinsics, limiting portability.
在稀疏或噪声点云（如标记任务）上表现不佳。 Struggles with sparse or noisy point clouds, e.g., marker tasks.
视角规格是手动选择的，未从数据中优化或学习。 View specification is manually chosen and not optimized or learned from data.
仅在 5 个真实世界任务（51 次演示）的有限集合上评估。 Evaluated only on a limited set of 5 real-world tasks with 51 demonstrations.
假设存在低级运动规划器和控制器来执行动作。 Assumes a low-level motion planner and controller for action execution.