PaLM-E: An Embodied Multimodal Language Model
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→大型语言模型在广泛复杂任务中表现出色，但在现实世界（如机器人问题）中实现通用推理面临落地挑战。我们提出具身语言模型，直接将真实世界的连续传感器模态纳入语言模型，从而建立词语与感知之间的联系。我们模型的输入是多模态句子，交织视觉、连续状态估计和文本输入编码。我们端到端地训练这些编码，与预训练的大型语言模型结合，用于多种具身任务，包括顺序机器人操作规划、视觉问答和图像描述。我们的评估表明，PaLM-E，一个单一的具身多模态大模型，能够处理多种具身推理任务，来自多种观察模态，在多个实体上，并进一步展现出正迁移：模型受益于跨互联网规模语言、视觉和视觉-语言领域的多样化联合训练。我们最大的模型 PaLM-E-562B，拥有 562B 参数，除了在机器人任务上训练外，还是一个视觉-语言通才，在 OK-VQA 上达到最先进性能，并随着规模扩大保持通才语言能力。
Large language models excel at a wide range of complex tasks. However, enabling general inference in the real world, e.g., for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks including sequential robotic manipulation planning, visual question answering, and captioning. Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. Our largest model, PaLM-E-562B with 562B parameters, in addition to being trained on robotics tasks, is a visual-language generalist with state-of-the-art performance on OK-VQA, and retains generalist language capabilities with increasing scale.
marginparsep 已被修改。topmargin 已被修改。marginparpush 已被修改。
marginparsep has been altered. topmargin has been altered. marginparpush has been altered.
页面布局违反了 ICML 样式。
The page layout violates the ICML style.
请不要更改页面布局，或包含诸如 geometry、savetrees 或 fullpage 之类的宏包，它们会为您更改布局。
Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you.
我们无法可靠地撤销对样式的任意更改。请移除有问题的宏包或更改布局的命令，然后重试。
We’re not able to reliably undo arbitrary changes to the style. Please remove the offending package(s), or layout-changing commands and try again.
PaLM-E：一种具身多模态语言模型
PaLM-E: An Embodied Multimodal Language Model
Danny Driess 1 2 Fei Xia 1 Mehdi S. M. Sajjadi 3 Corey Lynch 1 Aakanksha Chowdhery 3
Danny Driess 1 2 Fei Xia 1 Mehdi S. M. Sajjadi 3 Corey Lynch 1 Aakanksha Chowdhery 3
Brian Ichter 1 Ayzaan Wahid 1 Jonathan Tompson 1 Quan Vuong 1 Tianhe Yu 1 Wenlong Huang 1 Yevgen Chebotar 1 Pierre Sermanet 1 Daniel Duckworth 3 Sergey Levine 1 Vincent Vanhoucke 1 Karol Hausman 1 Marc Toussaint 2 Klaus Greff 3 Andy Zeng 1 Igor Mordatch 3 Pete Florence 1
Brian Ichter 1 Ayzaan Wahid 1 Jonathan Tompson 1 Quan Vuong 1 Tianhe Yu 1 Wenlong Huang 1 Yevgen Chebotar 1 Pierre Sermanet 1 Daniel Duckworth 3 Sergey Levine 1 Vincent Vanhoucke 1 Karol Hausman 1 Marc Toussaint 2 Klaus Greff 3 Andy Zeng 1 Igor Mordatch 3 Pete Florence 1
1 Google 机器人技术 2 柏林工业大学 3 Google Research
1 Robotics at Google 2 TU Berlin 3 Google Research
大型语言模型已被证明能够执行复杂任务。然而，在现实世界中实现通用推理（例如机器人问题）带来了接地（grounding）的挑战。我们提出具身语言模型（embodied language models），直接将现实世界的连续传感器模态纳入语言模型，从而建立词语与感知之间的联系。我们具身语言模型的输入是多模态句子，交织了视觉、连续状态估计和文本输入编码。我们与预训练的大型语言模型一起端到端地训练这些编码，用于多种具身任务，包括顺序机器人操作规划、视觉问答和图像描述。我们的评估表明，PaLM-E，一个单一的大型具身多模态模型，能够处理多种具身推理任务，涵盖多种观察模态和多种具身形态，并且展现出正迁移：该模型受益于跨互联网规模的语言、视觉和视觉-语言领域的多样化联合训练。我们最大的模型 PaLM-E-562B 拥有 562B 参数，除了在机器人任务上训练外，还是一个视觉-语言通才，在 OK-VQA 上达到最先进性能，并随着规模增大保持通才语言能力。
Large language models have been demonstrated to perform complex tasks. However, enabling general inference in the real world, e.g. for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks including sequential robotic manipulation planning, visual question answering, and captioning. Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. Our largest model, PaLM-E-562B with 562B parameters, in addition to being trained on robotics tasks, is a visual-language generalist with state-of-the-art performance on OK-VQA, and retains generalist language capabilities with increasing scale.
大型语言模型（LLM）在多个领域展现出强大的推理能力，包括对话、逐步推理、数学问题求解和代码编写。然而，这类模型在现实世界推理中的一个局限是接地（grounding）问题：虽然在大量文本数据上训练 LLM 可能产生与物理世界相关的表示，但将这些表示与真实世界的视觉和物理传感器模态连接起来，对于解决计算机视觉和机器人领域中更广泛的接地现实问题至关重要。先前的工作将 LLM 的输出与学习到的机器人策略和可供性函数接口以做出决策，但局限性在于 LLM 本身仅接收文本输入，这对于许多场景几何配置重要的任务来说是不够的。此外，在我们的实验中，我们表明当前最先进的视觉语言模型在典型视觉语言任务（如视觉问答（VQA））上训练后，无法直接解决机器人推理任务。
Large language models (LLMs) demonstrate strong reasoning capabilities across various domains, including dialogue, step-by-step reasoning, math problem solving, and code writing. However, a limitation of such models for inference in the real world is the issue of grounding: while training LLMs on massive textual data may lead to representations that relate to our physical world, connecting those representations to real-world visual and physical sensor modalities is essential to solving a wider range of grounded real-world problems in computer vision and robotics. Previous work interfaces the output of LLMs with learned robotic policies and affordance functions to make decisions, but is limited in that the LLM itself is only provided with textual input, which is insufficient for many tasks where the geometric configuration of the scene is important. Further, in our experiments we show that current state-of-the-art visual-language models trained on typical vision-language tasks such as visual-question-answering (VQA) cannot directly solve robotic reasoning tasks.
在本文中，我们提出了具身语言模型，它直接整合来自具身智能体传感器模态的连续输入，从而使语言模型本身能够在现实世界中进行更接地的顺序决策推理。图像和状态估计等输入被嵌入到与语言标记相同的潜在嵌入中，并由基于 Transformer 的 LLM 的自注意力层以与文本相同的方式处理。我们从预训练的 LLM 开始，通过编码器注入连续输入。这些编码器经过端到端训练，以自然文本形式输出顺序决策，这些决策可由具身智能体通过条件化低级策略来解释，或回答具身问题。我们在多种设置下评估了该方法，比较了不同的输入表示（例如，视觉输入的标准与对象中心 ViT 编码）、在训练编码器时冻结与微调语言模型，并研究了多任务协同训练是否能够实现迁移。
In this paper we propose embodied language models, which directly incorporate continuous inputs from sensor modalities of an embodied agent and thereby enable the language model itself to make more grounded inferences for sequential decision making in the real world. Inputs such as images and state estimates are embedded into the same latent embedding as language tokens and processed by the self-attention layers of a Transformer-based LLM in the same way as text. We start from a pre-trained LLM in which we inject the continuous inputs through an encoder. These encoders are trained end-to-end to output sequential decisions in terms of natural text that can be interpreted by the embodied agent by conditioning low-level policies or give an answer to an embodied question. We evaluate the approach in a variety of settings, comparing different input representations (e.g. standard vs. object-centric ViT encodings for visual input), freezing vs. finetuning the language model while training the encoders, and investigating whether co-training on multiple tasks enables transfer.
为了研究该方法的广度，我们在三个机器人操作领域（其中两个是真实世界的闭环）、标准视觉语言任务（如 VQA 和图像描述）以及语言任务上进行了评估。我们的结果表明，与在单个任务上训练模型相比，多任务训练提高了性能。我们表明，这种跨任务迁移可以带来机器人任务的高数据效率，例如，从少量训练示例中显著提高学习成功率，甚至展示了对新对象组合或未见对象的一次性（one-shot）或零样本（zero-shot）泛化能力。
To investigate the approach’s breadth, we evaluate on three robotic manipulation domains (two of which are closed-loop in the real-world), standard visual-language tasks such as VQA and image captioning, as well as language tasks. Our results indicate that multi-task training improves performance compared to training models on individual tasks. We show that this transfer across tasks can lead to high data-efficiency for robotics tasks, e.g. significantly increasing learning success from handfuls of training examples, and even demonstrating one-shot or zero-shot generalization to novel combinations of objects or unseen objects.
我们将 PaLM-E 扩展到 562B 参数，将 540B 的 PaLM LLM 和 22B 的视觉 Transformer（ViT）整合到据我们所知目前最大的视觉语言模型中。PaLM-E-562B 在 OK-VQA 基准上取得了最先进的性能，而无需依赖特定任务的微调。虽然这不是我们实验的重点，但我们也发现（图 2）PaLM-E-562B 展现出广泛的能力，包括零样本多模态思维链（CoT）推理、少样本提示、无 OCR 数学推理和多图像推理，尽管它仅在单图像示例上训练。零样本 CoT 最初是纯语言概念，已在多模态数据上通过特定任务程序得到展示，但据我们所知，尚未通过端到端模型实现。
We scale PaLM-E up to 562B parameters, integrating the 540B PaLM LLM and the 22B Vision Transformer (ViT) into, to our knowledge, the largest vision-language model currently reported. PaLM-E-562B achieves state-of-the-art performance on the OK-VQA benchmark, without relying on task-specific finetuning. Although not the focus of our experimentation, we also find (Fig. 2) that PaLM-E-562B exhibits a wide array of capabilities including zero-shot multimodal chain-of-thought (CoT) reasoning, few-shot prompting, OCR-free math reasoning, and multi-image reasoning, despite being trained on only single-image examples. Zero-shot CoT, originally a language-only concept, has been shown on multimodal data with task-specific programs but to our knowledge, not via an end-to-end model.
总结我们的主要贡献：（1）我们提出并证明，通过将具身数据混合到多模态大型语言模型的训练中，可以训练一个通用的、迁移学习的、多具身决策智能体。（2）我们表明，虽然当前最先进的通用视觉语言模型开箱即用（零样本）不能很好地解决具身推理问题，但可以训练一个既胜任通用视觉语言模型又高效具身推理器的模型。在研究如何最好地训练此类模型时，（3）我们引入了新颖的架构思想，如神经场景表示和实体标记多模态标记。最后，除了关注 PaLM-E 作为具身推理器外，（4）我们表明 PaLM-E 也是一个定量上胜任的视觉和语言通才，并且（5）证明扩展语言模型规模能够减少多模态微调中的灾难性遗忘。
To summarize our main contributions, we (1) propose and demonstrate that a generalist, transfer-learned, multi-embodiment decision-making agent can be trained via mixing in embodied data into the training of a multimodal large language model. We show that, (2) while current state-of-the-art general-purpose visual-language models out-of-the-box (zero-shot) do not well address embodied reasoning problems, it is possible to train a competent general-purpose visual-language model that is also an efficient embodied reasoner. In studying how to best train such models, we (3) introduce novel architectural ideas such as neural scene representations and entity-labeling multimodal tokens. Finally, in addition to our focus on PaLM-E as an embodied reasoner we (4) show that PaLM-E is also a quantitatively competent vision and language generalist, and (5) demonstrate that scaling the language model size enables multimodal finetuning with less catastrophic forgetting.
通用视觉-语言建模。在大型语言模型和视觉模型成功的基础上，近年来人们对大型视觉-语言模型（VLM）的兴趣日益增长。与之前的模型不同，VLM 能够同时理解图像和文本，并可应用于视觉问答、图像描述、光学字符识别和目标检测等任务。图像集成的方法各不相同。例如，有的方法通过一种机制增强预训练语言模型，使其能够直接关注单个上下文图像。相比之下，PaLM-E 将图像和文本表示为潜在向量的“多模态句子”，从而能够在句子的任何部分灵活地处理多个图像。与我们的工作更相关的是 Frozen，其中视觉编码器参数通过冻结的 LLM 进行反向传播优化。受此工作启发，我们在更广泛的范围内研究该设计，引入替代输入模态（例如神经场景表示），并且我们提出的方法在 VQAv2 基准上比 Frozen 高出 45% 以上。更重要的是，我们证明了 PaLM-E 不仅适用于感知任务，也适用于具身任务。
General vision-language modeling. Building on successes in large language and vision models, recent years have seen a growing interest in large vision-language models (VLMs). Unlike their predecessors, VLMs are capable of simultaneously understanding both images and text, and can be applied to tasks such as visual question answering, captioning, optical character recognition, and object detection. The methods by which images are integrated vary. For example, augments pretrained language models with a mechanism to directly attend to a single context image. In contrast, PaLM-E represents images and text as “multimodal sentences” of latent vectors, allowing it to process multiple images in a flexible way within any part of a sentence. More closely related to our work is Frozen where vision encoder parameters are optimized via backpropagation through a frozen LLM. Inspired by this work, we investigate the design in a broader scope by introducing alternative input modalities (e.g., neural scene representations), and our proposed approach empirically outperforms Frozen by more than 45% on the VQAv2 benchmark. More importantly, we demonstrate that PaLM-E is applicable not only to perceptual but also embodied tasks.
动作输出模型。先前的工作侧重于在具身环境中结合视觉和语言输入，目标是直接预测动作。在这些方法中，VIMA 探索了类似于 PaLM-E 的多模态提示。在这些工作中，语言的作用最恰当地描述为任务规范。相比之下，PaLM-E 生成高级指令作为文本；这样做，模型能够自然地以其自身预测为条件，并直接利用其参数中嵌入的世界知识。这不仅实现了具身推理，还实现了问答，正如我们的实验所示。在输出动作的工作中，最相似的可能是 Gato 中提出的方法，与 PaLM-E 一样，它是一个通用的多具身智能体。与 Gato 相比，我们展示了跨不同任务的正迁移，其中模型受益于跨多个领域的多样化联合训练。
Actions-output models. Prior works focus on combining vision and language inputs in an embodied setting with the goal of direct action prediction. Among these methods, VIMA explores multimodal prompts similar to PaLM-E. The role of language is perhaps most aptly described as task specification in these works. In contrast, PaLM-E generates high-level instructions as text; in doing so, the model is able to naturally condition upon its own predictions and directly leverage the world knowledge embedded in its parameters. This enables not only embodied reasoning but also question answering, as demonstrated in our experiments. Among works that output actions, perhaps most similar is the approach proposed in Gato which, like PaLM-E, is a generalist multi-embodiment agent. In contrast to Gato, we demonstrate positive transfer across different tasks where the model benefits from diverse joint training across multiple domains.
LLM 在具身任务规划中的应用。已有多种方法提出在具身领域利用 LLM。虽然许多工作侧重于理解自然语言目标，但较少有工作将自然语言作为规划的表征——这正是本工作的重点。LLM 包含大量内化的世界知识，但如果没有接地，生成的计划可能无法执行。一条研究路线采用提示方法直接从 LLM 中引出指令序列，要么利用 LLM 生成与合格指令集之间的语义相似性，要么结合可供性函数、视觉反馈、生成世界模型、基于图和地图的规划、视觉解释、程序生成，或将信息注入提示中。相比之下，PaLM-E 经过训练可直接生成计划，无需依赖辅助模型进行接地。这反过来使得预训练 LLM 中存储的丰富语义知识能够直接整合到规划过程中。
LLMs in embodied task planning. There have been several methods proposed to leverage LLMs in embodied domains. While many works focus on understanding natural language goals, fewer consider natural language as a representation for planning – the focus of this work. LLMs contain vast amounts of internalized knowledge about the world, but without grounding, generated plans may be impossible to execute. One line of research has employed prompting to elicit a sequence of instructions directly from an LLM either by leveraging semantic similarity between an LLM’s generation and an eligible set of instructions, incorporating affordance functions, visual feedback, generating world models, planning over graphs and maps, visual explanations, program generation, or injecting information into the prompt. In contrast, PaLM-E is trained to generate plans directly without relying on auxiliary models for grounding. This in turn enables direct integration of the rich semantic knowledge stored in pretrained LLMs into the planning process.
除了少数例外，许多工作中使用的 LLM 参数都是原样使用，没有进一步训练。在 LID 中，这一约束被放宽，LLM 参数被微调以产生用于生成高级指令的规划网络。(SL)³ 解决了同时微调两个 LLM 的更困难任务：一个规划网络（产生高级指令）和一个低级策略网络（选择动作）。对于 PaLM-E，我们的兴趣是不同且互补的：我们研究一个跨多种模态的通用多具身模型。
With few exceptions, the parameters of the LLMs employed in many of these works are employed as-is without further training. In LID, this constraint is relaxed and LLM parameters are finetuned to produce a planning network for generating high-level instructions. (SL)³ tackles the more challenging task of simultaneously finetuning two LLMs: a planning network, which produces high-level instructions, and a low-level policy network, which selects actions. With PaLM-E, our interests are distinct and complementary: we investigate a generalist, multi-embodiment model, across multiple modalities.
PaLM-E 的主要架构思想是将连续的、具身的观测（如图像、状态估计或其他传感器模态）注入到预训练语言模型的词嵌入空间中。这是通过将连续观测编码为与语言 token 嵌入空间维度相同的向量序列来实现的。因此，连续信息以类似于语言 token 的方式被注入语言模型。PaLM-E 是一个仅解码器的 LLM，给定前缀或提示，它自回归地生成文本补全。我们称我们的模型为 PaLM-E，因为我们使用 PaLM 作为预训练语言模型，并使其具身化。
The main architectural idea of PaLM-E is to inject continuous, embodied observations such as images, state estimates, or other sensor modalities into the language embedding space of a pre-trained language model. This is realized by encoding the continuous observations into a sequence of vectors with the same dimension as the embedding space of the language tokens. The continuous information is hence injected into the language model in an analogous way to language tokens. PaLM-E is a decoder-only LLM that generates textual completions autoregressively given a prefix or prompt. We call our model PaLM-E, since we use PaLM as the pre-trained language model, and make it Embodied.
PaLM-E 的输入包括文本和（多个）连续观测。与这些观测对应的多模态 token 与文本交错，形成多模态句子。这种多模态句子的一个例子是：Q: What happened between <img_1> and <img_2>? 其中 <img_i> 表示图像的嵌入。PaLM-E 的输出是由模型自回归生成的文本，可以是问题的答案，也可以是 PaLM-E 以文本形式产生的、应由机器人执行的一系列决策。当 PaLM-E 被要求产生决策或计划时，我们假设存在一个低级策略或规划器，可以将这些决策转换为低级动作。先前的工作已经讨论了训练这种低级策略的各种方法，我们直接使用这些先前的方法而不做修改。下面，我们更正式地描述我们的方法。
The inputs to PaLM-E consist of text and (multiple) continuous observations. The multimodal tokens corresponding to these observations are interleaved with the text to form multi-modal sentences. An example of such a multi-modal sentence is Q: What happened between <img_1> and <img_2>? where <img_i> represents an embedding of an image. The output of PaLM-E is text generated auto-regressively by the model, which could be an answer to a question, or a sequence of decisions produced by PaLM-E in textual form that should be executed by a robot. When PaLM-E is tasked with producing decisions or plans, we assume that there exists a low-level policy or planner that can translate these decisions into low-level actions. Prior work has discussed a variety of ways to train such low-level policies, and we use these prior methods directly without modification. In the following, we describe our approach more formally.
仅解码器 LLM。仅解码器的大型语言模型（LLM）是生成模型，训练目标是预测一段文本 \(p(w_{1:L})\) 的概率，该文本表示为 token 序列 \(w_{1:L}=(w_{1},\ldots,w_{L})\)，其中 \(w_{i}\in\mathcal{W}\)。典型的神经架构通过分解来实现这一点：
Decoder-only LLMs. Decoder-only large language models (LLMs) are generative models trained to predict the probability \(p(w_{1:L})\) of a piece of text \(w_{1:L}=(w_{1},\ldots,w_{L})\) that is represented as a sequence of tokens \(w_{i}\in\mathcal{W}\) . Typical neural architectures realize this by factorizing into
其中 \(p_{\text{LM}}\) 是一个大型 Transformer 网络。
where \(p_{\text{LM}}\) is a large transformer network.
前缀仅解码器 LLM。由于 LLM 是自回归的，预训练模型可以在前缀 \(w_{1:n}\) 上进行条件化，而无需改变架构。
Prefix-decoder-only LLMs. Since the LLM is auto-regressive, a pre-trained model can be conditioned on a prefix \(w_{1:n}\) without the necessity to change the architecture
前缀或提示 \(w_{1:n}\) 提供了上下文，基于此 LLM 继续预测后续的标记 \(w_{n+1:L}\)。这通常用于推理，以引导模型的预测。例如，提示可以包含 LLM 应解决的任务描述或类似任务的期望文本补全示例。
The prefix or prompt \(w_{1:n}\) provides the context based on which the LLM continues to predict the subsequent tokens \(w_{n+1:L}\). This is often used for inference to steer the predictions of the model. For example, the prompt can contain a description of the task the LLM should solve or examples of desired text completions for similar tasks.
标记嵌入空间。标记 \(w_{i}\) 是固定词汇表 \(\mathcal{W}\) 的元素，该词汇表是一个离散的有限集合，对应于自然语言中的（子）词。在内部，LLM 通过 \(\gamma:\mathcal{W}\rightarrow\mathcal{X}\) 将 \(w_{i}\) 嵌入到词标记嵌入空间 \(\mathcal{X}\subset\mathbb{R}^{k}\) 中，即 \(p_{\text{LM}}(w_{l}|x_{1:l-1})\)，其中 \(x_{i}=\gamma(w_{i})\in\mathbb{R}^{k}\)。映射 \(\gamma\) 通常表示为一个大小为 \(k\times|\mathcal{W}|\) 的大型嵌入矩阵，并进行端到端训练。在我们的案例中，\(|\mathcal{W}|=256\,000\)。
Token embedding space. The tokens \(w_{i}\) are elements of a fixed vocabulary \(\mathcal{W}\) which is a discrete, finite set corresponding to (sub)words in natural language. Internally, the LLM embeds \(w_{i}\) into a word token embedding space \(\mathcal{X}\subset\mathbb{R}^{k}\) via \(\gamma:\mathcal{W}\rightarrow\mathcal{X}\), i.e. \(p_{\text{LM}}(w_{l}|x_{1:l-1})\) with \(x_{i}=\gamma(w_{i})\in\mathbb{R}^{k}\). The mapping \(\gamma\) is typically represented as a large embedding matrix of size \(k\times|\mathcal{W}|\) and trained end-to-end. In our case, \(|\mathcal{W}|=256\,000\).
多模态句子：连续观测的注入。多模态信息（如图像观测）可以通过跳过离散标记层，直接将连续观测映射到语言嵌入空间 \(\mathcal{X}\) 来注入 LLM。为此，我们训练一个编码器 \(\phi:\mathcal{O}\rightarrow\mathcal{X}^{q}\)，将（连续的）观测空间 \(\mathcal{O}\)（详见第 4 节）映射为 \(\mathcal{X}\) 中的 \(q\) 个向量序列。然后，这些向量与正常的嵌入文本标记交错，形成 LLM 的前缀。这意味着前缀中的每个向量 \(x_{i}\) 由词标记嵌入器 \(\gamma\) 或编码器 \(\phi_{i}\) 形成：
Multi-modal sentences: injection of continuous observations. Multi-modal information such as image observations can be injected into the LLM by skipping the discrete token level and directly mapping the continuous observations into the language embedding space \(\mathcal{X}\). To this end, we train an encoder \(\phi:\mathcal{O}\rightarrow\mathcal{X}^{q}\) that maps a (continuous) observation space \(\mathcal{O}\) (refer to Section 4 for details) into a sequence of \(q\)-many vectors in \(\mathcal{X}\). These vectors are then interleaved with normal embedded text tokens to form the prefix for the LLM. This means that each vector \(x_{i}\) in the prefix is formed from either the word token embedder \(\gamma\) or an encoder \(\phi_{i}\):
注意，单个观测 \(O_{j}\) 通常被编码为多个嵌入向量。可以在前缀的不同位置交错不同的编码器 \(\phi_{i}\)，以组合例如来自不同观测空间的信息。以这种方式将连续信息注入 LLM，复用了其现有的位置编码。与其他 VLM 方法（例如，）相比，观测嵌入不是插入到固定位置，而是动态地放置在周围文本中。
Note that a single observation \(O_{j}\) is usually encoded into multiple embedding vectors. It is possible to interleave different encoders \(\phi_{i}\) at different locations in the prefix to combine, e.g., information from different observation spaces. Injecting the continuous information this way into the LLM reuses its existing positional encodings. In contrast to other VLM approaches (e.g., ), the observation embeddings are not inserted at fixed positions, but instead placed dynamically within the surrounding text.
具身化输出：PaLM-E 在机器人控制循环中。PaLM-E 是一个生成模型，基于多模态句子作为输入生成文本。为了将模型的输出连接到具身，我们区分两种情况。如果任务仅通过输出文本即可完成，例如在具身问答或场景描述任务中，那么模型的输出直接被视为任务的解决方案。
Embodying the output: PaLM-E in a robot control loop. PaLM-E is a generative model producing text based on multi-model sentences as input. In order to connect the output of the model to an embodiment, we distinguish two cases. If the task can be accomplished by outputting text only as, e.g., in embodied question answering or scene description tasks, then the output of the model is directly considered to be the solution for the task.
或者，如果使用 PaLM-E 来解决具身规划或控制任务，它会生成文本来调节低级命令。具体来说，我们假设可以访问能够从某个（小型）词汇表执行低级技能的策略，而 PaLM-E 的成功计划必须由一系列这样的技能组成。请注意，PaLM-E 必须根据训练数据和提示自行确定哪些技能可用，并且没有其他机制用于约束或过滤其输出。尽管这些策略是语言条件化的，但它们无法解决长时域任务或接受复杂指令。因此，PaLM-E 被集成到一个控制回路中，其预测的决策通过机器人的低级策略执行，从而产生新的观察，基于这些观察，PaLM-E 能够在必要时重新规划。从这个意义上说，PaLM-E 可以理解为一种高级策略，它编排和控制低级策略。
Alternatively, if PaLM-E is used to solve an embodied planning or control task, it generates text that conditions low-level commands. In particular, we assume to have access to policies that can perform low-level skills from some (small) vocabulary, and a successful plan from PaLM-E must consist of a sequence of such skills. Note that PaLM-E must determine on its own which skills are available based on the training data and the prompt, and no other mechanism is used to constrain or filter its outputs. Although these policies are language conditioned, they are not capable of solving long-horizon tasks or taking in complex instructions. PaLM-E is hence integrated into a control-loop, where its predicted decisions are executed through the low-level policies by a robot, leading to new observations based on which PaLM-E is able to replan if necessary. In this sense, PaLM-E can be understood as a high-level policy that sequences and controls the low-level policies.
在本节中，我们描述了纳入 PaLM-E 的各个模态，以及我们如何设置它们的编码器。我们为每个编码器 \(\phi:\mathcal{O}\rightarrow\mathcal{X}\) 提出了不同的架构选择，以将相应模态映射到语言嵌入空间。我们研究了状态估计向量、用于 2D 图像特征的 Vision Transformer（ViT），以及 3D 感知的对象场景表示 Transformer（OSRT）。除了全局表示输入场景的编码器外，我们还考虑了以对象为中心的表示，这些表示将观察分解为表示场景中单个对象的词元。
In this section, we describe the individual modalities that we incorporate into PaLM-E, and how we set up their encoders. We propose different architectural choices for each encoder \(\phi:\mathcal{O}\rightarrow\mathcal{X}\) to map the corresponding modality into the language embedding space. We investigate state estimation vectors, Vision Transformers (ViTs) for 2D image features, and the 3D-aware Object Scene Representation Transformer (OSRT). In addition to encoders that represent the input scene globally, we consider object-centric representations that factor observations into tokens that represent individual objects in the scene.
状态估计向量。状态向量，例如来自机器人或对象状态估计的向量，可能是输入到 PaLM-E 中最简单的形式。设 \(s\in\mathbb{R}^{S}\) 为描述场景中对象状态的向量。例如，\(s\) 可以包含这些对象的姿态、大小、颜色等。然后，MLP \(\phi_{\text{state}}\) 将 \(s\) 映射到语言嵌入空间。
State estimation vectors. State vectors, e.g. from a robot or a state estimate for objects, are perhaps the simplest to input into PaLM-E. Let \(s\in\mathbb{R}^{S}\) be a vector describing the state of the objects in a scene. For example, \(s\) could contain the pose, size, color etc. of those objects. Then, the MLP \(\phi_{\text{state}}\) maps \(s\) into the language embedding space.
Vision Transformer（ViT）。ViT \(\tilde{\phi}_{\text{ViT}}\) 是一种 Transformer 架构，将图像 \(I\) 映射为若干词元嵌入 \(\tilde{x}_{1:m}=\tilde{\phi}_{\text{ViT}}(I)\in\mathbb{R}^{m\times\tilde{k}}\) 。我们考虑了多种变体，包括来自 的 40 亿参数模型（我们称之为 ViT-4B），以及类似的 220 亿参数模型 ViT-22B，两者都已在图像分类上进行了预训练。我们还进一步研究了 ViT 词元学习器架构（ViT + TL），该架构从头开始端到端训练。注意，ViT 嵌入的维度 \(\tilde{k}\) 不一定与语言模型的维度相同。因此，我们将每个嵌入投影为 \(x_{i}=\phi_{\text{ViT}}(I)_{i}=\psi(\tilde{\phi}_{\text{ViT}}(I)_{i})\) ，其中 \(\psi\) 是一个学习到的仿射变换。
Vision Transformer (ViT). ViT \(\tilde{\phi}_{\text{ViT}}\) is a transformer architecture mapping an image \(I\) into a number of token embeddings \(\tilde{x}_{1:m}=\tilde{\phi}_{\text{ViT}}(I)\in\mathbb{R}^{m\times\tilde{k}}\) . We consider several variants, including the 4 billion parameter model from , which we refer to as ViT-4B, and a similar 22 billion parameter model, ViT-22B , both of which have been pre-trained on image classification. We further investigate the ViT token learner architecture (ViT + TL) which is trained end-to-end from scratch. Note that the dimensionality \(\tilde{k}\) of the ViT embeddings is not necessarily the same as that of the language model. We therefore project each embedding into \(x_{i}=\phi_{\text{ViT}}(I)_{i}=\psi(\tilde{\phi}_{\text{ViT}}(I)_{i})\) with \(\psi\) being a learned affine transformation.
以对象为中心的表示。与语言不同，视觉输入并未预先结构化为有意义的实体和关系：虽然 ViT 可能捕获语义，但表示的结构类似于静态网格，而不是对象实例的集合。这对与在符号上预训练的 LLM 的接口，以及解决需要与物理对象交互的具身推理，都构成了挑战。因此，我们还探索了结构化编码器，旨在将视觉输入在注入 LLM 之前分离为不同的对象。给定真实对象实例掩码 \(M_{j}\) ，我们可以将 ViT 的表示分解为对象 \(j\) 的 \(x_{1:m}^{j}=\phi_{\text{ViT}}(M_{j}\circ I)\) 。
Object-centric representations. Unlike language, visual input is not pre-structured into meaningful entities and relationships: while ViT may capture semantics, the structure of the representation resembles a static grid rather than a collection of object instances. This poses a challenge both for interfacing with LLMs which have been pre-trained on symbols, and for solving embodied reasoning which requires interaction with physical objects. We therefore also explore structured encoders that aim to separate visual inputs into distinct objects before injecting them into the LLM. Given ground-truth object instance masks \(M_{j}\) , we can decompose ViT’s representation into \(x_{1:m}^{j}=\phi_{\text{ViT}}(M_{j}\circ I)\) for object \(j\) .
对象场景表示 Transformer（OSRT）。不需要真实分割的替代方案是 OSRT：它不依赖关于对象的外部知识，而是通过架构中的归纳偏置以无监督方式发现对象。基于 SRT，OSRT 通过新颖视图合成任务在域内数据上学习 3D 中心的神经场景表示。其场景表示由对象槽 \(o_{j}=\bar{\phi}_{\text{OSRT}}(I_{1:v})_{j}\in\mathbb{R}^{\bar{k}}\) 组成。我们使用 MLP \(\psi\) 将每个槽投影为 \(x_{1:m}^{j}=\psi(\bar{\phi}_{\text{OSRT}}(I_{1:v})_{j})\) 。注意，单个对象总是被词元化为多个嵌入，即对于 OSRT，\(\psi:\mathbb{R}^{\bar{k}}\rightarrow\mathbb{R}^{m\times k}\) 映射到 \(m\) 个嵌入。
Object Scene Representation Transformer (OSRT). An alternative that does not require ground-truth segmentations is OSRT : rather than relying on external knowledge about objects, they are discovered in an unsupervised way through inductive biases in the architecture . Based on SRT , OSRT learns 3D-centric neural scene representations on in-domain data through a novel view synthesis task. Its scene representations consist of object slots \(o_{j}=\bar{\phi}_{\text{OSRT}}(I_{1:v})_{j}\in\mathbb{R}^{\bar{k}}\) . We project each of these slots into \(x_{1:m}^{j}=\psi(\bar{\phi}_{\text{OSRT}}(I_{1:v})_{j})\) with an MLP \(\psi\) . Note that individual objects are always tokenized into multiple embeddings each, i.e. \(\psi:\mathbb{R}^{\bar{k}}\rightarrow\mathbb{R}^{m\times k}\) for OSRT maps into \(m\) -many embeddings.
实体引用。对于具身规划任务，PaLM-E 必须能够在生成的计划中引用对象。在许多情况下，包括我们的大多数实验，场景中的对象可以通过其某些独特属性用自然语言识别。然而，也存在一些设置，其中对象不易用简短语言识别，例如，桌子上有多个颜色相同但位置不同的方块。对于像 OSRT 这样的对象中心表示，我们在输入提示中标记与对象对应的多模态 token，如下所示：对象 1 是 <obj_1>。\(\ldots\) 对象 \(j\) 是 <obj_ \(j\) >。这使得 PaLM-E 能够通过形如 obj_ \(j\) 的特殊 token 在其生成的输出句子中引用对象。在这种情况下，我们假设低级策略也操作这些 token。
Entity referrals. For embodied planning tasks, PaLM-E must be able to reference objects in its generated plan. In many cases, including the majority of our experiments, objects in a scene can be identified in natural language by some of their unique properties. However, there also exist settings where objects are not easily identifiable by language in few words, e.g. if there are multiple blocks on a table of the same color at different locations. For object-centric representations such as OSRT, we label the multi-modal tokens corresponding to an object in the input prompt as follows: Object 1 is <obj_1>. \(\ldots\) Object \(j\) is <obj_ \(j\) >. This enables PaLM-E to reference objects via special tokens of the form obj_ \(j\) in its generated output sentences. In this case, we assume that the low-level policies operate on these tokens as well.
PaLM-E 在一个形式为 \(D=\left\{\left(I_{1:u_{i}}^{i},w_{1:L_{i}}^{i},n_{i}\right)\right\}_{i=1}^{N}\) 的数据集上进行训练，其中每个样本 \(i\) 由 \(u_{i}\) 个连续观测 \(I_{j}^{i}\)、一段文本 \(w_{1:L_{i}}^{i}\) 和一个索引 \(n_{i}\) 组成。尽管是仅解码器模型，文本由两部分组成：一部分是到索引 \(n_{i}\) 为止的前缀部分，由多模态句子构成；另一部分是预测目标，仅包含文本词元。因此，损失函数是对非前缀词元 \(w_{n_{i}+1:L_{i}}^{i}\) 取平均的交叉熵损失。为了在模型内部形成多模态句子，我们在文本中设置特殊词元，这些词元在文本中的位置会被编码器的嵌入向量替换。我们将 PaLM-E 建立在预训练的 8B、62B 和 540B 参数规模的 PaLM 变体之上，作为仅解码器的大语言模型（LLM），通过输入编码器将连续观测注入其中。这些编码器要么是预训练的，要么是从头训练的，详见第 4 节。我们将 8B LLM 与 4B ViT 的组合称为 PaLM-E-12B，类似地，62B LLM + 22B ViT 称为 PaLM-E-84B，540B LLM + 22B ViT 称为 PaLM-E-562B。
PaLM-E is trained on a dataset of the form \(D=\left\{\left(I_{1:u_{i}}^{i},w_{1:L_{i}}^{i},n_{i}\right)\right\}_{i=1}^{N}\), where each example \(i\) consists of \(u_{i}\) continuous observations \(I_{j}^{i}\), a text \(w_{1:L_{i}}^{i}\), and an index \(n_{i}\). Despite being a decoder-only model, the text consists of a prefix part up to index \(n_{i}\) that is formed from multi-modal sentences, and the prediction target, which only contains text tokens. The loss function is therefore a cross-entropy loss averaged over the individual non-prefix tokens \(w_{n_{i}+1:L_{i}}^{i}\). To form the multi-modal sentences within the model, we have special tokens in the text that get replaced by the embedding vectors of the encoders at the locations in the text of those tokens. We base PaLM-E on the pre-trained 8B, 62B, and 540B parameter variants of PaLM as the decoder-only LLM into which we inject the continuous observations through the input encoders. Those encoders are either pre-trained or trained from scratch, see Sec. 4. We refer to an 8B LLM combined with a 4B ViT as PaLM-E-12B, similarly a 62B LLM + 22B ViT as PaLM-E-84B, and 540B LLM + 22B ViT as PaLM-E-562B.
模型冻结的变体。我们的大多数架构由三部分组成：编码器 \(\tilde{\phi}\)、投影器 \(\psi\) 和 LLM \(p_{\text{LM}}\)。在训练 PaLM-E 时，一种方式是更新所有这些组件的参数。然而，LLM 在提供合适的提示时展现出惊人的推理能力。因此，我们研究是否可能冻结 LLM 而只训练输入编码器，如果可能，不同模态的编码器又如何比较。在这种情况下，编码器必须产生嵌入向量，使得冻结的 LLM 能够基于观测进行接地，并将关于具身能力的信息传播给 LLM。训练这样的编码可以理解为一种输入条件的软提示形式，与普通软提示相关。在使用 \(\phi_{\text{OSRT}}\) 的实验中，我们还冻结了槽表示，即只更新小的投影器 \(\psi\)，它作为 OSRT 和 LLM 之间的接口。
Variation with Model freezing. Most of our architectures consist of three parts, an encoder \(\tilde{\phi}\), a projector \(\psi\), and the LLM \(p_{\text{LM}}\). When training PaLM-E, one way is to update the parameters of all these components. However, LLMs show impressive reasoning capabilities if supplied with a suitable prompt. Therefore, we investigate whether it is possible to freeze the LLM and to just train the input encoders, and if so, how different-modality encoders compare. In this case, the encoder has to produce embedding vectors such that the frozen LLM is grounded on the observations, and also propagate information to the LLM about the capabilities of an embodiment. Training such encodings can be understood as a form of input-conditioned soft-prompting, in relation to normal soft prompts. In experiments with \(\phi_{\text{OSRT}}\), we also freeze the slot representation, i.e. we only update the small projector \(\psi\) which serves as the interface between OSRT and the LLM.
跨任务协同训练。在我们的实验中，我们研究了在多种多样数据上协同训练模型的效果。"完整混合"（见附录 A）主要由来自各种任务的互联网规模视觉-语言数据组成。采样频率设置为：完整混合中只有 8.9% 是具身数据，并且每个具身都有多个任务。
Co-training across tasks. In our experiments, we investigate the effects of co-training our models on a variety of diverse data. The “full mixture”, see Appendix A, consists primarily of a diverse set of internet-scale vision-and-language data, from a variety of tasks. The sampling frequencies are set such that only 8.9% of the full mixture is embodied data, and there are several tasks for each embodiment.
我们的实验考虑了三种不同机器人形态下的多样化机器人（移动）操作任务，涵盖仿真环境和两种不同的真实机器人。我们参考 https://palm-e.github.io 上的视频，展示了 PaLM-E 在这些任务上的能力。虽然这不是我们工作的重点，但我们也在通用视觉-语言任务上评估了 PaLM-E，例如视觉问答（VQA）、图像描述和已有的语言建模任务。
Our experiments consider diverse robotic (mobile) manipulation tasks across three different robot embodiments, in simulation and with two different real robots. We refer to https://palm-e.github.io for videos showing the capabilities of PaLM-E on those tasks. Although not the focus of our work, we evaluate PaLM-E also on general vision-language tasks such as visual-question-answering (VQA), image captioning, and established language modeling tasks.
我们将实验研究分为两大类。首先，我们比较第 4 节中不同的输入表示在性能、泛化和数据效率方面的表现。第二组实验聚焦于一种架构，即主要的 PaLM-E 版本，由预训练的 ViT 和 PaLM 语言模型组成，以原始图像作为连续输入。在这里，我们展示了单个模型在多种数据集、跨任务和跨机器人形态的混合训练下，能够同时在这些任务上取得高性能。关键的是，我们研究了在这些数据集上的联合训练是否能够实现迁移（图 3）：尽管任务和形态不同，通过在任务混合上训练，各个任务的性能都有所提升。我们研究了联合训练策略和模型参数规模对性能、泛化和数据效率的影响。最后，我们考虑了冻结 LLM 并仅训练将视觉注入 LLM 的 ViT 是否是一条可行的路径。
We split our experimental investigation into two broad categories. First, we compare the different input representations from Sec. 4 with respect to performance, generalization, and data-efficiency. The second thread of experiments focuses on one architecture, the main PaLM-E version, consisting of a pre-trained ViT and PaLM language model that takes in raw images as the continuous inputs. Here we show that a single model, trained on a mixture of many datasets, across diverse tasks, and across robot embodiments, can simultaneously achieve high performance on all of those tasks. Crucially, we investigate whether co-training on these datasets enables transfer (Fig. 3): despite different tasks and embodiments, the performance on the individual tasks increases by training on the mixture of tasks. We study the influence on performance, generalization, and data efficiency with respect to co-training strategies and model parameter size. Finally, we consider if freezing the LLM and just training the ViT that injects vision into the LLM is a viable path.
作为基线，我们考虑了最先进的视觉语言模型 PaLI（未在具身机器人数据上训练），以及提供 oracle 可供性的 SayCan 算法。
As baselines, we consider the state-of-the art visual language model PaLI , which has not been trained on embodiment robot data, as well as the SayCan algorithm , supplied with oracle affordances.
我们的三个机器人环境（图 1）包括一个任务与运动规划（TAMP）领域，其中机器人需要操作（抓取和堆叠）物体；一个桌面推挤环境；以及一个移动操作领域。在每个领域中，PaLM-E 都使用该领域的专家数据进行训练。在许多情况下，每个任务的数据量是稀疏的。TAMP 任务涉及大量可能的计划组合，且许多决策序列是不可行的。PaLM-E 必须生成包含多个步骤、具有复杂决策边界的计划。多物体桌面推挤环境取自公开可用的 Language-Table 数据集，由于包含多个物体、语言基数大以及复杂的推挤动力学，该环境具有挑战性。对于 TAMP 和 Language-Table 环境，PaLM-E 都需要推理物体的位姿。仅仅知道桌子上有哪些物体或它们的大致关系是不够的，场景几何的更细粒度细节对于解决任务至关重要。最后，我们考虑一个类似于 SayCan 的移动操作领域，其中机器人需要在厨房环境中解决各种任务，包括在抽屉中寻找物体、拾取物体并将其带给人类。对于所有领域，我们都考虑这些环境中的规划任务和 VQA 任务。对于移动操作和 Language-Table 环境，PaLM-E 被集成到控制回路中，以在现实世界中执行计划，并在存在外部干扰或低级控制策略失败时调整计划。
Our three robot environments (Fig. 1) include a Task and Motion Planning (TAMP) domain where a robot has to manipulate (grasp and stack) objects, a table-top pushing environment, and a mobile manipulation domain. In each domain, PaLM-E is trained on expert data from that domain. In many cases, this is a sparse amount of data per task. The TAMP tasks involve large combinatorics over possible plans, and many decision sequences are infeasible. PaLM-E has to generate plans that consist of multiple steps, with complicated decision boundaries. The multi-object tabletop pushing environment is taken from the publicly available Language-Table dataset and is challenging since it includes several objects, large cardinality of language, and complex pushing dynamics. For both the TAMP and Language-Table environment, PaLM-E has to reason about the poses of the objects. It is not sufficient to know which objects are on the table or knowing their rough relationships, the more fine-grained details about the scene geometry are important for solving the tasks. Finally, we consider a mobile manipulation domain similar to SayCan, where a robot has to solve a variety of tasks in a kitchen environment, including finding objects in drawers, picking them, and bringing them to a human. For all domains we consider both planning and VQA tasks in those environments. For the mobile manipulation and Language-Table environments, PaLM-E is integrated into the control loop to execute the plans in the real world, and has to adjust the plan in presence of external disturbances or failures of the low-level control policies.
表 7（附录）展示了 TAMP 环境的规划成功率和 VQA 性能。在这些实验中，LLM 被冻结（针对预训练 LLM）。对于表 7 中报告的结果，输入表示在仅包含 TAMP 环境的 96,000 个训练场景的数据集上进行训练，即不包含其他数据。当场景中有 3-5 个物体（与训练集中的数量相同）时，大多数输入表示的表现相似。然而，当增加物体数量时，使用预训练 LLM 显著提升了性能，尤其是在实体引用方面。此外，我们表明，与 8B 变体相比，62B LLM 展现出更好的分布外泛化能力，而未预训练的 LLM 基本上没有分布外泛化能力。SayCan 基线利用 oracle 可供性函数，但在解决该环境时遇到困难，因为可供性函数仅约束当前可能的行为，不足以让 LLM 在 TAMP 环境中构建长期规划。
Tab. 7 (appendix) shows planning success rates and VQA performance for the TAMP environment. The LLM is frozen in these experiments (for pre-trained LLM). For the results reported in Tab. 7, the input representations are trained on a dataset containing 96,000 training scenes of solely the TAMP environment, i.e. no other data is part of the mixture. For 3-5 objects in the scene, which is the same number as in the training set, most input representations perform similarly well. However, when increasing the number of objects, it turns out that using a pre-trained LLM improves performance considerably, especially with entity referrals. Furthermore, we show that a 62B LLM shows better out-of-distribution generalization compared to the 8B variant, while a non-pretrained LLM shows basically no out-of-distribution generalization. The SayCan baseline utilizes oracle affordance functions and has difficulties solving this environment, since affordance functions only constrain what is possible right now, but are not informative enough for the LLM to construct long-horizon plans in TAMP environments.
表 1 显示了在 1% 数据集上训练时 3-5 个物体的结果，这对应于两个规划任务各仅有 320 个示例。在这里，我们看到输入表示之间存在显著差异，尤其是在规划任务上。首先，在低数据状态下，预训练 LLM 对状态输入有益。其次，两种 ViT 变体（ViT+TL、ViT-4B）在如此少的数据下解决规划任务时表现不佳。然而，如果我们在所有其他机器人环境以及通用视觉-语言数据集（ViT-4B 通用型）上进行联合训练，ViT-4B 的性能会提高一倍以上。这表明不同机器人具身和任务之间存在显著的迁移效应。最后，使用 OSRT 作为输入表示在此处取得了最佳性能，展示了 3D 感知物体表示的优势。我们还观察到另一个迁移实例：当我们移除 TAMP VQA 数据，仅使用 640 个规划任务示例进行训练时，性能会（略微）下降。最先进的视觉-语言模型 PaLI 未在机器人数据上训练，无法解决这些任务。我们仅对其评估了 \(\text{q}_{2}\)（桌子上物体的左/右/中位置）和 \(\text{q}_{3}\)（垂直物体关系），因为这些最接近典型的 VQA 任务。
Tab. 1 shows results for 3-5 objects when training on 1% of the dataset, which corresponds to only 320 examples for each of the two planning tasks. Here we see that there are significant differences between the input representations, especially for the planning tasks. First, pre-training the LLM is beneficial in the low data regime for state inputs. Second, both ViT variants (ViT+TL, ViT-4B) do not perform well in solving the planning tasks for this little data. However, if we co-train on all other robot environments as well as general vision-language datasets (ViT-4B generalist), then the performance of the ViT-4B more than doubles. This shows a significant transfer effect between different robot embodiments and tasks. Finally, using OSRT as the input representation leads to the best performance here, demonstrating the strengths of 3D-aware object representations. We also observe another instance of transfer here: when we remove the TAMP VQA data and only train on the 640 planning tasks examples, there is a (slight) drop in performance. The state-of-the art vision-language model PaLI that was not trained on robot data is not able to solve the tasks. We only evaluated it on \(\text{q}_{2}\) (objects left/right/center on the table) and \(\text{q}_{3}\) (vertical object relations), since those most resemble typical VQA tasks.
表 3 报告了 Language-Table 环境中长时程任务的成功率。PaLM-E 被集成到一个控制回路中，该回路以长时程任务和当前图像为输入，并为低层策略输出指令。我们发现，在互联网规模的视觉和语言上进行联合训练，能够产生更有效的机器人规划模型，尤其是在每个任务仅有 10 个演示的少样本场景中。将 12B 模型扩展到 84B 模型，在 3 个任务中的 2 个上带来了改进。与 TAMP 环境类似，SayCan 和零样本 PaLI 均无效，无法解决所测试的最简单任务。
Tab. 3 reports success rates on long-horizon tasks from the Language-Table environment. PaLM-E is integrated into a control loop that takes as input the long-horizon task and the current image, and outputs an instruction for the low-level policy. We see that joint training on internet-scale vision and language results in a more effective model for robot planning, particularly in the few-shot regime with only 10 demos per task. Scaling the 12B model to the 84B model leads to improvements on 2 of 3 tasks. As with the TAMP environment, neither SayCan nor zero-shot PaLI are effective, unable to solve the easiest task tested.
真实机器人结果与少样本泛化。在图 7(a) 中，我们看到 PaLM-E 能够引导真实机器人完成多阶段桌面操作任务，同时对对抗性干扰保持鲁棒性。给定观察到的图像和长时程目标（例如“将积木按颜色分类到角落”），PaLM-E 以 1 Hz 的频率向策略输出语言子目标，这些策略以 5 Hz 的频率输出低层机器人动作。先前的工作则需要人类参与循环，交互式地指导子目标和修正。在图 5(b) 中，我们看到 PaLM-E 具备一次和零样本学习能力。在这里，我们在 100 个不同的长时程任务上对 PaLM-E 进行了微调，每个任务只有一个训练示例，例如“将所有积木放在中心”、“从线条中移除蓝色积木”。我们还看到，PaLM-E 可以零样本泛化到涉及新物体对的任务（图 7(c)），以及涉及原始机器人数据集或微调数据集中未见过的物体的任务，例如玩具乌龟（图 5(d)）。
Real Robot Results and Few-Shot Generalization. In Fig. 7, a), we see PaLM-E is capable of guiding a real robot through a multi-stage tabletop manipulation task, while remaining robust to adversarial disturbances. Given the observed image and a long-horizon goal, e.g. “sort the blocks by colors into corners”, PaLM-E outputs language subgoals at 1 Hz to the policies from , that output low-level robot actions at 5 Hz. Prior work instead involved a human in the loop to interactively guide subgoals and corrections. In Figure 5, b) we see PaLM-E is capable of one-shot and zero-shot learning. Here, we finetuned PaLM-E on 100 different long horizon tasks with a single training example each, e.g. “put all the blocks in the center”, “remove the blue blocks from the line”. We additionally see that PaLM-E can generalize zero-shot to tasks involving novel object pairs (Figure 7, c) and to tasks involving objects that were unseen in either the original robot dataset or the finetuning datasets, e.g. a toy turtle (Figure 5, d).
我们在具有挑战性和多样化的移动操作任务上展示了 PaLM-E 的性能。我们大体上遵循 [ref] 中的设置，其中机器人需要根据人类的指令规划一系列导航和操作动作。例如，给定指令“我把饮料洒了，你能给我拿点东西来清理吗？”，机器人需要规划一个包含“1. 找到海绵，2. 拿起海绵，3. 把它带给用户，4. 放下海绵”的序列。受这些任务的启发，我们开发了 3 个用例来测试 PaLM-E 的具身推理能力：可负担性预测、失败检测和长时程规划。底层策略来自 RT-1 [ref]，这是一个 Transformer 模型，接收 RGB 图像和自然语言指令，并输出末端执行器控制命令。
We demonstrate the performance of PaLM-E on challenging and diverse mobile manipulation tasks. We largely follow the setup in [ref], where the robot needs to plan a sequence of navigation and manipulation actions based on an instruction by a human. For example, given the instruction “I spilled my drink, can you bring me something to clean it up?”, the robot needs to plan a sequence containing “1. Find a sponge, 2. Pick up the sponge, 3. Bring it to the user, 4. Put down the sponge.” Inspired by these tasks, we develop 3 use cases to test the embodied reasoning abilities of PaLM-E: affordance prediction, failure detection, and long-horizon planning. The low-level policies are from RT-1 [ref], a transformer model that takes RGB image and natural language instruction, and outputs end-effector control commands.
可负担性预测。我们研究了 PaLM-E 在可负担性预测上的性能，即底层策略的技能是否可以在当前环境中执行。这可以表述为 VQA 问题：给定 <img>。问：这里可以 <技能> 吗？。PaLM-E 优于 PaLI（零样本），也优于使用 QT-OPT 训练的值函数阈值方法（表 4）。
Affordance prediction. We investigate PaLM-E’s performance at affordance prediction, i.e. whether a skill of the low-level policy can be executed in the current environment. This can be formulated as the VQA problem: Given <img>. Q: Is it possible to <skill> here?. PaLM-E outperforms PaLI (zero-shot), as well as thresholding on value functions trained with QT-OPT (Tab. 4).
失败检测。对于机器人进行闭环规划，检测失败也很重要，如 [ref] 所示。多模态提示为：给定 <img>。问：<技能> 成功了吗？。表 4 显示，PaLM-E 在该数据集上优于 PaLI（零样本）以及微调版本的 CLIP。PaLM-E 还优于 [ref] 中提出的算法，该算法利用两个使用事后重标记数据训练的 CLIP 模型。该方法比我们的方法能访问更多信息，并且专门设计用于仅解决该数据集上的失败检测。
Failure detection. For a robot to do closed-loop planning, it is also important to detect failures, as is shown in [ref]. The multi-modal prompt is: Given <img>. Q: Was <skill> successful?. Tab. 4 shows that PaLM-E outperforms PaLI (zero-shot), as well as a fine-tuned version of CLIP on this dataset. PaLM-E also outperforms the algorithm proposed in [ref] that leverages two CLIP models trained with hindsight relabeled data. This method has access to more information than our method, and was specifically designed to just solve failure detection on this dataset.
真实机器人结果：长时程规划。最后，我们使用 PaLM-E 对移动操作任务进行端到端的具身规划。此任务的提示结构为：人类：<指令> 机器人：<步骤历史>。我看到 <img>。PaLM-E 被训练为根据已执行步骤的历史和当前场景的图像观察来生成计划的下一步。在解码每一步后，我们将其映射到 [ref] 中定义的底层策略。此过程以自回归方式执行，直到 PaLM-E 输出“终止”。我们使用 [ref] 中的运行数据训练模型，其中包含 2912 个序列。我们在真实厨房中定性评估了模型，发现该模型可以执行长时程移动操作任务，即使在对抗性干扰下也是如此（图 5）。
Real robot results: Long-horizon planning. Finally, we use PaLM-E to perform embodied planning end-to-end for mobile manipulation tasks. The prompt structure for this task is: Human: <instruction> Robot: <step history>. I see <img>. PaLM-E is trained to generate the next step of the plan, conditioned on the history of taken steps and the current image observation of the scene. After each step is decoded, we map them to a low-level policy as defined in [ref]. This process is done in an autoregressive manner, until PaLM-E outputs “terminate”. We train the model by using the runs from [ref], which contains 2912 sequences. We qualitatively evaluated the model in a real kitchen and found the model can carry out long-horizon mobile manipulation tasks, even under adversarial disturbances (Fig. 5).
尽管这不是我们工作的重点，我们在表 5 中报告了通用视觉-语言任务上的结果，包括 OK-VQA、VQA v2 和 COCO 图像描述。一个单一的通用型 PaLM-E-562B 模型在 OK-VQA 上取得了最高报告分数，甚至超过了专门在 OK-VQA 上微调的模型。与先前工作相比，据我们所知，PaLM-E 在使用冻结大语言模型（LLM）的情况下在 VQA v2 上取得了最高性能。这表明 PaLM-E 不仅是一个有竞争力的视觉-语言通用模型，还是一个在机器人任务上具备具身推理能力的模型。
Although it is not the focus of our work, we report in Tab. 5 results on general vision-language tasks, including OK-VQA, VQA v2, and COCO captioning. A single, generalist PaLM-E-562B model achieves the highest reported number on OK-VQA, including outperforming models fine-tuned specifically on OK-VQA. Compared to prior work, PaLM-E achieves the highest performance on VQA v2 with a frozen LLM to the best of our knowledge. This establishes that PaLM-E is a competitive visual-language generalist, in addition to being an embodied reasoner on robotic tasks.
表 8 报告了 PaLM-E 在 21 个通用语言基准上的平均性能，这些基准涵盖自然语言理解（NLU）和自然语言生成（NLG）任务。一个显著的趋势是，随着模型规模的增大，语言能力的灾难性遗忘显著减少。如图 6 所示，对于最小的模型（PaLM-E-12B），在多模态训练期间其 NLG 性能（相对值）下降了 87.3%，而对于最大的模型（PaLM-E-562B），仅下降了 3.9%。
Tab. 8 reports the averaged performance of PaLM-E on 21 general language benchmarks for Natural Language Understanding (NLU) and Natural Language Generation (NLG) tasks. The notable trend is that with increasing model scale, there is considerably less catastrophic forgetting of language capabilities. As seen in Fig. 6, while for the smallest (PaLM-E-12B) model 87.3% of its NLG performance (relative) has degraded during multimodal training, merely 3.9% have been degraded for the largest model (PaLM-E-562B).
通用模型与专用模型——迁移。如图 3 所总结，我们在本工作中展示了若干迁移实例，即 PaLM-E 同时在不同任务和数据集上训练，相比单独在不同任务上训练的模型，性能显著提升。在图 4 中，在“完整混合”上进行协同训练实现了超过两倍的性能。在表 9 中，我们看到如果加入 LLM/ViT 预训练，并在完整混合数据而非仅移动操作数据上训练，性能会有显著提升。对于表 3 中的 Language-Table 实验，我们观察到类似的行为。
Generalist vs specialist models – transfer. As summarized in Fig. 3, we have shown several instances of transfer in this work, meaning that PaLM-E trained on different tasks and datasets at the same time leads to significantly increased performance relative to models trained separately on the different tasks alone. In Fig. 4, co-training on the “full mixture” achieves more than double the performance. In Tab. 9, we see significant improvements in performance if we add LLM/ViT pre-training, and training on the full mixture instead of the mobile manipulation data alone. For the Language-Table experiment in Tab. 3, we observe analogous behaviour.
数据效率。与现有的大规模语言或视觉-语言数据集相比，机器人数据要少得多。如上一段所讨论，我们的模型表现出迁移能力，这有助于 PaLM-E 从极少量的机器人领域训练样本中解决机器人任务，例如 Language Table 的 10 到 80 个样本，或 TAMP 的 320 个样本。OSRT 的结果通过使用几何输入表示展示了数据效率的另一个实例。未来一个有前景的方向是将此与受益于大规模视觉数据的方法相结合。
Data efficiency. Compared to available massive language or vision-language datasets, robotics data is significantly less abundant. As discussed in the last paragraph, our model exhibits transfer, which aids PaLM-E to solve robotics tasks from very few training examples in the robotics domain, e.g. between 10 and 80 for Language Table or 320 for TAMP. The OSRT results show another instance of data-efficiency by using a geometric input representation. A promising opportunity for future work is to combine this with a method benefitting from large-scale visual data.
保留语言能力。我们展示了在多模态训练期间保留模型语言能力的两种途径。一种选择是冻结 LLM，仅训练输入编码器，这是构建具身语言模型的可行路径，尽管这种方法在机器人任务上偶尔会遇到困难（表 3）。另一种替代路线是，当整个模型进行端到端训练时，随着模型规模的增大，模型保留的原始语言性能显著更多（图 6）。
Retaining language capabilities. We have shown two paths to retain the language capabilities of the model during multimodal training. As one option, freezing the LLM and only training the input encoders is a viable path for building embodied language models, although this approach occasionally struggled for robotics tasks (Tab. 3). As an alternative route, when the whole model is trained end-to-end, the model retains significantly more of its original language performance with increasing model scale (Fig. 6).
我们提出通过将图像等多模态信息注入预训练大语言模型（LLM）的嵌入空间来构建具身语言模型。实验表明，在通用 VQA 和图像描述任务上训练的现成最先进视觉-语言模型不足以应对具身推理任务，同时也揭示了近期通过可供性（affordances）来接地语言模型的提案的局限性。为克服这些局限，我们提出了 PaLM-E，这是一个单一模型，能够控制模拟和现实世界中的不同机器人，同时在通用 VQA 和图像描述任务上表现出定量上的竞争力。尤其值得一提的是，将神经场景表示（即 OSRT）注入模型这一新颖架构思想非常有效，即使没有大规模数据也是如此。PaLM-E 在多个机器人具身以及通用视觉-语言任务的混合多样任务上进行训练。重要的是，我们证明了这种多样化训练带来了从视觉-语言领域到具身决策制定的多种迁移途径，使得机器人规划任务能够以数据高效的方式完成。虽然我们的结果表明冻结语言模型是实现通用具身多模态模型（完全保留其语言能力）的可行路径，但我们也揭示了另一种使用非冻结模型的替代路线：扩大语言模型规模在成为具身智能体的同时，显著减少了灾难性遗忘。我们最大的模型 PaLM-E-562B 展示了涌现能力，如多模态思维链推理，以及尽管仅使用单图像提示进行训练，却能对多图像进行推理的能力。
We proposed to build an embodied language model by injecting multi-modal information such as images into the embedding space of a pre-trained LLM. Experiments showed that off-the-shelf state-of-the-art vision-language models trained on general VQA and captioning tasks are not sufficient for embodied reasoning tasks, as well as limitations of a recent proposal for grounding language models through affordances. To overcome these limitations, we proposed PaLM-E, a single model that is able to control different robots in simulation and in the real world, while at the same time being quantitatively competent at general VQA and captioning tasks. In particular the novel architectural idea of ingesting neural scene representations (i.e., OSRT) into the model is particularly effective, even without large-scale data. PaLM-E is trained on a mixture of diverse tasks across multiple robot embodiments as well as general vision-language tasks. Importantly, we have demonstrated that this diverse training leads to several avenues of transfer from the vision-language domains into embodied decision making, enabling robot planning tasks to be achieved data efficiently. While our results indicate that frozen language models are a viable path towards general-purpose embodied multimodal models that fully retain their language capabilities, we have also surfaced an alternative route with unfrozen models: scaling up the language model size leads to significantly less catastrophic forgetting while becoming an embodied agent. Our largest model, PaLM-E-562B, showcases emergent capabilities like multimodal chain of thought reasoning, and the ability to reason over multiple images, despite being trained on only single-image prompts.