Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→大型语言模型能够编码关于世界的丰富语义知识，这对于旨在根据自然语言表达的高层、时间延展指令行动的机器人来说极为有用。然而，语言模型的一个显著弱点是缺乏真实世界经验，这使得在特定具身形态下利用它们进行决策变得困难。例如，要求语言模型描述如何清理溢出物可能会产生合理的叙述，但对于需要在特定环境中执行此任务的特定代理（如机器人）来说，可能并不适用。我们提出通过预训练技能来提供真实世界的基础，这些技能用于约束模型，使其提出既可行又上下文适当的自然语言动作。机器人可以作为语言模型的“手和眼”，而语言模型则提供关于任务的高层语义知识。我们展示了如何将低层技能与大型语言模型相结合，使得语言模型提供关于执行复杂和时间延展指令的程序的高层知识，而与这些技能相关的价值函数则提供了必要的根基，将这些知识与特定的物理环境联系起来。我们在多个真实世界机器人任务上评估了我们的方法，展示了真实世界根基的必要性，并且该方法能够在移动操作器上完成长时域、抽象的自然语言指令。项目和视频可在 https://say-can.github.io/ 找到。
Large language models can encode a wealth of semantic knowledge about the world. Such knowledge could be extremely useful to robots aiming to act upon high-level, temporally extended instructions expressed in natural language. However, a significant weakness of language models is that they lack real-world experience, which makes it difficult to leverage them for decision making within a given embodiment. For example, asking a language model to describe how to clean a spill might result in a reasonable narrative, but it may not be applicable to a particular agent, such as a robot, that needs to perform this task in a particular environment. We propose to provide real-world grounding by means of pretrained skills, which are used to constrain the model to propose natural language actions that are both feasible and contextually appropriate. The robot can act as the language model's "hands and eyes," while the language model supplies high-level semantic knowledge about the task. We show how low-level skills can be combined with large language models so that the language model provides high-level knowledge about the procedures for performing complex and temporally-extended instructions, while value functions associated with these skills provide the grounding necessary to connect this knowledge to a particular physical environment. We evaluate our method on a number of real-world robotic tasks, where we show the need for real-world grounding and that this approach is capable of completing long-horizon, abstract, natural language instructions on a mobile manipulator. The project's website and the video can be found at https://say-can.github.io/.