Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→在这项工作中，我们提出了一种可扩展的强化学习方法，用于从大型离线数据集中训练多任务策略，该方法可以利用人类演示和自主收集的数据。我们的方法使用 Transformer 为通过离线时间差分备份训练的 Q 函数提供可扩展的表示，因此我们将该方法称为 Q-Transformer。通过将每个动作维度离散化并将每个动作维度的 Q 值表示为单独的标记，我们可以应用有效的高容量序列建模技术进行 Q 学习。我们提出了几个设计决策，以实现良好的离线 RL 训练性能，并表明 Q-Transformer 在大型多样化的真实世界机器人操作任务套件上优于先前的离线 RL 算法和模仿学习技术。项目和视频可在 https://qtransformer.github.io 找到。
In this work, we present a scalable reinforcement learning method for training multi-task policies from large offline datasets that can leverage both human demonstrations and autonomously collected data. Our method uses a Transformer to provide a scalable representation for Q-functions trained via offline temporal difference backups. We therefore refer to the method as Q-Transformer. By discretizing each action dimension and representing the Q-value of each action dimension as separate tokens, we can apply effective high-capacity sequence modeling techniques for Q-learning. We present several design decisions that enable good performance with offline RL training, and show that Q-Transformer outperforms prior offline RL algorithms and imitation learning techniques on a large diverse real-world robotic manipulation task suite. The project's website and videos can be found at https://qtransformer.github.io
在这项工作中，我们提出了一种可扩展的强化学习方法，用于从大规模离线数据集中训练多任务策略，该方法可以利用人类演示和自主收集的数据。我们的方法使用 Transformer 为通过离线时序差分备份训练的 Q 函数提供可扩展的表示。因此，我们将该方法称为 Q-Transformer。通过将每个动作维度离散化，并将每个动作维度的 Q 值表示为独立的 token，我们可以将有效的高容量序列建模技术应用于 Q 学习。我们提出了若干设计决策，使得离线强化学习训练能够获得良好性能，并展示了 Q-Transformer 在大型多样化的真实世界机器人操作任务套件上优于先前的离线强化学习算法和模仿学习技术。项目网站和视频可在 qtransformer.github.io 找到。
In this work, we present a scalable reinforcement learning method for training multi-task policies from large offline datasets that can leverage both human demonstrations and autonomously collected data. Our method uses a Transformer to provide a scalable representation for Q-functions trained via offline temporal difference backups. We therefore refer to the method as Q-Transformer. By discretizing each action dimension and representing the Q-value of each action dimension as separate tokens, we can apply effective high-capacity sequence modeling techniques for Q-learning. We present several design decisions that enable good performance with offline RL training, and show that Q-Transformer outperforms prior offline RL algorithms and imitation learning techniques on a large diverse real-world robotic manipulation task suite. The project's website and videos can be found at qtransformer.github.io
Yevgen Chebotar∗, Quan Vuong∗, Alex Irpan, Karol Hausman, Fei Xia, Yao Lu, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, Keerthana Gopalakrishnan, Julian Ibarz, Ofir Nachum, Sumedh Sontakke, Grecia Salazar, Huong T Tran, Jodilyn Peralta, Clayton Tan, Deeksha Manjunath, Jaspiar Singht, Brianna Zitkovich, Tomas Jackson, Kanishka Rao, Chelsea Finn, Sergey Levine
Yevgen Chebotar∗, Quan Vuong∗, Alex Irpan, Karol Hausman, Fei Xia, Yao Lu, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, Keerthana Gopalakrishnan, Julian Ibarz, Ofir Nachum, Sumedh Sontakke, Grecia Salazar, Huong T Tran, Jodilyn Peralta, Clayton Tan, Deeksha Manjunath, Jaspiar Singht, Brianna Zitkovich, Tomas Jackson, Kanishka Rao, Chelsea Finn, Sergey Levine
结合大规模多样化数据集与高容量表达模型（如 Transformer）的机器人学习方法，有潜力获得可泛化且广泛适用的策略，这些策略能在各种任务上表现良好。例如，这些策略可以遵循自然语言指令、执行多阶段行为，并在环境、物体甚至机器人形态上广泛泛化。然而，机器人学习文献中许多近期提出的高容量模型都是通过监督学习方法训练的。因此，所得策略的性能受到人类演示者提供高质量演示数据程度的限制。这有两个原因。首先，我们希望机器人系统比人类遥控操作员更熟练，充分利用硬件的潜力来快速、流畅且可靠地执行任务。其次，我们希望机器人系统能够通过自主收集的经验不断改进，而不是完全依赖高质量的演示。
Robotic learning methods that incorporate large and diverse datasets in combination with high-capacity expressive models, such as Transformers, have the potential to acquire generalizable and broadly applicable policies that perform well on a wide variety of tasks. For example, these policies can follow natural language instructions, perform multi-stage behaviors, and generalize broadly across environments, objects, and even robot morphologies. However, many of the recently proposed high-capacity models in the robotic learning literature are trained with supervised learning methods. As such, the performance of the resulting policy is limited by the degree to which human demonstrators can provide high-quality demonstration data. This is limiting for two reasons. First, we would like robotic systems that are more proficient than human teleoperators, exploiting the full potential of the hardware to perform tasks quickly, fluently, and reliably. Second, we would like robotic systems that get better with autonomously gathered experience, rather than relying entirely on high-quality demonstrations.
强化学习原则上提供了这两种能力。许多有前景的近期进展展示了大规模机器人强化学习在不同场景中的成功，例如机器人抓取和堆叠、使用人类指定奖励学习异构任务、学习多任务策略、学习目标条件策略以及机器人导航。然而，使用强化学习算法训练高容量模型（如 Transformer）已被证明在大规模上难以有效实现。在本文中，我们旨在将来自多样化真实世界数据集的大规模机器人学习与现代高容量基于 Transformer 的策略架构相结合。
Reinforcement learning in principle provides both of these capabilities. A number of promising recent advances demonstrate the successes of large-scale robotic RL in varied settings, such as robotic grasping and stacking, learning heterogeneous tasks with human-specified rewards, learning multi-task policies, learning goal-conditioned policies, and robotic navigation. However, training high-capacity models such as Transformers using RL algorithms has proven more difficult to instantiate effectively at large scale. In this paper, we aim to combine large-scale robotic learning from diverse real-world datasets with modern high-capacity Transformer-based policy architectures.
虽然原则上简单地将现有架构（如 ResNet 或较小的卷积神经网络）替换为 Transformer 在概念上是直接的，但设计一种有效利用此类架构的方法则更具挑战性。高容量模型只有在训练大规模多样化数据集时才有意义——小而窄的数据集根本不需要这么大的容量，也不会从中受益。虽然先前的工作使用仿真来创建此类数据集，但最具代表性的数据来自真实世界。因此，我们专注于能够使用 Transformer 并通过离线强化学习整合大规模先前收集数据集的强化学习方法。离线强化学习方法在先前数据上训练，旨在从给定数据集中推导出最有效的策略。当然，该数据集可以通过额外自主收集的数据进行扩充，但训练与数据收集分离，为大规模机器人应用提供了一种有吸引力的工作流程。
While in principle simply replacing existing architectures (e.g., ResNets or smaller convolutional neural networks) with a Transformer is conceptually straightforward, devising a methodology that effectively makes use of such architectures is considerably more challenging. High-capacity models only make sense when we train on large and diverse datasets – small, narrow datasets simply do not require this much capacity and do not benefit from it. While prior works used simulation to create such datasets, the most representative data comes from the real world. Therefore, we focus on reinforcement learning methods that can use Transformers and incorporate large, previously collected datasets via offline RL. Offline RL methods train on prior data, aiming to derive the most effective possible policy from a given dataset. Of course, this dataset can be augmented with additionally autonomously gathered data, but the training is separated from data collection, providing an appealing workflow for large-scale robotics applications.
将 Transformer 模型应用于强化学习的另一个问题是设计能够有效训练此类模型的强化学习系统。有效的离线强化学习方法通常通过时间差分更新进行 Q 函数估计。由于 Transformer 对离散词元序列进行建模，我们将 Q 函数估计问题转化为离散词元序列建模问题，并为序列中的每个词元设计合适的损失函数。简单地对动作空间进行离散化会导致动作基数的指数爆炸，因此我们采用逐维度离散化方案，其中动作空间的每个维度被视为强化学习的一个单独时间步。离散化中的不同区间对应不同的动作。逐维度离散化方案使我们能够使用简单的离散动作 Q 学习方法，并结合保守正则化器来处理分布偏移。我们提出了一种特定的正则化器，它最小化数据集中未采取的每个动作的值，并表明我们的方法可以从狭窄的演示类数据和具有探索噪声的更广泛数据中学习。最后，我们利用一种混合更新，将蒙特卡洛和 \(n\) 步回报与时间差分备份相结合，并表明这样做提高了我们基于 Transformer 的离线强化学习方法在大规模机器人学习问题上的性能。
Another issue in applying Transformer models to RL is to design RL systems that can effectively train such models. Effective offline RL methods generally employ Q-function estimation via temporal difference updates. Since Transformers model discrete token sequences, we convert the Q-function estimation problem into a discrete token sequence modeling problem, and devise a suitable loss function for each token in the sequence. Naively discretizing the action space leads to exponential blowup in action cardinality, so we employ a per-dimension discretization scheme, where each dimension of the action space is treated as a separate time step for RL. Different bins in the discretization correspond to distinct actions. The per-dimension discretization scheme allows us to use simple discrete-action Q-learning methods with a conservative regularizer to handle distributional shift. We propose a specific regularizer that minimizes values of every action that was not taken in the dataset and show that our method can learn from both narrow demonstration-like data and broader data with exploration noise. Finally, we utilize a hybrid update that combines Monte Carlo and \(n\)-step returns with temporal difference backups, and show that doing so improves the performance of our Transformer-based offline RL method on large-scale robotic learning problems.
总之，我们的主要贡献是 Q-Transformer，一种用于机器人离线强化学习的基于 Transformer 的架构，它利用 Q 值的逐维度词元化，并且可以轻松应用于大规模多样化的机器人数据集，包括真实世界数据。我们在图 1 中总结了 Q-Transformer 的组成部分。我们的实验评估通过大规模文本条件多任务策略的学习验证了 Q-Transformer，既在仿真中进行严格比较，也在大规模真实世界实验中进行现实验证。我们的真实世界实验使用了包含 38,000 个成功演示和 20,000 个自主收集的失败片段的数据集，涉及 700 多个任务，由 13 个机器人组成的车队收集。Q-Transformer 优于先前提出的大规模机器人强化学习架构，以及先前提出的基于 Transformer 的模型，如 Decision Transformer。
In summary, our main contribution is the Q-Transformer, a Transformer-based architecture for robotic offline reinforcement learning that makes use of per-dimension tokenization of Q-values and can readily be applied to large and diverse robotic datasets, including real-world data. We summarize the components of Q-Transformer in Figure 1. Our experimental evaluation validates the Q-Transformer by learning large-scale text-conditioned multi-task policies, both in simulation for rigorous comparisons and in large-scale real-world experiments for realistic validation. Our real-world experiments utilize a dataset with 38,000 successful demonstrations and 20,000 failed autonomously collected episodes on more than 700 tasks, gathered with a fleet of 13 robots. Q-Transformer outperforms previously proposed architectures for large-scale robotic RL, as well as previously proposed Transformer-based models such as the Decision Transformer.
离线强化学习（offline RL）在近期的研究中已被广泛探讨。保守 Q 学习（CQL）学习受限于价值函数保守下界的策略。我们的目标并非为离线 RL 开发新的算法原理，而是设计一个能与高容量 Transformer 集成并扩展到现实世界多任务机器人学习的离线 RL 系统。为此，我们开发了一个 CQL 版本，特别适用于在混合质量数据上训练基于 Transformer 的大型 Q 函数。尽管一些工作指出模仿学习在演示数据上优于离线 RL，但其他工作表明离线 RL 技术在理论和实践中均能有效利用演示。然而，将“狭窄”的演示数据与“广泛”的次优（例如自主收集的）数据相结合的场景被认为特别困难，尽管这在许多机器人学习环境中很自然，因为我们可能希望用相对廉价的低质量自主收集数据来扩充核心演示集。我们相信，我们的方法在此场景中的有效性对实践者尤其具有吸引力。
Offline RL has been extensively studied in recent works. Conservative Q-learning (CQL) learns policies constrained to a conservative lower bound of the value function. Our goal is not to develop a new algorithmic principle for offline RL, but to devise an offline RL system that can integrate with high-capacity Transformers and scale to real-world multi-task robotic learning. We thus develop a version of CQL particularly effective for training large Transformer-based Q-functions on mixed quality data. While some works have noted that imitation learning outperforms offline RL on demonstration data, other works showed offline RL techniques to be effective with demonstrations both in theory and in practice. Nonetheless, a setting that combines “narrow” demonstration data with “broad” sub-optimal (e.g., autonomously collected) data is known to be particularly difficult, though it is quite natural in many robotic learning settings where we might want to augment a core set of demonstrations with relatively inexpensive low-quality autonomously collected data. We believe that the effectiveness of our method in this setting is of particular interest to practitioners.
基于 Transformer 的架构已在近期机器人研究中得到探索，既用于学习可泛化的任务空间，也用于直接学习多任务甚至多领域的序列策略。尽管这些工作大多在监督学习设置中考虑 Transformer（例如从演示中学习），但也有工作将 Transformer 用于强化学习和条件模仿学习。在我们的实验中，我们特别与决策 Transformer（DT）进行了比较，DT 通过奖励条件扩展了条件模仿学习以使用序列模型，并在结构上类似于已成功用于机器人控制的模仿学习方法。尽管 DT 融入了 RL 的元素（即奖励函数），但它并未提供机制来改进演示行为或重新组合数据集部分以合成更优行为，并且确实已知存在理论局限性。另一方面，此类基于模仿的方法之所以流行，或许是因为将 Transformer 架构与更强大的时间差分方法（如 Q 学习）集成存在困难。我们展示了若干简单但重要的设计决策对于实现这一目标至关重要，并且我们的方法在大规模多任务机器人控制评估中显著优于非 TD 方法（如 DT）以及模仿学习。扩展决策 Transformer 的工作提出了将 Q 函数与基于 Transformer 的策略结合使用，但 Q 函数本身并未采用基于 Transformer 的架构。原则上，我们的 Q 函数可以与该方法结合，但我们的重点特别在于直接训练 Transformer 来表示 Q 值。
Transformer-based architectures have been explored in recent robotics research, both to learn generalizable task spaces and to learn multi-task or even multi-domain sequential policies directly. Although most of these works considered Transformers in a supervised learning setting, e.g., learning from demonstrations, there are works on employing Transformers for RL and conditional imitation learning. In our experiments, we compare to Decision Transformer (DT) in particular, which extends conditional imitation learning with reward conditioning to use sequence models, and structurally resembles imitation learning methods that have been used successfully for robotic control. Although DT incorporates elements of RL (namely, reward functions), it does not provide a mechanism to improve over the demonstrated behavior or recombine parts of the dataset to synthesize more optimal behaviors, and indeed is known to have theoretical limitations. On the other hand, such imitation-based recipes are popular perhaps due to the difficulty of integrating Transformer architectures with more powerful temporal difference methods (e.g., Q-learning). We show that several simple but important design decisions are needed to make this work, and our method significantly outperforms non-TD methods such as DT, as well as imitation learning, on our large-scale multi-task robotic control evaluation. Extending Decision Transformer, proposed to use a Q-function in combination with a Transformer-based policy, but the Q-function itself did not use a Transformer-based architecture. Our Q-function could in principle be combined with this method, but our focus is specifically on directly training Transformers to represent Q-values.
为了开发基于 Transformer 的 Q 学习方法，我们将动作空间的每个维度离散化，每个维度作为一个独立的时间步。自回归生成离散动作已被探索，他们提出了 MDP 的层次分解，然后利用 LSTM 进行自回归离散化。我们的离散化方案类似但更简单，因为我们不使用任何层次分解，而是简单地将每个维度视为一个时间步。然而，由于我们的目标是在现实世界基于图像的任务上大规模执行离线 RL（相对于通过在线 RL 学习的较小状态空间任务），我们提出了若干额外的设计决策来施加保守正则化，从而能够大规模训练我们基于 Transformer 的离线 Q 学习方法，提供一个完整的机器人学习系统。
To develop a Transformer-based Q-learning method, we discretize each action space dimension, with each dimension acting as a distinct time step. Autoregressive generation of discrete actions has been explored by , who propose a hierarchical decomposition of an MDP and then utilize LSTM for autoregressive discretization. Our discretization scheme is similar but simpler, in that we do not use any hierarchical decomposition but simply treat each dimension as a time step. However, since our goal is to perform offline RL at scale with real-world image based tasks (vs. the smaller state-space tasks learned via online RL by ), we present a number of additional design decisions to impose a conservative regularizer, enabling training our Transformer-based offline Q-learning method at scale, providing a complete robotic learning system.
在强化学习（RL）中，我们学习策略 \(\pi\)，以最大化马尔可夫决策过程（MDP）中的期望总回报，其中状态为 \(s\)，动作为 \(a\)，折扣因子为 \(\gamma\in(0,1]\)，转移函数为 \(T(s'|s,a)\)，奖励函数为 \(R(s,a)\)。动作 \(a\) 的维度为 \(d_{\mathcal{A}}\)。基于价值的强化学习方法学习一个 Q 函数 \(Q(s,a)\)，表示总折扣回报 \(\sum_{t}\gamma^{t}R(s_{t},a_{t})\)，策略为 \(\pi(a|s)=\arg\max_{a}Q(s,a)\)。Q 函数可以通过迭代应用贝尔曼算子来学习：
In RL, we learn policies \(\pi\) that maximize the expected total reward in a Markov decision process (MDP) with states \(s\), actions \(a\), discount factor \(\gamma\in(0,1]\), transition function \(T(s'|s,a)\), and a reward function \(R(s,a)\). Actions \(a\) have dimensionality \(d_{\mathcal{A}}\). Value-based RL approaches learn a Q-function \(Q(s,a)\) representing the total discounted return \(\sum_{t}\gamma^{t}R(s_{t},a_{t})\), with policy \(\pi(a|s)=\arg\max_{a}Q(s,a)\). The Q-function can be learned by iteratively applying the Bellman operator:
通过函数逼近和采样进行近似。离线强化学习设置假设可以访问由某个未知行为策略 \(\pi_{\beta}(a|s)\) 生成的转移或回合的离线数据集，但不假设在训练期间能够进行额外的在线交互。这对于现实世界的机器人学习很有吸引力，因为在线策略数据收集非常耗时。从离线数据集学习需要解决分布偏移问题，因为通常使 \(Q(s_{t+1},a_{t+1})\) 最大化的动作可能位于数据分布之外。缓解此问题的一种方法是添加保守惩罚，降低数据集中任何动作 \(a\) 的 Q 值 \(Q(s,a)\)，从而确保最大价值动作在分布内。
approximated via function approximation and sampling. The offline RL setting assumes access to an offline dataset of transitions or episodes, produced by some unknown behavior policy \(\pi_{\beta}(a|s)\), but does not assume the ability to perform additional online interaction during training. This is appealing for real-world robotic learning, where on-policy data collection is time-consuming. Learning from offline datasets requires addressing distributional shift, since in general the action that maximizes \(Q(s_{t+1},a_{t+1})\) might lie outside of the data distribution. One approach to mitigate this is to add a conservative penalty that pushes down the Q-values \(Q(s,a)\) for any action \(a\) outside of the dataset, thus ensuring that the maximum value action is in-distribution.
在这项工作中，我们考虑具有稀疏奖励的任务，其中在回合的最后时间步分配二元奖励 \(R\in\{0,1\}\)（表示成功或失败）。尽管我们的方法并非专门针对这种设置，但这种奖励结构在机器人操作任务中很常见，这些任务在每个回合要么成功要么失败，并且由于缺乏奖励塑形，对强化学习来说可能特别具有挑战性。
In this work, we consider tasks with sparse rewards, where a binary reward \(R\in\{0,1\}\) (indicating success or failure) is assigned at the last time step of episodes. Although our method is not specific to this setting, such reward structure is common in robotic manipulation tasks that either succeed or fail on each episode, and can be particularly challenging for RL due to the lack of reward shaping.
在本节中，我们介绍 Q-Transformer，一种用于离线 Q 学习的 Transformer 架构，它基于三个主要组成部分。首先，我们描述如何应用离散化和自回归来使 Transformer 架构能够进行 TD 学习。其次，我们引入一种特定的保守 Q 函数正则化器，使其能够从离线数据集中学习。最后，我们展示如何使用蒙特卡洛和\(n\)步回报来提高学习效率。
In this section, we introduce Q-Transformer, an architecture for offline Q-learning with Transformer models, which is based on three main ingredients. First, we describe how we apply discretization and autoregression to enable TD-learning with Transformer architectures. Next, we introduce a particular conservative Q-function regularizer that enables learning from offline datasets. Lastly, we show how Monte Carlo and \(n\)-step returns can be used to improve learning efficiency.
将 Transformer 与 Q 学习结合使用面临两个挑战：(1) 我们必须对输入进行分词以有效应用注意力机制，这需要对动作空间进行离散化；(2) 我们必须对离散动作进行 Q 值最大化，同时避免维度灾难。在标准 Q 学习框架内解决这些问题需要新的建模决策。我们自回归 Q 学习更新的直觉是将每个动作维度视为本质上独立的时间步。这样，我们可以对单个维度（一维量）进行离散化，而不是对整个动作空间，从而避免维度灾难。这可以看作是对 [参考文献] 中提出的方案的简化版本，但我们将其应用于高容量 Transformer 模型，扩展到离线强化学习设置，并扩展到真实世界的机器人学习。
Using Transformers with Q-learning presents two challenges: (1) we must tokenize the inputs to effectively apply attention mechanisms, which requires discretizing the action space; (2) we must perform maximization of Q-values over discretized actions while avoiding the curse of dimensionality. Addressing these issues within the standard Q-learning framework requires new modeling decisions. The intuition behind our autoregressive Q-learning update is to treat each action dimension as essentially a separate time step. That way, we can discretize individual dimensions (1D quantities), rather than the entire action space, avoiding the curse of dimensionality. This can be viewed as a simplified version of the scheme proposed in [reference], though we apply this to high-capacity Transformer models, extend it to the offline RL setting, and scale it up to real-world robotic learning.
设 \(\tau=(s_{1},a_{1},\dots,s_{T},a_{T})\) 为来自离线数据集 \(\mathcal{D}\) 的长度为 \(T\) 的机器人经验轨迹。对于给定的时间步 \(t\) 以及轨迹中对应的动作 \(a_{t}\)，我们定义动作 \(a_{t}\) 的逐维度视图。设 \(a^{1:i}_{t}\) 表示从第一个维度 \(a^{1}_{t}\) 到第 \(i\) 个维度 \(a^{i}_{t}\) 的动作维度向量，其中 \(i\) 可以从 1 到动作维度的总数，我们将其记为 \(d_{\mathcal{A}}\)。然后，对于状态历史的时间窗口 \(w\)，我们使用自回归 Q 函数定义第 \(i\) 个维度上动作 \(a^{i}_{t}\) 的 Q 值，该函数以该时间窗口 \(s_{t-w:t}\) 的状态和当前时间步的先前动作维度 \(a_{t}^{1:i-1}\) 为条件。为了训练 Q 函数，我们定义了逐维度的贝尔曼更新。对于所有维度 \(i\in\{1,\dots,d_{\mathcal{A}}\}\)：
Let \(\tau=(s_{1},a_{1},\dots,s_{T},a_{T})\) be a trajectory of robotic experience of length \(T\) from an offline dataset \(\mathcal{D}\) . For a given time-step \(t\) , and the corresponding action \(a_{t}\) in the trajectory, we define a per-dimension view of the action \(a_{t}\) . Let \(a^{1:i}_{t}\) denote the vector of action dimensions from the first dimension \(a^{1}_{t}\) until the \(i\) -th dimension \(a^{i}_{t}\) , where \(i\) can range from \(1\) to the total number of action dimensions, that we denote as \(d_{\mathcal{A}}\) . Then, for a time window \(w\) of state history, we define the Q-value of the action \(a^{i}_{t}\) in the \(i-th\) dimension using an autoregressive Q-function conditioned on states from this time window \(s_{t-w:t}\) and previous action dimensions for the current time step \(a_{t}^{1:i-1}\) . To train the Q-function, we define a per-dimension Bellman update. For all dimensions \(i\in\{1,\dots,d_{\mathcal{A}}\}\) :
奖励仅应用于最后一个维度（方程中的第二行），因为我们在执行整个动作之前不会收到任何奖励。此外，我们仅在时间步之间对 Q 值进行折扣，并在每个时间步内对所有非最后一个维度保持折扣为 1.0，以确保与原始 MDP 相同的折扣。图 2 说明了这一过程，其中每个黄色框代表 Q 目标计算，并包含下一小节描述的额外保守性和蒙特卡洛回报。需要注意的是，通过将每个动作维度视为贝尔曼更新的时间步，我们不会改变 Q 学习算法的一般优化性质，并且对于给定的 MDP，贝尔曼最优性原理仍然成立，因为我们在给定未来所有动作维度最优的情况下对动作维度进行最大化。我们在附录 A 中证明了该方法提供了优化原始 MDP 的理论一致方式，并在附录 B 中给出了表格设置下的收敛性证明。
The reward is only applied on the last dimension (second line in the equation), as we do not receive any reward before executing the whole action. In addition, we only discount Q-values between the time steps and keep discounting at \(1.0\) for all but the last dimension within each time step, to ensure the same discounting as in the original MDP. Figure 2 illustrates this process, where each yellow box represents the Q-target computation with additional conservatism and Monte Carlo returns described in the next subsections. It should be noted that by treating each action dimension as a time step for the Bellman update, we do not change the general optimization properties of Q-learning algorithms and the principle of the Bellman optimality still holds for a given MDP as we maximize over an action dimension given the optimality of all action dimensions in the future. We show that this approach provides a theoretically consistent way to optimize the original MDP in Appendix A, with a proof of convergence in the tabular setting in Appendix B.
在定义了用于 Transformer 的 Q 学习的 Bellman 备份之后，我们现在开发一种技术，使其能够从离线数据中学习，包括人类演示和自主收集的数据。这通常需要解决由于分布偏移导致的过估计问题，即当目标值的 Q 函数在与训练时不同的动作上被查询时。保守 Q 学习（CQL）在分布外动作上最小化 Q 函数，这可能导致 Q 值显著小于任何轨迹中可获得的最小可能累积奖励。当处理稀疏奖励 \(R\in\{0,1\}\) 时，结果表明，使用标准保守目标正则化的 Q 函数可能取负值，即使瞬时奖励都是非负的。本节提出了一种改进的保守 Q 学习版本，以解决我们问题设置中的这一问题。
Having defined a Bellman backup for running Q-learning with Transformers, we now develop a technique that enables learning from offline data, including human demonstrations and autonomously collected data. This typically requires addressing over-estimation due to the distributional shift, when the Q-function for the target value is queried at an action that differs from the one on which it was trained. Conservative Q-learning (CQL) minimizes the Q-function on out-of-distribution actions, which can result in Q-values that are significantly smaller than the minimal possible cumulative reward that can be attained in any trajectory. When dealing with sparse rewards \(R\in\{0,1\}\), results in show that the Q-function regularized with a standard conservative objective can take on negative values, even though instantaneous rewards are all non-negative. This section presents a modified version of conservative Q-learning that addresses this issue in our problem setting.
我们设计的关键见解是，与其最小化数据中未出现的动作的 Q 值，不如将这些 Q 值正则化到接近最小可能累积奖励。具体来说，将任务的最小可能奖励记为 \(R_{\text{min}}\)，任务的时间范围记为 \(T\)，我们的方法将数据集中未覆盖的动作的 Q 值正则化到 \(R_{\text{min}}\cdot T\)，在我们的问题设置中该值等于 \(0\)（即 \(R_{\text{min}}=0\)）。为简化符号，我们在呈现目标时省略了动作维度索引，但需注意，以下训练目标应用于上一节所述的所有动作维度的 Bellman 备份。设 \(\pi_{\beta}\) 为生成给定数据集 \(\mathcal{D}\) 的行为策略，并设 \(\tilde{\pi}_{\beta}(a|s)=\frac{1}{Z(s)}\cdot\left(1.0-\pi_{\beta}(a|s)\right)\) 为在 \(\pi_{\beta}(a|s)\) 下密度非常低的所有动作上的分布。我们训练 Q 函数的目标是：
The key insight behind our design is that, rather than minimizing the Q-values on actions not in the data, we can instead regularize these Q-values to be close to the minimal attainable possible cumulative reward. Concretely, denoting the minimal possible reward on the task as \(R_{\text{min}}\), and the time horizon of the task as \(T\), our approach regularizes the Q-values on actions not covered by the dataset towards \(R_{\text{min}}\cdot T\), which in our problem setting is equal to \(0\) (i.e., \(R_{\text{min}}=0\)). For simplicity of notation, we omit the action dimension indices in presenting the resulting objective, but remark that the training objective below is applied to Bellman backups on all action dimensions as described in the previous section. Let \(\pi_{\beta}\) be the behavioral policy that induced a given dataset \(\mathcal{D}\), and let \(\tilde{\pi}_{\beta}(a|s)=\frac{1}{Z(s)}\cdot\left(1.0-\pi_{\beta}(a|s)\right)\) be the distribution over all actions which have a very low density under \(\pi_{\beta}(a|s)\). Our objective to train the Q-function is:
其中第一项 \((i)\) 通过最小化公式 1 中定义的时序差分误差目标来训练 Q 函数，第二项 \((ii)\) 在 \(\tilde{\pi}_{\beta}\) 诱导的动作分布下，将 Q 值正则化到最小可能 Q 值 \(0\) 的期望，我们将其称为保守正则化项 \(\mathcal{L}_{C}\)。项 \((ii)\) 还乘以一个乘数 \(\alpha\)，该乘数调节保守正则化的强度。我们在附录 D.2 中讨论了 \(\alpha\) 的选择，并在附录 C 中分析了保守项的行为，提供了该正则化器在表格设置中如何修改学习到的 Q 函数的简单表征。
where the first term \((i)\) trains the Q-function by minimizing the temporal difference error objective as defined in Eq. 1, and the second term \((ii)\) regularizes the Q-values to the minimal possible Q-value of \(0\) in expectation under the distribution of actions induced by \(\tilde{\pi}_{\beta}\), which we denote as a conservative regularization term \(\mathcal{L}_{C}\). Term \((ii)\) is also weighted by a multiplier \(\alpha\), which modulates the strength of this conservative regularization. We discuss the choice of \(\alpha\) in our implementation in Appendix D.2 and analyze the behavior of the conservatism term in Appendix C, providing a simple characterization of how this regularizer modifies the learned Q-function in tabular settings.
当数据集中既包含一些优质轨迹（例如示范）又包含一些次优轨迹（例如自主采集的试验）时，利用蒙特卡洛回报估计来加速 Q 学习可以带来显著的性能提升，因为沿着较优轨迹的蒙特卡洛估计会导致更快的价值传播。这一现象在先前的工作中也已被观察到。基于这一观察，我们提出了对 Q-Transformer 的一个简单改进，并在实践中发现其相当有效。蒙特卡洛回报定义为离线轨迹 \(\tau\) 内的累积奖励：\(\text{MC}_{t:T}=\sum_{j=t}^{T}\gamma^{j-t}R(s_{j},a_{j})\)。这与行为策略 \(\pi_{\beta}\) 的 Q 值相匹配，并且由于最优 \(Q^{*}(s,a)\) 大于任何其他策略的 Q 值，我们有 \(Q^{*}(s_{t},a_{t})\geq\text{MC}_{t:T}\)。由于蒙特卡洛回报是最优 Q 函数的下界，我们可以增强贝尔曼更新，使其取 MC 回报与当前 Q 值之间的最大值：\(\max\left(\text{MC}_{t:T},Q(s_{t},a_{t})\right)\)，而不会改变贝尔曼更新的收敛目标。
When the dataset contains some good trajectories (e.g., demonstrations) and some suboptimal trajectories (e.g., autonomously collected trials), utilizing Monte Carlo return-to-go estimates to accelerate Q-learning can lead to significant performance improvements, as the Monte Carlo estimates along the better trajectories lead to much faster value propagation. This has also been observed in prior work. Based on this observation, we propose a simple improvement to Q-Transformer that we found to be quite effective in practice. The Monte Carlo return is defined by the cumulative reward within the offline trajectory \(\tau\): \(\text{MC}_{t:T}=\sum_{j=t}^{T}\gamma^{j-t}R(s_{j},a_{j})\). This matches the Q-value of the behavior policy \(\pi_{\beta}\), and since the optimal \(Q^{*}(s,a)\) is larger than the Q-value for any other policy, we have \(Q^{*}(s_{t},a_{t})\geq\text{MC}_{t:T}\). Since the Monte Carlo return is a lower bound of the optimal Q-function, we can augment the Bellman update to take the maximum between the MC-return and the current Q-value: \(\max\left(\text{MC}_{t:T},Q(s_{t},a_{t})\right)\), without changing what the Bellman update will converge to.
尽管这不会改变收敛性，但包含这种最大化操作可以加速学习（见第 5.3 节）。我们提出了一个关于为何会发生这种现象的假设。在实践中，最终时间步 \((s_{T},a_{T})\) 的 Q 值首先被学习，然后在后续的梯度步骤中向后传播。Q 值可能需要多个梯度才能一直传播到 \((s_{1},a_{1})\)。\(\max(\text{MC},Q)\) 允许我们在训练开始时、在 Q 值传播之前就对 \(Q(s_{1},a_{1})\) 施加有用的梯度。
Although this does not change convergence, including this maximization speeds up learning (see Section 5.3). We present a hypothesis why this occurs. In practice, Q-values for final timesteps \((s_{T},a_{T})\) are learned first and then propagated backwards in future gradient steps. It can take multiple gradients for the Q-value to propagate all the way to \((s_{1},a_{1})\). The \(\max(\text{MC},Q)\) allows us to apply useful gradients to \(Q(s_{1},a_{1})\) at the start of training before the Q-values have propagated.
在我们的实验中，我们还注意到，在动作维度上额外采用 \(n\)-步回报可以显著提高学习速度。我们选择 \(n\) 使得下一步最后一个维度的最终 Q 值被用作 Q 目标。这是因为我们只有在推断并执行整个动作之后才能获得新的状态和奖励，而不是动作的一部分，这意味着中间奖励在最后一个动作维度之前一直保持为 0。虽然这会给贝尔曼备份引入偏差，正如使用 \(n\)-步回报的离策略学习通常会出现的情况一样，但我们在第 5.3 节的消融研究中发现，这种偏差的有害影响很小，而训练速度的提升却非常显著。这与先前报告的结果一致。关于我们的 Transformer 序列模型架构（如图 3 所示）、保守 Q 学习实现以及机器人系统的更多细节，可在附录 D 中找到。
In our experiments, we also notice that additionally employing \(n\)-step returns over action dimensions can significantly help with the learning speed. We pick \(n\) such that the final Q-value of the last dimension of the next time step is used as the Q-target. This is because we get a new state and reward only after inferring and executing the whole action as opposed to parts of it, meaning that intermediate rewards remain 0 all the way until the last action dimension. While this introduces bias to the Bellman backups, as is always the case with off-policy learning with \(n\)-step returns, we find in our ablation study in Section 5.3 that the detrimental effects of this bias are small, while the speedup in training is significant. This is consistent with previously reported results. More details about our Transformer sequence model architecture (depicted in Figure 3), conservative Q-learning implementation, and the robot system can be found in Appendix D.
在我们的实验中，我们旨在回答以下问题：（1）Q-Transformer 能否从演示和次优数据的组合中学习？（2）Q-Transformer 与其他方法相比如何？（3）Q-Transformer 中的特定设计选择有多重要？（4）Q-Transformer 能否应用于大规模真实世界机器人操作问题？
In our experiments, we aim to answer the following questions: (1) Can Q-Transformer learn from a combination of demonstrations and sub-optimal data? (2) How does Q-Transformer compare to other methods? (3) How important are the specific design choices in Q-Transformer? (4) Can Q-Transformer be applied to large-scale real world robotic manipulation problems?
训练数据集。我们实验中使用的离线数据由 13 台机器人组成的团队收集，包含[参考文献]所述演示数据的一个子集，并结合了较低质量的自主收集数据。演示数据通过人类远程操作收集，涵盖超过 700 个不同任务，每个任务都有独立的语言描述。每个任务最多使用 100 条演示，总计约 38,000 条演示。所有这些演示均成功完成各自任务，并获得 1.0 的奖励。数据集的其余部分通过自主运行机器人、执行基于行为克隆学习的策略来收集。
Training dataset. The offline data used in our experiments was collected with a fleet of 13 robots, and consists of a subset of the demonstration data described by [reference], combined with lower quality autonomously collected data. The demonstrations were collected via human teleoperation for over 700 distinct tasks, each with a separate language description. We use a maximum of 100 demonstrations per task, for a total of about 38,000 demonstrations. All of these demonstrations succeed on their respective tasks and receive a reward of 1.0. The rest of the dataset was collected by running the robots autonomously, executing policies learned via behavioral cloning.
为确保 Q-Transformer 与模仿学习方法之间的公平比较，在训练我们的方法时，我们丢弃了自主收集数据中所有成功的回合，以确保通过包含自主数据，Q-Transformer 不会比模仿学习基线观察到更多成功的试验。这样我们额外获得了约 20,000 个自主收集的失败回合，每个回合奖励为 0.0，数据集总规模约为 58,000 个回合。回合平均长度为 35 个时间步。任务示例见图 4。
To ensure a fair comparison between Q-Transformer and imitation learning methods, we discard all successful episodes in the autonomously collected data when we train our method, to ensure that by including the autonomous data the Q-Transformer does not get to observe more successful trials than the imitation learning baselines. This leaves us with about 20,000 additional autonomously collected failed episodes, each with a reward of 0.0, for a dataset size of about 58,000 episodes. The episodes are on average 35 time steps in length. Examples of the tasks are shown in Figure 4.
性能评估。为了评估 Q-Transformer 在从真实世界离线数据集学习并有效整合自主收集的失败回合时的表现，我们在 72 个独特的操作任务上评估 Q-Transformer，涵盖多种技能，如“抽屉取放”、“开关抽屉”、“将物体移近目标”，分别包含 18、7 和 48 个独特的任务指令，以指定不同的物体组合和抽屉。因此，表 4 中的平均成功率是 72 个任务的平均值。
Performance evaluation. To evaluate how well Q-Transformer can perform when learning from real-world offline datasets while effectively incorporating autonomously collected failed episodes, we evaluate Q-Transformer on 72 unique manipulation tasks, and a variety of different skills, such as “drawer pick and place”, “open and close drawer”, “move object near target”, each consisting of 18, 7, and 48 unique task instructions respectively to specify different object combinations and drawers. As such, the average success rate in Table 4 is the average over 72 tasks.
由于训练集中每个任务最多只有 100 条演示，我们从图 4 观察到，像 RT-1 这样的模仿学习算法（也使用类似的 Transformer 架构）在仅从有限的成功机器人演示中学习时难以获得良好性能。现有的离线强化学习方法，如 IQL 和基于 Transformer 的方法如 Decision Transformer，可以从成功演示和失败回合中学习，并显示出比 RT-1 更好的性能，尽管提升幅度相对较小。Q-Transformer 取得了最高的成功率，优于行为克隆基线（RT-1）和离线强化学习基线（Decision Transformer、IQL），平均性能比之前最佳方法高出约 70%。这表明 Q-Transformer 能够利用自主收集的次优数据有效改进人类演示。
Since each task in the training set only has a maximum of 100 demonstrations, we observe from Figure 4 that an imitation learning algorithm like RT-1, which also uses a similar Transformer architecture, struggles to obtain a good performance when learning from only the limited pool of successful robot demonstrations. Existing offline RL methods, such as IQL and a Transformer-based method such as Decision Transformer, can learn from both successful demonstrations and failed episodes, and show better performance compared to RT-1, though by a relatively small margin. Q-Transformer has the highest success rate and outperforms both the behavior cloning baseline (RT-1) and offline RL baselines (Decision Transformer, IQL), exceeding the average performance of the best-performing prior method by about 70%. This demonstrates that Q-Transformer can effectively improve upon human demonstrations using autonomously collected sub-optimal data.
附录 8 还表明，Q-Transformer 可以与最近提出的语言任务规划器成功结合，用于执行可供性估计和机器人动作执行。Q-Transformer 在规划和执行长时程任务方面优于先前方法。
Appendix 8 also shows that Q-Transformer can be successfully applied in combination with a recently proposed language task planner to perform both affordance estimation and robot action execution. Q-Transformer outperforms prior methods for planning and executing long-horizon tasks.
在本节中，我们在一个具有挑战性的仿真离线强化学习任务上评估 Q-Transformer，该任务需要结合次优数据来解决。具体来说，我们使用图 5 中描绘的视觉仿真抓取任务，其中包含少量位置控制的人类演示（约占数据的 8%）。这些演示通过添加噪声进行重放，以生成更多轨迹（约占数据的 92%）。图 5 展示了与多种离线算法的对比，例如使用 CQL 的 QT-Opt、IQL、AW-Opt 和 Decision Transformer，以及仅使用演示进行行为克隆的 RT-1。正如我们所见，能够有效执行 TD 学习以结合最优和次优数据的算法（如 Q-Transformer 和 QT-Opt）表现优于其他算法。使用 RT-1 的行为克隆无法利用次优数据。Decision Transformer 在演示和次优数据上训练，但无法利用噪声数据进行策略改进，最终表现不如我们的方法。尽管 IQL 和 AW-Opt 执行 TD 学习，但 actor 仍然过于接近数据，无法充分利用次优数据。Q-Transformer 既能从演示中引导策略，又能通过 TD 学习传播信息快速改进。我们还在附录 F 中通过多个随机种子训练来评估结果的统计显著性。
In this section, we evaluate Q-Transformer on a challenging simulated offline RL task that requires incorporating sub-optimal data to solve the task. In particular, we use a visual simulated picking task depicted in Figure 5, where we have a small amount of position controlled human demonstrations (approximately 8% of the data). The demonstrations are replayed with noise to generate more trajectories (approximately 92% of the data). Figure 5 shows a comparison to several offline algorithms, such as QT-Opt with CQL, IQL, AW-Opt, and Decision Transformer, along with RT-1 using Behavioral Cloning on demonstrations only. As we see, algorithms that can effectively perform TD-learning to combine optimal and sub-optimal data (such as Q-Transformer and QT-Opt) perform better than others. BC with RT-1 is not able to take advantage of sub-optimal data. Decision Transformer is trained on both demonstrations and sub-optimal data, but is not able to leverage the noisy data for policy improvement and does not end up performing as well as our method. Although IQL and AW-Opt perform TD-learning, the actor remains too close to the data and cannot fully leverage the sub-optimal data. Q-Transformer is able to both bootstrap the policy from demonstrations and also quickly improve through propagating information with TD-learning. We also analyze the statistical significance of the results by training with multiple random seeds in Appendix F.
我们在仿真中对我们方法的设计选择进行了一系列消融实验，结果如图 6（左）所示。首先，我们证明了我们对 Q-Transformer 的保守性选择优于标准的 CQL 正则化器，后者对应于在 Q 函数输出之上添加一个 softmax 层，并在数据集动作与该 softmax 输出之间使用交叉熵损失。该正则化器与我们提出的正则化器作用类似，即降低分布外动作的 Q 值，并更贴近行为策略。
We perform a series of ablations of our method design choices in simulation, with results presented in Figure 6 (left). First, we demonstrate that our choice of conservatism for Q-Transformer performs better than the standard CQL regularizer, which corresponds to a softmax layer on top of the Q-function outputs with a cross-entropy loss between the dataset action and the output of this softmax. This regularizer plays a similar role to the one we propose, decreasing the Q-values for out-of-distribution actions and staying closer to the behavior policy.
如图 6（左）所示，使用 softmax 保守性的性能下降到大约演示片段的比例（约 8%）。这表明，当保守性惩罚过于有效地约束行为策略分布时，会导致策略崩溃到行为策略。由于 softmax 的性质，降低未观测动作的 Q 值同时也会提高已观测动作的 Q 值，我们推测这使得难以将次优的分布内动作的 Q 值保持在较低水平，而这些动作未能获得高奖励。接下来，我们展示了使用保守性的重要性。当完全移除保守性时，我们观察到性能崩溃。数据集中稀有的动作会被高估 Q 值，因为它们没有经过离线 Q 学习过程的训练。由此产生的高估值会传播并导致整个 Q 函数崩溃，如先前工作所述。最后，我们对蒙特卡洛回报进行了消融，再次观察到性能崩溃。这表明，加入采样未来回报的信息显著有助于引导大型架构（如 Transformer）的训练。
As we see in Figure 6 (left), performance with softmax conservatism drops to around the fraction of demonstration episodes ( \(\sim\) 8%). This suggests a collapse to the behavior policy as the conservatism penalty becomes too good at constraining to the behavior policy distribution. Due to the nature of the softmax, pushing Q-values down for unobserved actions also pushes Q-values up for the observed actions, and we theorize this makes it difficult to keep Q-values low for sub-optimal in-distribution actions that fail to achieve high reward. Next, we show that using conservatism is important. When removing conservatism entirely, we observe that performance collapses. Actions that are rare in the dataset will have overestimated Q-values, since they are not trained by the offline Q-learning procedure. The resulting overestimated values will propagate and collapse the entire Q-function, as described in prior work. Finally, we ablate the Monte-Carlo returns and again observe performance collapse. This demonstrates that adding information about the sampled future returns significantly helps in bootstrapping the training of large architectures such as Transformers.
我们还在真实机器人上对第 4.3 节中 \(n\) 步回报的选择进行了消融实验，观察到使用 \(n\) 步回报相比使用 1 步回报，在梯度步数和墙钟时间上显著加快了训练速度，且性能损失极小，如图 6（右上）所示。
We also ablate the choice of \(n\) -step returns from the Section 4.3 on real robots and observe that using \(n\) -step returns leads to a significantly faster training speed as measured by the number of gradient steps and wall clock time compared to using 1-step returns, with a minimal loss in performance, as shown in Figure 6 (top right).
上一节的实验使用了包含成功演示和失败自主尝试的大型数据集，其规模与之前一些利用演示数据的最大实验相当。我们还进行了一项初步实验，使用更大的数据集来研究 Q-Transformer 在数据集规模扩大时的性能表现。
The experiments in the previous section used a large dataset that included successful demonstrations and failed autonomous trials, comparable in size to some of the largest prior experiments that utilized demonstration data. We also carry out a preliminary experiment with a much larger dataset to investigate the performance of Q-Transformer as we scale up the dataset size.
该实验包含了 13 个机器人收集的所有数据，包括 RT-1 使用的演示和成功的自主回合，对应约 115,000 次成功试验，以及额外的 185,000 次失败自主回合，总数据集规模约为 300,000 次试验。模型架构和超参数保持完全相同，因为实验的计算成本使得进一步调整超参数变得不可行（事实上，我们只训练了一次模型）。请注意，在如此多的成功演示下，即使是使用 RT-1 架构的标准模仿学习也已经表现非常好，达到了 82% 的成功率。然而，如图 6（右下）所示，Q-Transformer 即使在如此高的成功率基础上还能进一步提升。该实验表明，Q-Transformer 可以继续扩展到极大的数据集规模，并且持续优于使用 RT-1 的模仿学习和 Decision Transformer。
This experiment includes all of the data collected with 13 robots and comprises of the demonstrations used by RT-1 and successful autonomous episodes, corresponding to about 115,000 successful trials, and an additional 185,000 failed autonomous episodes, for a total dataset size of about 300,000 trials. Model architecture and hyperparameters were kept exactly the same, as the computational cost of the experiment made further hyperparameter tuning prohibitive (in fact, we only train the models once). Note that with this number of successful demonstrations, even standard imitation learning with the RT-1 architecture already performs very well, attaining 82% success rate. However, as shown in Figure 6 (bottom right), Q-Transformer was able to improve even on this very high number. This experiment demonstrates that Q-Transformer can continue to scale to extremely large dataset sizes, and continues to outperform both imitation learning with RT-1 and Decision Transformer.
在本文中，我们介绍了 Q-Transformer，一种用于离线强化学习的高容量 Transformer 架构，适用于大规模多任务机器人强化学习。我们的框架确实存在一些局限性。首先，我们专注于稀疏二元奖励任务，即每次试验的成功或失败。虽然这种设置对于广泛的回合制机器人操作问题是合理的，但它并不普遍，我们期望 Q-Transformer 未来也能扩展到更一般的设置。
In this paper, we introduced the Q-Transformer, an architecture for offline reinforcement learning with high-capacity Transformer models that is suitable for large-scale multi-task robotic RL. Our framework does have several limitations. First, we focus on sparse binary reward tasks corresponding to success or failure for each trial. While this setup is reasonable for a broad range of episodic robotic manipulation problems, it is not universal, and we expect that Q-Transformer could be extended to more general settings as well in the future.
其次，我们采用的逐维度动作离散化方案在更高维度（例如控制人形机器人）中可能变得更加繁琐，因为模型的序列长度和推理时间随动作维度增加。虽然 \(n\)-步返回在一定程度上缓解了这一问题，但序列长度仍然随动作维度增加。对于这种高维动作空间，也可以采用自适应离散化方法，例如通过训练离散自编码器模型并降低表示维度。均匀动作离散化也可能给需要大范围运动粒度的操作任务带来问题，例如粗运动和细运动。在这种情况下，可以基于动作分布使用自适应离散化来表示这两种类型的运动。
Second, the per-dimension action discretization scheme that we employ may become more cumbersome in higher dimensions (e.g., controlling a humanoid robot), as the sequence length and inference time for our model increases with action dimensionality. Although \(n\)-step returns mitigate this to a degree, the length of the sequences still increases with action dimensionality. For such higher-dimensional action space, adaptive discretization methods might also be employed, for example by training a discrete autoencoder model and reducing representation dimensionality. Uniform action discretization can also pose problems for manipulation tasks that require a large range of motion granularities, e.g. both coarse and fine movements. In this case, adaptive discretization based on the distribution of actions could be used for representing both types of motions.
最后，在这项工作中，我们专注于离线强化学习设置。然而，将 Q-Transformer 扩展到在线微调是未来工作的一个令人兴奋的方向，这将使复杂机器人策略的更有效的自主改进成为可能。
Finally, in this work we concentrated on the offline RL setting. However, extending Q-Transformer to online finetuning is an exciting direction for future work that would enable even more effective autonomous improvement of complex robotic policies.