Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→当前训练通用机器人模型的一个主要障碍是异构性。以往的机器人学习方法通常针对特定任务和特定本体收集数据进行训练，这既昂贵又容易过拟合。本研究探讨了通过在不同本体和任务的大规模机器人数据上进行异构预训练来学习策略表征的问题。我们提出了异构预训练 Transformer（HPT），它预训练一个大型可共享的策略神经网络主干，以学习与任务和本体无关的共享表征。该通用架构将来自不同本体的特定本体感觉和视觉输入对齐为短序列的令牌，然后处理这些令牌以映射到不同任务的机器人控制。利用近期的大规模多本体真实世界机器人数据集以及仿真、部署机器人和人类视频数据集，我们研究了跨异构性的策略预训练。我们进行了实验，以探究训练目标在多达 52 个数据集上的缩放行为。HPT 在多个仿真基准和真实世界设置中优于多种基线，并在未见任务上将微调策略性能提升了超过 20%。项目网站（https://liruiw.github.io/hpt/）提供代码和视频。
One of the roadblocks for training generalist robotic models today is heterogeneity. Previous robot learning methods often collect data to train with one specific embodiment for one task, which is expensive and prone to overfitting. This work studies the problem of learning policy representations through heterogeneous pre-training on robot data across different embodiments and tasks at scale. We propose Heterogeneous Pre-trained Transformers (HPT), which pre-train a large, shareable trunk of a policy neural network to learn a task and embodiment agnostic shared representation. This general architecture aligns the specific proprioception and vision inputs from distinct embodiments to a short sequence of tokens and then processes such tokens to map to control robots for different tasks. Leveraging the recent large-scale multi-embodiment real-world robotic datasets as well as simulation, deployed robots, and human video datasets, we investigate pre-training policies across heterogeneity. We conduct experiments to investigate the scaling behaviors of training objectives, to the extent of 52 datasets. HPTs outperform several baselines and enhance the fine-tuned policy performance by over 20% on unseen tasks in multiple simulator benchmarks and real-world settings. See the project website (https://liruiw.github.io/hpt/) for code and videos.
当前训练通用机器人模型的一个障碍是异构性。以往的机器人学习方法通常针对单一任务和特定具身形态收集数据进行训练，这既昂贵又容易过拟合。本文研究了通过在不同具身形态和任务的大规模机器人数据上进行异构预训练来学习策略表示的问题。我们提出了异构预训练 Transformer（HPT），它预训练一个大型可共享的策略神经网络主干，以学习与任务和具身形态无关的共享表示。该通用架构将来自不同具身形态的特定本体感觉和视觉输入对齐为短序列的 token，然后处理这些 token 以映射到不同任务的控制机器人。利用近期的大规模多具身真实世界机器人数据集以及仿真、已部署机器人和人类视频数据集，我们研究了跨异构性的策略预训练。我们进行了实验，以研究训练目标的扩展行为，涉及多达 52 个数据集。HPT 在多个仿真基准和真实世界设置中优于多个基线，并在未见任务上将微调策略性能提升了超过 20%。
One of the roadblocks for training generalist robotic models today is heterogeneity. Previous robot learning methods often collect data to train with one specific embodiment for one task, which is expensive and prone to overfitting. This work studies the problem of learning policy representations through heterogeneous pre-training on robot data across different embodiments and tasks at scale. We propose Heterogeneous Pre-trained Transformers (HPT), which pre-train a large, shareable trunk of a policy neural network to learn a task and embodiment agnostic shared representation. This general architecture aligns the specific proprioception and vision inputs from distinct embodiments to a short sequence of tokens and then processes such tokens to map to control robots for different tasks. Leveraging the recent large-scale multi-embodiment real-world robotic datasets as well as simulation, deployed robots, and human video datasets, we investigate pre-training policies across heterogeneity. We conduct experiments to investigate the scaling behaviors of training objectives, to the extent of 52 datasets. HPTs outperform several baselines and enhance the fine-tuned policy performance by over 20% on unseen tasks in multiple simulator benchmarks and real-world settings.
如今构建机器人策略十分困难：通常需要为每个机器人、任务和环境收集特定数据，而且学到的策略无法泛化到这些特定设置之外。一个彻底改变机器学习的歷史教训是，在大规模、高质量、多样化的数据上进行预训练可以带来通用模型，这些模型通常优于特定模型。开源大规模数据收集的最新进展使这条道路成为可能，但大规模机器人数据中存在的异质性（如不同的机器人硬件和不同的环境）带来了重大挑战。该领域现在的一个核心问题是如何利用异质机器人数据来预训练机器人基础模型。
Building robotic policies today is hard: it often requires collecting specific data for each robot, task, and environment, and the learned policies do not generalize beyond these specific settings. A historical lesson that has revolutionized machine learning is that pre-training on large-scale, high-quality, and diverse data can bring general models that usually outperform specific models. Recent progress in open-source large-scale data collection has made this path possible, but the heterogeneity (such as varying robot hardware and different environments) present in large-scale robotic data has posed a significant challenge. A central question for the field now is how to leverage the heterogeneous robot data to pre-train robotic foundation models.
来自自然语言处理和计算机视觉的基础模型展示了一种范式，即通过在海量且多样化的数据上进行预训练，实现通用的、与任务无关的模型。除了更多数据带来的好处外，用多样化任务进行训练也促使表示更加泛化。这些基础模型能够在各种任务上取得较高的任务成功率，对异常值更加鲁棒，并且能够灵活地适应新任务。这些方法将来自不同领域和任务的输入信号映射到高维表示空间，并表现出一致的缩放行为。之后，只需极少的微调即可将表示迁移到下游任务以获得良好性能。
Foundation models from natural language processing and computer vision have shown a paradigm to achieve general-purpose task-agnostic models through pre-training on massive amounts and diversity of data. In addition to the benefits from more data, training with diverse tasks also enforces the representation to be more generalized. These foundation models can achieve high task success rates for various tasks, are more robust to outliers, and are flexible for adapting to new tasks. These approaches map input signals from distinct domains and tasks into a high-dimensional representation space, and exhibit consistent scaling behaviors. After that, minimal fine-tuning is required to transfer the representation for downstream tasks to achieve good performance.
机器人领域的异质性带来了独特的挑战：不同的机器人是硬件在不同环境中运行的物理上不同的具身。每个具身可能具有不同的本体感觉，包括不同的自由度、末端执行器、运动控制器以及为特定应用构建的工作空间配置。机器人中另一个常见的异质性是视觉异质性。机器人通常配备安装在不同位置（例如腕部和/或第三人称）的不同相机传感器，并且每个机器人的视觉外观因环境和任务而差异巨大。本体感觉和视觉信息对于机器人中复杂、接触密集、长时程的行为至关重要。对这些信息的学习不足可能导致过拟合行为，例如针对特定场景和任务甚至轨迹重复运动。
The heterogeneity in robotics presents a distinct challenge: different robots are physically different embodiments of hardware acting in different environments. Each embodiment can have a distinct proprioception, including different degrees of freedom, end-effectors, motion controllers, and workspace configurations built for a specific application. Another common heterogeneity in robotics is vision heterogeneity. Robots are often equipped with different camera sensors mounted at different places (e.g. wrist and/or third-person) and the visual appearance of each robot varies dramatically due to environments and tasks. Both proprioception and vision information are crucial for complex, contact-rich, long-horizon behaviors in robotics. Poor learning of such information can lead to overfitting behaviors such as repeating motions for a particular scene and task or even trajectory.
在这项工作中，我们提出通过异质预训练将不同具身的本体感觉和视觉信息对齐到共享的“策略语言”来解决这个问题（图 1）。通过这种共享表示，新的具身只需最少的数据和训练即可将其特定设置“翻译”为共享“语言”。换句话说，我们希望预训练与任务无关、与具身无关的基础模型，将来自各个具身的原始传感器信号映射到共享的潜在空间。以往的工作在仅对策略的视觉部分进行人类视频预训练，以及使用统一模型和数据集格式（例如使用语言）对完整策略进行预训练方面取得了重大进展。此外，他们假设预训练中不包含本体感觉，并在迁移学习中事后添加。
In this work, we propose to address this issue by aligning the proprioception and vision information from different embodiments to a shared “language” of policies through heterogenous pre-training (Figure 1). With such a shared representation, a new embodiment only requires minimal data and training to “translate” its specific setup to the shared “languages”. In other words, we want to pre-train task-agnostic and embodiment-agnostic foundational models that can map raw sensor signals from individual embodiments into a shared latent space. Previous works have made significant progress in pre-training only the vision part of the policy on human videos and pre-training the full policy with a unified model and dataset format (e.g. using languages). Additionally, they assume no proprioception in pre-training and add it post hoc in transfer learning.
我们引入了异质预训练 Transformer（HPT），这是一个旨在跨异质具身数据可扩展学习的架构家族。HPT 将通用策略网络架构模块化（图 2），并通过监督学习预训练潜在 Transformer 的策略表示。受多模态数据学习的启发，我们使用特定于具身的标记器（称为“茎”）来对齐各种传感器输入，如相机视图和本体感觉输入。“躯干”是共享的，并在数据集间进行预训练，在适应预训练期间未知的新具身和任务时进行迁移。此外，我们使用特定于任务的行动解码器（称为“头”）来产生行动输出。关键在于，在“对每个具身进行标记化”之后，HPT 在短序列潜在标记的共享空间上操作。这种层次结构的灵感来自于人类如何在脊髓神经回路层面处理特定运动反应与感知刺激之间的反馈回路。
We introduce Heterogeneous Pre-trained Transformers (HPT), a family of architecture designed to scalably learn from data across heterogeneous embodiments. HPT modularizes a general policy network architecture (Figure 2) and pre-trains the policy representation of a latent transformer with supervised learning. Inspired by learning from multimodal data, we use embodiment-specific tokenizers, dubbed “stem”, to align various sensor inputs such as camera views and proprioception inputs. The “trunk” is shared and pre-trained across datasets and is transferred when adapting to new embodiments and tasks that are unknown during the pre-training times. Moreover, we use task-specific action decoders, dubbed “head”, to produce the action outputs. Crucially, after “tokenizing each embodiment”, HPT operates on a shared space of a short sequence of latent tokens. This hierarchy is motivated by how humans handle feedback loops between specific motor responses and perceived stimuli at the level of the spinal cord’s neural circuitry.
我们广泛研究了策略预训练的缩放行为和各种设计，使用了超过 50 个独立数据源（是之前工作的 2 倍多）以及超过 10 亿参数的模型规模。与缩放定律类似，我们发现，在一定程度上，HPT 随数据集数量和多样性以及模型和训练算力的增加而扩展。
We extensively investigated the scaling behaviors and various designs of policy pre-training, using more than 50 individual data sources (2 times more than previous work) and model sizes exceeding 1 billion parameters. Analogous to scaling laws, we found that, to some extent, HPT scales with dataset quantity and diversity, as well as model and training compute.
此外，异质性可能出现在不同的具身域中，例如真实机器人硬件、仿真域和人类视频。我们将许多可用的不同具身的具身数据集（包括真实机器人、仿真和互联网人类视频）纳入预训练过程，并展示了我们框架的通用性，包括超越昂贵的真实世界机器人遥操作的具身。
In addition, heterogeneity can occur across different embodiment domains, such as real robot hardware, simulation domains, and human videos. We incorporate many available embodied datasets from different embodiments, including real robots, simulation, and internet human videos, into the pre-training process, and demonstrate the generality of our framework, including embodiments beyond expensive real-world on-robot teleoperation.
通过在多个仿真基准和真实世界灵巧任务上的迁移学习实验，我们与几个基线和从头训练的对应模型进行了比较。总体而言，基于预训练目标，HPT 可以随模型、数据、算力以及跨真实机器人、仿真和人类视频的机器人数据集的异质性而扩展。这些预训练过程和模型可以简化为新具身和新任务构建可靠机器人策略的数据需求和泛化性能。作为扩展异质性预训练的一次尝试，我们的代码和权重已开源，我们希望 HPT 能为从异质具身和任务中学习机器人表示提供一些启示。
Through transfer learning experiments across multiple simulation benchmarks and real-world dexterous tasks, we compare with several baselines and from-scratch counterparts. Overall, based on the pre-training objectives, HPT can scale with model, data, compute, and the heterogeneity of robotic datasets across real robots, simulations, and human videos. These pre-training procedures and models can simplify building reliable robotic policies for new embodiments and new tasks in terms of data requirements and generalized performance. As an attempt to scale heterogeneous pre-training, our code and weights are open-sourced, and we hope that HPT can shed light on learning robot representations from heterogeneous embodiments and tasks.
预训练通过直接监督和/或自监督，已被证明能够学习到对计算机视觉、自然语言及其交叉领域中未见过的下游任务有用的表示。从 ImageNet 或网络规模数据中学习到的表示对分布偏移具有鲁棒性，并且可以迁移到新任务。
Pre-training, through direct supervision and/or self-supervision, has been shown to learn representations useful for unseen downstream tasks in computer vision and natural language, and their intersections. The representation learned from ImageNet or web-scale data shows robustness to distribution shifts and can be transferred to new tasks.
最近基础模型的兴起通过将任务无关的目标应用于多任务数据，扩展了这些表示学习方法。此外，近期研究表明，可以使用小型投影层来对齐基础模型的预训练特征空间。与其他领域不同，机器人领域的数据量和多样性较少，但异质性更强。
The recent surge of foundation models scales these representation learning methods by applying task-agnostic objectives to multitask data. Moreover, recent works show that small projection layers can be used to align the pre-trained feature spaces of the foundation models. Different from other fields, robotics has less data quantity and diversity but much more heterogeneity.
对齐。最近的工作如 Flamingo、Perceiver 和 ImageBind 提出了通过将图像、语言和音频等多模态数据的不同模态对齐到同一潜在空间来组合表示的方法，以追求表示学习。我们的架构设计也受到多模态学习社区中 LLaVA 等方法的启发。最近，GPT-4o、Gemini、MM1、X-VILA 和 Chameleon 展示了从多种模态出发并服务于多种模态的异构预训练通用 Transformer 的能力。跨模态和/或跨具身的对齐思想，在我们扩展到使用异构具身并重用来自不同具身的数据时非常重要。
Alignment. Recent works such as Flamingo, Perceiver, and ImageBind proposed ways to combine representations from multimodal data such as image, language, and audio by aligning these different modalities to the same latent space in the pursuit of representation learning. Our architecture design is also motivated by methods such as LLaVA in the multimodal learning community. Very recently, GPT-4o, Gemini, MM1, X-VILA, and Chameleon demonstrated the capabilities of heterogeneous pre-training a universal transformer from and for multiple modalities. The idea of alignment, across modalities and/or embodiments, is important as we scale to use heterogeneous embodiments and reuse data from distinct embodiments.
机器人社区已经探索了表示学习。先前的工作如 R3M、VC-1、Voltron 和 SpatialVLM 通过使用人类视频和机器人数据训练策略来研究视觉表示。最近的工作还针对机器人任务对齐来自多种模态和数据分布的表示。在预训练之后，在目标领域中使用冻结表示和/或微调进行迁移学习。
Representation learning has been explored in the robotic community. Previous works such as R3M, VC-1, Voltron, and SpatialVLM investigate visual representations by training the policy with human videos and robotic data. Recent works also align representations from multiple modalities and data distributions for robotic tasks. After pre-training, transfer learning with the frozen representation and/or fine-tuning is conducted in the target domains.
通用策略。机器人中的大规模策略学习已经分别利用了来自真实机器人、人类视频和仿真领域的多样化数据。还有多任务学习、元学习、少样本学习和车队学习等工作。最近，RT-X、Octo、OpenVLA 在来自不同机器人形态的数据集上训练通用的视觉-语言-动作机器人策略。
Generalist Policies. Large-scale policy learning in robotics has leveraged diverse data from real robots, human videos, and simulation domain separately. There are also works in multi-task learning, meta-learning, few-shot learning, and fleet learning. Recently, RT-X, Octo, OpenVLA train generalist vision-language-action robotic policies on datasets from diverse robotic embodiments.
与这些工作相比，HPT 处理了更广泛的异质性，包括本体感觉和视觉，在更异构的领域（包括真实机器人、人类视频和仿真数据）上探索缩放行为，并在仿真基准测试中以更大规模进行评估。
Compared with these works, HPT handles broader heterogeneity including proprioception and vision, explores scaling behaviors on more heterogeneous domains including real robots, human videos, and simulation data, and is evaluated at a larger scale in simulation benchmarks.
我们的架构设计与条件计算和专家混合（MoE）相关，其中我们为每种具身形态创建一个专家，而路由器（针对整个网络）由具身形态决定。该技术已被用于将语言模型扩展到相当大的规模。
Our architecture design is related to works in conditional computation and Mixture of Experts (MoE), where we create one expert for each embodiment, and the router (for the whole network) is determined by the embodiment. This technique has been used to scale language models to a substantial size.
在跨具身的异构机器人学习中，数据来自不同领域，如仿真和真实机器人，跨越多种感知模态，如 RGB 图像、语言指令、深度图、3D 点云和触觉图像。每个机器人都是独特的硬件具身，具有不同的自由度、末端执行器、传感器配置、控制器和动作空间，以及特定应用场景的物理设置。
In heterogeneous robot learning with cross embodiments, the data are generated from different domains such as simulation and real robots, across sensory modalities such as RGB images, language instructions, depth maps, 3D point clouds, and tactile images. Each robot is a unique hardware embodiment with varying degrees of freedom, end-effectors, sensor configurations, controller and action spaces, and application-specific physical setups.
在以下章节中，我们讨论 HPT 网络架构和训练流程以解决上述异构性。我们将网络架构（图 2）模块化为具身特定的茎（stem）、共享的主干（trunk）和任务特定的头（head）。直观上，如图 3 所示的茎是神经网络的早期层，将来自异构具身和模态的感知输入对齐到共享表示空间。网络的共享中间部分称为主干，它将感知表示处理为可用于多种任务的潜在表示。最后，网络的最后部分是头，它将潜在表示映射到感兴趣任务的个体动作空间。训练流程被称为异构预训练，它根据采样的具身和任务数据分配并对齐特定的茎/头对，同时仍然享受共享主干中联合训练的好处。这可以被视为使用神经网络对每个具身进行分词，并减轻了在标准训练流程中将具身统一为同质数据形式的需求。
In the following sections, we discuss the HPT network architecture and the training procedure to address the heterogeneity above. We modularize the network architecture (Figure 2) into the embodiment-specific stem, the shared trunk, and the task-specific heads. Intuitively, the stems, shown in Figure 3, are earlier layers of the neural network that align sensory inputs from heterogeneous embodiment and modalities into the shared representation space. The shared middle part of the network is called the trunk, which processes the sensory representation into a latent representation that can be used for multiple tasks. Finally, the last part of the network is the head, which maps that latent representation to the action space in individual tasks of interest. The training procedure, dubbed heterogeneous pre-training, assigns and aligns specific stem/head pairs based on the sampled embodiment and task data, and still enjoys the benefits of joint training in the shared trunk. This can be thought of as tokenizing each embodiment using neural networks and alleviating the need to unify embodiments into a homogeneous data form in standard training procedures.
HPT 中的主干 \(\theta_{\text{stem}}\)（图 3）由本体感觉分词器和视觉分词器组成。这些分词器将来自不同具身形态的异构输入映射为固定数量、固定维度的词元，从而使主干能够在高度异构的情况下以相同方式处理它们，并享受固定上下文长度下的 Scaling（规模扩张）和推理优势。关键思想是利用注意力机制将固定数量的可学习词元关注到观测特征上。虽然我们主要关注本体感觉和视觉，但处理触觉、3D 和动作输入等其他类型的传感器异构性可以在主干中灵活扩展。
The stem \(\theta_{\text{stem}}\) (Figure 3) in HPT is composed of a proprioceptive tokenizer and a vision tokenizer. These tokenizers map heterogeneous inputs from different embodiments to a fixed number of tokens with fixed dimensions, which enables the trunk to treat them in the same manner despite large heterogeneity, as well as enjoy the scaling and inference benefits on fixed context length. The key idea is to leverage attention to attend a fixed number of learnable tokens to features of the observations. Although we mainly focus on proprioception and vision, handling other kinds of sensor heterogeneity in tactile, 3D, and action inputs can be flexibly extended in stems.
本体感觉分词器。在图 3（左）中，对于具身形态 \(k\)，本体感觉分词器将任意维度的机器人本体感觉信息序列（例如末端执行器位姿为 7 维）映射为 \(N_{p}\)（例如 \(N_{p}=16\)）个维度为 \(d\) 的词元，其中 \(d\) 的取值范围为 128 到 1024。为实现这一点，我们首先使用 MLP 将本体感觉输入映射到维度为 \(d\) 的特征空间。然后应用正弦位置编码，并在状态特征和可学习词元之间使用注意力机制，将其映射为 16 个维度为 \(d\) 的词元。本体感觉信息在机器人策略学习中至关重要，但其使用方式通常只是与视觉编码器进行特征拼接。
Proprioception Tokenizers. In Figure 3 (left), for embodiment \(k\), the proprioceptive tokenizer maps any sequence of robot proprioceptive information with dimension \(d^{k}_{p}\) (e.g. 7 for end-effector pose) into \(N_{p}\) (e.g. \(N_{p}=16\)) tokens with dimension \(d\) with values ranging from 128 to 1024. To achieve this, we first use an MLP to map the proprioceptive input into a feature space with dimension \(d\). We then apply sinusoidal position encoding and use attention across the state feature and the learnable tokens, to map into \(16\) tokens with dimension \(d\). Proprioceptive information is critical in robot policy learning, but its usage is often as simple as feature concatenation with a vision encoder.
视觉分词器。在图 3（右）中，视觉分词器可以将任意维度的相机图像序列（多视角视频）映射为 \(N_{v}\)（默认使用 \(N_{v}=16\)）个维度为 \(d\) 的词元。为实现这一点，我们首先使用预训练的冻结特征网络（例如 ResNet 的 7×7 特征），然后展平特征。之后，我们再次在这些特征和可学习词元之间使用注意力机制，将视觉输入映射为 16 个维度为 \(d\) 的词元。
Vision Tokenizers. In Figure 3 (right), the vision tokenizer can map any sequence of camera images (videos of multiple views) with dimension \(H\times W\times 3\) into \(N_{v}\) (we use \(N_{v}=16\) by default) tokens with dimension \(d\). To achieve this, we first use pre-trained frozen feature networks (e.g. 7 by 7 features from ResNet) and then flatten the features. After that, we again use attention across these features and learnable tokens, to map the vision input into \(16\) tokens with dimension \(d\).
在按时间顺序分别处理每种模态后，我们拼接所有模态的词元，并添加额外的模态嵌入和正弦位置嵌入。这作为输入序列输入到下面介绍的主干中。为避免过拟合，主干仅包含少量参数（一个 MLP 和一个注意力层）。
After processing each modality individually in the time sequence order, we concatenate all modality tokens and add additional modality embeddings and sinusoidal positional embeddings. This is used as the input sequence to the trunk that we introduce below. To avoid overfitting, the stem only has a small number of parameters (one MLP and one attention layer).
相关研究（如 Octo 等）大多专注于通过掩码或自监督预训练策略的视觉骨干。它们通常将单视角图像序列沿通道堆叠用于特定机器人，或使用大量词元（如 256 个）。相比之下，HPT 使用带有预训练视觉编码器的主干，将任意图像序列映射为短词元序列（16 个）。此外，相关研究在迁移时加入本体感觉，而 HPT 则从异构数据集中联合预训练视觉和本体感觉部分。
Related works such as Octo and others mostly focus on pre-training the vision backbone of the policy through masking or self-supervision. They often stack sequences of single-view images along channels for a particular robot or use a large number of tokens (256 in ). In contrast, HPT uses stems with pre-trained vision encoders to map arbitrary image sequences to a short sequence of tokens (16). Moreover, rather than add in proprioception during transfer in related works, HPT jointly pre-trains the vision and proprioception parts, from heterogeneous datasets.
作为预训练的核心组件，主干网络架构采用 Transformer，由潜在空间中的参数\(\theta^{\text{trunk}}\)（维度为\(d\)）参数化。输出词元序列长度\(L\)与输入词元序列长度相同。输出词元序列被简单池化为观测的最终组合特征。主干网络在不同具身和任务之间共享，以捕获复杂的输入-输出关系（即，主干网络参数的数量是固定的，与具身和任务的数量无关）。
As the central component for pre-training, the trunk architecture follows a transformer, parametrized by \(\theta^{\text{trunk}}\) in the latent space with dimension \(d\). The output token sequence length \(L\) is the same as the input token sequence length. The output token sequence is simply pooled as the final combined feature for the observation. The trunk is shared across different embodiments and tasks to capture the complex input-output relationships (i.e., the number of trunk parameters is fixed independent of the number of embodiments and tasks).
策略头 \(\theta_{\text{head}}\) 接收主干 Transformer 的输出，并将其映射到每个数据集中的动作空间 \(\mathcal{A}\)。对于每个具身和任务，策略头可以是任意架构（如 MLP），它接收主干的池化特征作为输入，并输出归一化的动作轨迹。在迁移到新的具身时，策略头会被重新初始化。
The policy head \(\theta_{\text{head}}\) takes the output of the trunk transformer and maps it to the action space \(\mathcal{A}\) in each dataset. For each embodiment and task, the policy head can be an arbitrary architecture (e.g. MLP) that takes as input the pooled feature of the trunk and outputs a normalized action trajectory. The policy head is reinitialized for transferring to a new embodiment.
给定总共 \(K\) 个数据集，这些数据集包含从不同分布 \(\mathcal{D}_{1},...,\mathcal{D}_{k},...,\mathcal{D}_{K}\) 采样的异构具身。我们令 \(\mathcal{D}_{k}=\{\tau^{(i)}\}_{1\leq i\leq M_{k}}\) 表示数据集 \(\mathcal{D}_{k}\) 中的 \(M_{k}\) 条轨迹集合，其中 \(\tau^{(i)}=\{o_{t}^{(i)},a_{t}^{(i)}\}_{1\leq t\leq T}\) 表示第 \(i\) 条最大长度为 \(T\) 的观测和动作元组轨迹。目标是最小化以下跨数据集的损失
Given a total of \(K\) datasets with heterogeneous embodiments sampled from different distributions \(\mathcal{D}_{1},...,\mathcal{D}_{k},...,\mathcal{D}_{K}\), we let \(\mathcal{D}_{k}=\{\tau^{(i)}\}_{1\leq i\leq M_{k}}\) denote a set of \(M_{k}\) trajectories in dataset \(\mathcal{D}_{k}\), with \(\tau^{(i)}=\{o_{t}^{(i)},a_{t}^{(i)}\}_{1\leq t\leq T}\) denoting the \(i\)-th trajectory of maximum length \(T\) of observation and action tuples. The objective is to minimize the following loss across datasets
\(\mathcal{L}\) 是行为克隆损失，计算为基于数据集统计的归一化动作标签与网络动作预测之间的 Huber 损失。\(\theta=\bigcup_{k=1}^{K}\{\theta^{\text{stem}}_{k},\theta^{\text{head}}_{k}\}\cup\theta^{\text{trunk}}\) 表示网络参数，由数据集 \(k\) 的具身特定主干和头部 \(\theta^{\text{stem}}_{k},\theta^{\text{head}}_{k}\) 以及所有具身共享的一组主干参数 \(\theta_{\text{trunk}}\) 组成。该训练过程具有两个数据扩展维度：单个数据集 \(D_{k}\) 的数量 \(M_{k}\) 和数据集总数 \(K\)。在预训练阶段，每次迭代仅更新主干参数，而每个异构具身和任务的主干和头部则根据训练批次采样进行更新。实现细节见附录 A.3 节。
\(\mathcal{L}\) is behavior cloning loss computed as the Huber loss between the normalized action labels based on dataset statistics and the network's action predictions. \(\theta=\bigcup_{k=1}^{K}\{\theta^{\text{stem}}_{k},\theta^{\text{head}}_{k}\}\cup\theta^{\text{trunk}}\) denotes the network parameters comprised of embodiment-specific stem and head \(\theta^{\text{stem}}_{k},\theta^{\text{head}}_{k}\) for dataset \(k\), and a single set of shared trunk parameters \(\theta_{\text{trunk}}\) across all embodiments. This training procedure has two axes of data scaling: the quantity \(M_{k}\) for one dataset \(D_{k}\) and the total number of datasets \(K\). In the pre-training stage, only the trunk parameters are updated at every iteration, and the stems and heads for each heterogeneous embodiment and task are updated based on the training batch sampling. See implementation details in Appendix Section A.3.
策略迁移的过程类似于将新领域的特征（通过预训练的茎编码器）对齐到躯干的预训练嵌入空间。给定来自新具身的新数据集 \(\mathcal{D}_{K+1}\)，目标可以与预训练相同或采用其他替代方案。我们使用具身特定的输入和输出维度（如不同的本体感觉和动作维度）重新初始化头部和茎的参数，并冻结躯干的权重。
The policy transfer process is similar to aligning the features of the new domain (through pre-trained stem encoders) to the pre-trained embedding space of the trunk. Given a new dataset \(\mathcal{D}_{K+1}\) from a new embodiment, the objective can be the same as pre-training or alternatives. We reinitialize the head and stem parameters with embodiment-specific input and output dimensions (such as different proprioception and action dimensions), and freeze the weights of the trunk.
在本节中，我们旨在回答以下问题：HPT 预训练在跨领域的异构数据下是否具有缩放行为？
In this section, we aim to answer the following question: Does HPT pre-training exhibit scaling behavior under heterogeneous data across domains?
默认设置。我们使用 27 个机器人遥操作数据集，包括最近公开的 Open-X Embodiment 数据集的一个子集作为训练语料。默认情况下，我们使用场景的一个相机视角，并通过预训练的冻结 ResNet18 图像编码器计算视觉特征。只要可用且提供，我们就使用本体感觉信息，如末端执行器位姿和关节位置。我们从每个数据集中最多使用 1000 条轨迹，总计 16k 条轨迹，并保留一个验证数据集，每个数据源最多 200 条轨迹。此外，我们使用一个主干大小为 317 万参数的模型，记为 HPT-Small（表 1）。训练使用批量大小 256，迭代 80k 次，这相当于输入到 HPT 的潜在空间中的约 0.65B 个 token，以及视觉和本体感觉 token 空间中的约 5B 个 token（取决于视界）。虽然除了归一化之外，我们没有对齐或预处理动作空间或观察空间，但数据清理和过滤将非常有帮助。
Default Setting. We use 27 robot teleoperation datasets, including a subset of the recently public Open-X Embodiment dataset as the training corpus. By default, we use one camera view of the scene with a pre-trained frozen ResNet18 image encoder to compute vision features. We use proprioception information, such as end-effector poses and joint positions, whenever they are available and provided. We use a maximum of 1000 trajectories from each dataset and a total number of 16k trajectories, and a held-out validation dataset with a maximum of 200 trajectories per data source. Furthermore, we use a model with a trunk size of 3.17 million parameters, denoted as HPT-Small (Table 1). The training uses a batch size of 256 for 80k iterations, which is around 0.65B tokens in the latent space that feeds into HPTs and around 5B tokens in the vision and proprioception token spaces (horizon-dependent). While we do not align or preprocess action space or observation space other than normalization, data cleanup and filtering would be very helpful.
扩展设置。我们在训练过程中使用 200k 条轨迹，涵盖 52 个数据集，包括来自不同具身形态的仿真（例如，）、部署机器人（例如，）和人类视频（例如，）。这包括许多公开可访问的机器人数据集。除了不同机构的不同任务外，这些异构数据集混合（图 4 和图 13）还包含多视角、语言输入以及不同环境中的不同观察输入。
Scaled Setting. We use 200k trajectories with 52 datasets, including simulation (e.g., ), deployed robots (e.g., ), human videos (e.g., ), from distinct embodiments in the training process. This includes many public and accessible robotic datasets. In addition to different tasks in different institutes, these heterogeneous mixtures of datasets (Fig. 4 and Fig. 13) come with multiple views, language inputs, and different observation inputs in different environments.
我们使用预训练最后一次迭代的平均验证损失（对未见轨迹的预测误差）来评估 HPT 的预训练性能。这些验证数据集在训练过程中是固定的，与轨迹数量和模型无关。除非特别说明，验证数据集来自默认设置中的相同 27 个数据集。需要注意的是，在大规模真实机器人环境中评估预训练模型是不现实的，而如果忽略这一目标，衡量大规模预训练进展的替代方案非常少。在 NLP 等领域，训练损失目标（如困惑度）常用于衡量预训练的进展。诚然，该指标存在一些注意事项，包括闭环性能差距和任务成功率差距。我们将在第 5 节关于 HPT 迁移学习的内容中解决这些问题。更多细节和讨论请参见附录 A 节和 D 节。
We evaluate the HPT pre-training performance with the averaged validation loss (prediction errors on unseen trajectories) at the last iteration of pre-training. These validation datasets are fixed independent of the trajectory counts and models during training. Unless particularly noted, the validation datasets come from the same 27 datasets in the Default Setting. Note that it is unrealistic to evaluate the pre-trained models on many real-world robotic environments at scale and there are very few evaluation alternatives to measure large-scale pre-training if we ignore this objective. In fields such as NLP, training loss objective (e.g. perplexity) is often used to measure the progress of pre-training. Admittedly, there are several caveats to this metric including the closed-loop performance gap and the task success rate gap. We will address these issues in Section 5 on HPT transfer learning. See Appendix Section A and Section D for more details and discussions.
数据 Scaling（规模扩张）。在图 5 (a) 中，我们观察到即使在日益异构的具身体上，验证损失也保持稳定并呈现 Scaling（规模扩张）趋势。此外，我们发现算力（例如每次训练运行中看到的样本数）和数据量需要同步扩展，才能在训练过程中更接近收敛。在图 5 (a) 的红线中，我们观察到随着轨迹总数增加，通过使用更大的模型并每增加一个数量级的轨迹数就加倍批量大小，验证损失得到改善。如果严格增加数据而保持其他因素受限（HPT-S 和固定迭代次数），可能会导致性能过早达到平台期，每个数据集最多约 1000 条轨迹，如图 5 中的蓝线所示。在图 5 (b) 中，我们还使用固定数量的 epoch 在越来越多的数据集上进行预训练，并在固定子集（前 10 个数据集）上评估。我们假设使用更多具身进行训练有助于主干网络的泛化。这些实验可以扩展到 20 万条轨迹和 52 个数据集。
Data Scaling. In Figure 5 (a), we observe stable and scaling validation losses even on increasingly heterogeneous embodiments. Moreover, we found the compute (e.g., samples seen per training run) and the data amounts needed to scale in tandem to get closer to convergence in the training process. In the red line in Figure 5 (a), we observe better validation losses as we scale up the total number of trajectories, by using a larger model and doubling the batch size every order of magnitude increase in trajectory counts. Strictly increasing data while keeping others bottlenecked (HPT-S and fixed iterations) might cause an early plateau performance at around 1000 trajectories max per dataset, as shown in the blue line in Figure 5. In Figure 5 (b), we also pre-train on an increasing number of datasets with a fixed number of epochs and evaluate on the fixed subset (first 10 datasets). We hypothesize that training with more embodiments contributes to the generalization of the trunk. These experiments can scale to the extent of 200k trajectories and 52 datasets.
模型 Scaling（规模扩张）。在图 7 中，我们固定 RT-X 中的数据集数量（27 个），每个数据集最多使用 1000 条轨迹。我们沿着模型大小（从 1M 到 1B）进行扩展，并逐渐将批量大小从 256 增加到 2048（每增加一个数量级的模型大小就加倍），并使用包含 17 万条轨迹的更大数据集。我们观察到，当扩展到更大的模型和更多的算力时（红线），预训练可以达到较低的验证损失，直到达到平台期。我们没有发现扩展深度或扩展宽度之间存在显著差异。
Model Scaling. In Figure 7, we fix the number of datasets (27) in RT-X and use a maximum of 1000 trajectories for each dataset. We scale along model size (from 1M to 1B) and gradually increase the batch sizes from 256 to 2048 (doubles every order of model size increase) and use the larger dataset with 170k trajectories. We observe that when we scale to bigger models with larger amounts of compute (red line), the pre-training can achieve low validation losses until it is plateaued. We do not find a significant difference between scaling depth or scaling width.
Epoch Scaling（轮次扩展）。在本实验中，我们固定数据集数量（27 个），每个数据集最多使用 1000 条轨迹。在图 6 中，我们观察到增加批量大小（左图）可以有效扩展训练 token 数（右图），通常可以改善模型性能直至收敛。另一个观察是使用分布式工作器从尽可能多的数据集加载数据以聚合每个批次。我们假设使用大批量可以减少异构数据集训练中的大方差。更多实验细节见附录 B。
Epoch Scaling. In this experiment, we fix the number of datasets (27) and use a maximum of 1000 trajectories for each dataset. In Figure 6, we observe that increasing batch sizes (Left), which effectively scales training tokens (Right), can generally improve the model performance until convergence. Another observation we have is to use distributed workers to load from as many datasets as possible to aggregate each batch. We hypothesize that the large variance of training on heterogeneous datasets can be reduced by using a large batch size. See Appendix B for more experiment details.
我们在真实世界机器人遥操作数据之外进行了实验，因为这类数据采集和扩展成本高昂。对于额外数据集，我们考虑了 7 个仿真数据集，涵盖多个流行模拟器：Drake、Mujoco、Isaac Sim、PyBullet，以及 Sapien 和 Flex，均包含图像输入和专家演示。对于缺乏本体感觉和动作信息的人类数据集，我们使用姿态和 2D 位置作为监督策略学习目标的替代。我们总共使用了来自 EPIC kitchen 和 PoCo 的 300 条轨迹，最大轨迹长度为 1000。更多数据集组成细节见附录图 13 和表 4。
We experiment beyond real-world robot teleop data, which is expensive to collect and scale. For the additional datasets, we consider 7 simulation datasets across many popular simulators Drake, Mujoco, Isaac Sim, and PyBullet, as well as Sapien and Flex, with image inputs and expert demonstrations. For the human datasets that lack proprioception and action information, we use poses and 2D positions as surrogates for the supervised policy learning objectives. We use in total 300 trajectories from EPIC kitchen and PoCo with a maximum trajectory length 1000. See Appendix Figure 13 and Table 4 for more details on the dataset compositions.
在图 8 中，我们对每个数据集最多使用 1000 条轨迹，并与 27 个数据集的基线进行比较，在所有预训练数据集上进行评估。我们表明，尽管与真实机器人在具身差距上较大，但在仿真和人类视频等额外具身数据集上进行预训练是可行的。这些数据集为纯遥操作数据提供了互补的具身数据，并展示了 HPT 框架能够处理多大的异质性。
In Figure 8, we use a maximum of 1000 trajectories for each dataset and compare against the baseline of 27 datasets with evaluation on all the pre-trained datasets. We show that pre-training on additional embodiment datasets such as simulation and human video datasets can be possible, despite the large embodiment gaps with real robots. These datasets provide complimentary embodiment data to pure teleop data, and they illustrate how much heterogeneity can be handled in the HPT framework.
在上一节中，我们使用验证损失评估了预训练。在本节中，我们通过迁移学习中的任务成功率来回答以下问题：预训练的 HPT 模型能否迁移到仿真和现实世界中的新具身、新任务和新环境？
In the previous section, we evaluated pre-training using validation losses. In this section, we answer the following question with task success rates in transfer learning: Can the pre-trained HPT model be transferred to new embodiments, tasks, and environments in simulation and the real world?
我们在机器人操作仿真基准 Meta-world、RoboMimic 和 Fleet-Tools 上评估预训练表示。每个训练数据集每个任务使用 20-100 条轨迹，每个测试包含 50 个具有不同初始条件的回合。策略使用 HPT-Small 作为预训练主干，并重新初始化茎和头以进行迁移。
We evaluate the pre-trained representations on robot manipulation simulation benchmarks Meta-world, RoboMimic, and Fleet-Tools. Each training dataset uses from 20-100 trajectories per task and each testing covers 50 episodes with different initial conditions. The policies use HPT-Small as the pre-trained trunk and reinitialize the stem and head for transferring.
在评估阶段，我们比较以下模型：No Trunk 仅使用茎和头，没有中间的主干，并按照常见做法从头训练。From Scratch 从头训练整个策略（包含主干），Pretrained Frozen 在迁移学习中使用并冻结预训练主干，Pretrained Finetuned 加载预训练的 HPT-Base 主干并端到端微调整个网络，Pretrained Finetuned (HPT-XL) 使用相同的微调过程，但使用预训练的 HPT-XL 主干，其预训练验证损失较低。为了减少方差，我们对每个模型进行 5 次独立的训练和评估，并取平均值。在 RTX 3070 GPU 上，迁移期间的推理时间对于 HPT-base 为 47Hz，对于 HPT-XL 为 19Hz，而更新的 GPU（如 A100）可以快 3-4 倍。
During the evaluation phase, we compare the following models: No Trunk uses only the stem and head without the trunk in the middle and trains from scratch as common practice. From Scratch trains the entire policy from scratch with the trunk, Pretrained Frozen uses and freezes the pre-trained trunk during transfer learning and Pretrained Finetuned loads the pre-trained HPT-Base trunk and finetunes the whole network end-to-end, and Pretrained Finetuned (HPT-XL) uses the same fine-tuning procedure with a pre-trained HPT-XL trunk with a lower pre-training validation loss. To reduce the variance, we conduct independent training runs and evaluations 5 times and average for each model. The inference time during transfer on an RTX 3070 GPU is 47Hz for HPT-base and 19Hz for HPT-XL, while a more recent GPU like A100 can be 3-4 times faster.
在图 10(a)中，我们在闭环仿真中测试了模型在下游任务上的表现，观察到使用从 HPT-B 到 HPT-XL 的预训练模型时任务成功率有所提高，尽管仿真实验的预训练仅发生在真实世界的具身环境中。
In Figure 10 (a), we test the model on the downstream tasks in closed-loop simulation and observe improved task success rate using the pre-trained models ranging from HPT-B to HPT-XL, although pre-training for the simulation experiments only happens in the real-world embodiments.
在图 10(b)中，我们在最近发布的 Simpler 基准上运行 HPT，该基准允许在高保真仿真中与 Octo、RT1-X 和 RT2-X 进行比较。我们聚焦于 Google EDR 具身环境中的三个不同任务：关闭抽屉、移动靠近和拿起可乐罐。对于每个任务，我们测试了多种不同的初始化，所有任务总共超过 300 个回合。需要注意的是，HPT-S 的预训练语料不包含仿真数据，且仿真任务侧重于语言条件化，不提供本体感觉输入，这并不适合 HPT。为了解决这些问题，我们在仿真协议下对 HPT 在包含约 50 条轨迹的监督数据集上进行微调。我们使用 HPT-base 作为该实验的主干网络。我们使用来自原始论文的基线结果。更多实现和实验细节参见 A.4 节。
In Figure 10 (b), we run HPT on the recently released Simpler Benchmark, which allows for comparing with Octo, RT1-X, and RT2-X on a high-fidelity simulation. We focus on three different tasks: Close Drawer, Move Near, and Pick Coke Can in the Google EDR embodiment. For each task, we test several different initializations with a total of over 300 episodes for all tasks. Note that the pre-training corpus of HPT-S does not include [the simulation data], and simulation tasks have a focus on language conditioning and do not expose proprioception inputs, which is not suitable for HPT. To address these issues, we finetune HPT on the supervised datasets with around 50 trajectories under the simulation protocol. We use HPT-base as the backbone for this experiment. We use the baseline results from [the original papers]. See Section A.4 for more implementation and experiment details.
在真实世界实验中，我们在两个不同的具身平台上评估 HPT，分别用于宠物护理和装配任务，这些任务不在预训练数据集中。具体来说，对于这两个机器人，我们实验了不同的观测空间（1 个摄像头 vs. 2 个摄像头）以及不同的动作空间（相对位姿 vs. 绝对位姿）。在数据收集方面，我们同时使用了 Oculus Quest 来收集相对位姿控制作为动作标签，以及动觉教学。真实世界遥操作的情节长度从 50 步到 150 步不等，控制频率为 10 Hz。我们实验了 Sweep Leftover、Fill Water、Scoop Food 和 Switch Insertion 等任务，这些任务需要与颗粒状或小物体进行 5-20 秒的精细接触交互，如图 11 所示。我们为每个任务收集约 100 个演示，并评估 15 次试验以测量平均成功率。
For the real-world experiments, we evaluate the HPTs on two different embodiments for tasks in pet care and assembly, which are not covered in the pre-training datasets. In particular, for these two robots, we experiment with different observation spaces (1 camera vs. 2 cameras) as well as different action spaces (relative pose vs. absolute pose). For data collection, we experiment with both an Oculus Quest to collect relative pose control as action labels as well as kinesthetic teaching. The episode lengths of real-world teleoperation vary from 50 steps to 150 steps with 10 Hz control frequencies. We experiment with the tasks Sweep Leftover, Fill Water, Scoop Food, and Switch Insertion, which require 5-20 seconds of interactions with granular or small objects with fine contacts, shown in Figure 11. We collect around 100 demos for each task and evaluate them for 15 trials to measure the average success rate.
我们采用与前一节类似的迁移学习方法，并在真实世界评估协议下评估预训练的 HPT 表示。我们以 20,000 次迭代训练策略，批量大小为 256，学习率为 \(5e^{-6}\)。实现细节推迟到附录 A.5 节。定量上，在图 12 中，我们观察到预训练策略在成功率上优于 No-Trunk 和 From-Scratch 基线。特别是，Fill-Water 中的 From-Scratch 基线使用了最先进的扩散策略架构，以说明预训练表示的灵活性。定性上，在图 11 中，我们观察到预训练 HPT 对不同的姿态和颗粒物体数量、不同的相机配置和光照条件具有更好的泛化性和鲁棒性。
We adopt a similar transfer learning method as in the previous section and evaluate the pre-trained HPT representations under real-world evaluation protocols. We train the policy with 20,000 iterations with a batch size of 256 and a learning rate of \(5e^{-6}\). We defer implementation details to Appendix Section A.5. Quantitatively in Figure 12, we observe that pre-trained policies attain a better success rate over the No-Trunk and the From-Scratch baselines. In particular, the From-Scratch baselines in Fill-Water use the state-of-the-art diffusion policy architecture to illustrate the flexibility of the pre-trained representations. In Figure 11, qualitatively, we observe better generalization and robustness to varying poses and numbers of granular objects, and varying camera configurations and lighting conditions with pre-trained HPT.
在表 3 中，我们对 Sweep Leftover 任务进行了消融研究。我们还与 R3M、Voltron 和 VC-1 进行了比较。我们使用带有已发布主干和权重的微调模型。我们注意到，这些先前的工作仅关注使用人类视频预训练策略的视觉编码器。最后，我们与从头训练的策略（From Scratch）以及在预训练期间不使用本体感觉并在之后添加本体感觉的策略（No Prop. Finetuned）进行了比较。我们所有的实验都使用预训练编码器，可训练参数（主干和头部）可少至参数的 2%。
In Table 3, we perform an ablation study for the Sweep Leftover task. We also compare with R3M, Voltron, and VC-1. We use a fine-tuned model with the released backbone and weights. We note that these previous works focus on only pre-training the vision encoders of the policies with human videos. Finally, we compared with policies that train from scratch (From Scratch) and policies that do not use proprioception during pre-training (No Prop. Finetuned) and add in proprioception afterwards. All of our experiments use pre-trained encoders and the trainable parameters (stem and head) can be as few as 2% of the parameters.
在许多方面仍有改进空间，包括数据集构建和预训练目标。具体来说，我们平衡数据集混合中的具身划分相当简单。此外，确保数据质量的仔细数据过滤在本工作中尚未充分探索。同时，本工作专注于监督学习作为预训练目标，且数据规模（以词元计）和训练算力（以 FLOPs 计）仅达到 LLM 训练的适中规模，以确保完全收敛。尽管模型架构和训练过程是模块化的且与具身设置无关，但异构预训练可能收敛缓慢。在评估方面，仿真和真实世界评估任务均局限于固定具身的短视域操作任务，这可能限制了使用更高容量模型的优势。此外，学习到的策略在测试任务上仍未提供很高的可靠性（通常低于 90%）。有关失败模式的示例，请参见附录§C。
There is room for improvement in many aspects, including dataset curation and pre-training objectives. Specifically, the embodiment splits in our balanced dataset mixture are rather simple. Moreover, careful data filtering to ensure data quality is under-explored in this work. Also, this work has focused on supervised learning as the pre-training objective, and the data size in tokens and training compute sizes in FLOPs only reach a moderate scale of LLM training to ensure full convergence. Although the model architecture and training procedure are modular and independent of embodiment setups, heterogeneous pre-training can converge slowly. For evaluation, both the simulation and real-world evaluation tasks are restricted to short-horizon manipulation tasks with a fixed embodiment, which might limit the benefits of using a higher-capacity model. Furthermore, the learned policies still do not offer very high reliability on the tested tasks (typically below 90%). See Appendix §C for some failure modes.
鉴于近期数据规模的激增，机器人学习仍受限于其泛化性，原因在于异构性，包括不同的具身、任务和机器人运行的环境。为了应对机器人技术中常见的异构性，我们提出了 HPT，一种模块化架构和框架，通过预训练来拥抱这种异构性。我们探索并扩展了 HPT，使用异构数据集扩展到超过 50 个可用数据集。学习到的表示可以迁移并提升仿真和真实世界的性能，且与预训练性能表现出相关性。代码（https://github.com/liruiw/HPT 和 https://github.com/liruiw/lerobot）已开源供未来研究使用。我们希望这一视角能启发未来在处理机器人数据的异构性以构建机器人基础模型方面的工作。
Given the recent surge of scaled data, robot learning is still limited by its generality because of the heterogeneity, including different embodiments, tasks, and environments where the robots are operated. To handle the heterogeneity common in robotics, we propose HPT, a modular architecture and framework to embrace this heterogeneity through pre-training. We explore and scale HPT with heterogeneous datasets to over 50 available datasets. The learned representation can be transferred and improve performance in both simulation and the real world, and it shows correlations with pre-training performance. The code (https://github.com/liruiw/HPT and https://github.com/liruiw/lerobot) is open-source for future research. We hope this perspective will inspire future work in handling the heterogeneous nature of robotic data for robotic foundation models.