Self-supervised approaches for speech representation learning are challenged by three unique problems: (1) there are multiple sound units in each input utterance, (2) there is no lexicon of input sound units during the pre-training phase, and (3) sound units have variable lengths with no explicit segmentation. To deal with these three problems, we propose the Hidden-Unit BERT (HuBERT) approach for self-supervised speech representation learning, which utilizes an offline clustering step to provide aligned target labels for a BERT-like prediction loss. A key ingredient of our approach is applying the prediction loss over the masked regions only, which forces the model to learn a combined acoustic and language model over the continuous inputs. HuBERT relies primarily on the consistency of the unsupervised clustering step rather than the intrinsic quality of the assigned cluster labels. Starting with a simple k-means teacher of 100 clusters, and using two iterations of clustering, the HuBERT model either matches or improves upon the state-of-the-art wav2vec 2.0 performance on the Librispeech (960h) and Libri-light (60,000h) benchmarks with 10min, 1h, 10h, 100h, and 960h fine-tuning subsets. Using a 1B parameter model, HuBERT shows up to 19% and 13% relative WER reduction on the more challenging dev-other and test-other evaluation subsets.
核心贡献 · Key contributions
提出 HuBERT，一种通过离线聚类隐藏单元的掩码预测进行自监督语音表示学习的方法。 Proposes HuBERT, a self-supervised speech representation learning method using masked prediction of hidden units from offline clustering.
仅在掩码区域应用预测损失，迫使模型学习联合声学与语言模型。 Applies prediction loss only on masked regions, forcing the model to learn combined acoustic and language models.
表明目标一致性比正确性更重要，从而能够从简单的 k-means 聚类中稳健学习。 Shows that target consistency matters more than correctness, enabling robust learning from simple k-means clusters.
在 Librispeech 和 Libri-light 基准上取得最先进结果，匹配或超越 wav2vec 2.0。 Achieves state-of-the-art results on Librispeech and Libri-light benchmarks, matching or outperforming wav2vec 2.0.
证明使用学习到的表示迭代优化聚类分配可提升表示质量。 Demonstrates iterative refinement of cluster assignments using learned representations improves representation quality.
扩展到 1B 参数模型，在挑战性子集上实现高达 19%的相对词错误率降低。 Scales to a 1B parameter model, achieving up to 19% relative WER reduction on challenging subsets.
局限 · Limitations
需要离线聚类步骤，增加计算开销并依赖聚类质量。 Requires offline clustering step, adding computational overhead and dependency on clustering quality.
性能依赖于目标的迭代优化，可能并非在所有设置下都能最优收敛。 Performance relies on iterative refinement of targets, which may not converge optimally in all settings.
仅在英语有声书数据上评估；未验证对其他语言或噪声条件的泛化能力。 Evaluated only on English audiobook data; generalization to other languages or noisy conditions is not verified.
微调需要为每个子集仔细调整超参数，限制了实际部署。 Fine-tuning requires careful hyperparameter tuning for each subset, limiting practical deployment.
高达 1B 参数的模型需要大量计算资源进行预训练和微调。 Model size up to 1B parameters demands significant computational resources for pre-training and fine-tuning.
论文章节 · Sections（共 19）
摘要Abstract
一、引言I Introduction
二-A 学习 HuBERT 的隐藏单元II-A Learning the Hidden Units for HuBERT
二-B 通过掩码预测进行表示学习II-B Representation Learning via Masked Prediction
二-C 使用聚类集成学习II-C Learning with Cluster Ensembles
二-D 聚类分配的迭代优化II-D Iterative Refinement of Cluster Assignments
二-E 实现II-E Implementation
三、相关工作III Related Work
四-A 数据IV-A Data
四-B 无监督单元发现IV-B Unsupervised Unit Discovery
四-C 预训练IV-C Pre-Training
四-D 监督微调与解码IV-D Supervised Fine-Tuning and Decoding
四-E 目标质量度量IV-E Metrics of Target Quality
五-A 主要结果：低资源与高资源设置V-A Main Results: Low- and High-Resource Setups
五-B 分析：K-Means 稳定性V-B Analysis: K-Means Stability
五-C 分析：跨层与迭代的聚类质量V-C Analysis: Clustering Quality Across Layers and Iterations
五-D 消融：预测掩码帧的重要性V-D Ablation: The Importance of Predicting Masked Frames
五-E 消融：聚类集成的影响V-E Ablation: The Effect of Cluster Ensembles