We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on transcribed speech can outperform the best semi-supervised methods while being conceptually simpler. wav2vec 2.0 masks the speech input in the latent space and solves a contrastive task defined over a quantization of the latent representations which are jointly learned. Experiments using all labeled data of Librispeech achieve 1.8/3.3 WER on the clean/other test sets. When lowering the amount of labeled data to one hour, wav2vec 2.0 outperforms the previous state of the art on the 100 hour subset while using 100 times less labeled data. Using just ten minutes of labeled data and pre-training on 53k hours of unlabeled data still achieves 4.8/8.2 WER. This demonstrates the feasibility of speech recognition with limited amounts of labeled data.
核心贡献 · Key contributions
提出 wav2vec 2.0，一种从原始音频中学习语音表示的自监督框架。 Proposes wav2vec 2.0, a self-supervised framework for learning speech representations from raw audio.
通过对比任务联合学习离散语音单元和上下文表示。 Jointly learns discrete speech units and contextualized representations via a contrastive task.
仅用 10 分钟标注数据即在 Librispeech 上达到最优结果。 Achieves state-of-the-art results on Librispeech with only 10 minutes of labeled data.
在概念上更简单的同时优于之前的半监督方法。 Outperforms previous semi-supervised methods while being conceptually simpler.
展示了使用 53k 小时无标注预训练实现超低资源语音识别的可行性。 Demonstrates feasibility of ultra-low resource speech recognition with 53k hours of unlabeled pre-training.
在 TIMIT 音素识别和完整 Librispeech 基准上创下新纪录。 Sets new state of the art on TIMIT phoneme recognition and full Librispeech benchmark.
局限 · Limitations
评估仅限于英语语音数据集（Librispeech、TIMIT、LibriVox）。 Evaluation limited to English speech datasets (Librispeech, TIMIT, LibriVox).
需要大量无标注数据（53k 小时）才能达到最佳性能。 Requires large amounts of unlabeled data (53k hours) for best performance.
微调仍需转录语音；未探索零样本识别。 Fine-tuning still needs transcribed speech; zero-shot recognition not explored.
模型规模和算力需求高（Large 模型使用 128 个 GPU）。 Model size and compute requirements are high (Large model uses 128 GPUs).
基于字符的词汇表可能限制性能，相比词片方法。 Character-based vocabulary may limit performance compared to word pieces.
论文章节 · Sections（共 18）
摘要Abstract
1 引言1 Introduction
2 模型2 Model
3 训练3 Training
3.1 掩码3.1 Masking
3.2 目标函数3.2 Objective
3.3 微调3.3 Fine-tuning
4.1 数据集4.1 Datasets
4.2 预训练4.2 Pre-training
4.3 微调4.3 Fine-tuning
4.4 语言模型与解码4.4 Language Models and Decoding
5.1 低资源标注数据评估5.1 Low-Resource Labeled Data Evaluation
5.2 Librispeech 高资源标注数据评估5.2 High-Resource Labeled Data Evaluation on Librispeech