Conformer: 用于语音识别的卷积增强 Transformer

Conformer: Convolution-augmented Transformer for Speech Recognition

安莫尔·古拉蒂 Anmol Gulati · Google · 2020-05-16 · arXiv:2005.08100 ↗ · 被引 4161

摘要 · Abstract

最近，基于 Transformer 和卷积神经网络（CNN）的模型在自动语音识别（ASR）中展现出优于循环神经网络（RNN）的结果。Transformer 模型擅长捕捉基于内容的全局交互，而 CNN 则能有效利用局部特征。在这项工作中，我们通过研究如何以参数高效的方式结合卷积神经网络和 Transformer 来同时建模音频序列的局部和全局依赖，实现了两全其美。为此，我们提出了用于语音识别的卷积增强 Transformer，命名为 Conformer。Conformer 显著优于之前的 Transformer 和 CNN 模型，达到了最先进的准确率。在广泛使用的 LibriSpeech 基准测试中，我们的模型在未使用语言模型的情况下，在 test/testother 上实现了 2.1%/4.3%的词错误率（WER），使用外部语言模型时达到 1.9%/3.9%。我们还观察到，仅使用 10M 参数的小模型也取得了 2.7%/6.3%的竞争性性能。

Recently Transformer and Convolution neural network (CNN) based models have shown promising results in Automatic Speech Recognition (ASR), outperforming Recurrent neural networks (RNNs). Transformer models are good at capturing content-based global interactions, while CNNs exploit local features effectively. In this work, we achieve the best of both worlds by studying how to combine convolution neural networks and transformers to model both local and global dependencies of an audio sequence in a parameter-efficient way. To this regard, we propose the convolution-augmented transformer for speech recognition, named Conformer. Conformer significantly outperforms the previous Transformer and CNN based models achieving state-of-the-art accuracies. On the widely used LibriSpeech benchmark, our model achieves WER of 2.1%/4.3% without using a language model and 1.9%/3.9% with an external language model on test/testother. We also observe competitive performance of 2.7%/6.3% with a small model of only 10M parameters.

核心贡献 · Key contributions

提出 Conformer，结合卷积和 Transformer 用于语音识别。
Proposes Conformer, combining convolution and Transformer for speech recognition.
在 LibriSpeech test/testother 上达到 1.9%/3.9%的词错误率，为当前最优。
Achieves state-of-the-art WER of 1.9%/3.9% on LibriSpeech test/testother.
引入三明治结构，在注意力与卷积模块两侧放置两个前馈模块。
Introduces a sandwich structure with two feed-forward modules around attention and convolution.
通过消融实验证明卷积模块对性能至关重要。
Demonstrates convolution module is critical for performance via ablation studies.
展示参数效率：10M 参数模型达到 2.7%/6.3%的竞争性词错误率。
Shows parameter efficiency: 10M model achieves competitive 2.7%/6.3% WER.
研究注意力头数、卷积核大小、激活函数和模块放置的影响。
Studies effects of attention heads, kernel size, activation, and module placement.

局限 · Limitations

仅在 LibriSpeech 上评估，未验证在其他数据集上的泛化性。
Evaluation only on LibriSpeech; generalizability to other datasets unverified.
需要仔细调整超参数，如卷积核大小和注意力头数。
Requires careful tuning of hyperparameters like kernel size and attention heads.
使用外部语言模型时的性能可能因语言模型质量而异。
Performance with external language model may vary with LM quality.
深度可分离卷积的计算成本可能限制在资源受限设备上的部署。
Computational cost of depthwise convolution may limit deployment on resource-constrained devices.
研究仅限于语音识别，未探索在其他领域的适用性。
Study limited to speech recognition; applicability to other domains not explored.

论文章节 · Sections（共 12）

摘要 Abstract
1 引言 1 Introduction
2 卷积编码器 2 Conformer Encoder
2.1 多头自注意力模块 2.1 Multi-Headed Self-Attention Module
2.2 卷积模块 2.2 Convolution Module
2.3 前馈模块 2.3 Feed Forward Module
2.4 卷积块 2.4 Conformer Block
3.1 数据 3.1 Data
3.2 卷积转换器 3.2 Conformer Transducer
3.3 LibriSpeech 上的结果 3.3 Results on LibriSpeech
3.4 消融研究 3.4 Ablation Studies
4 结论 4 Conclusion

阅读逐段中英对照全文 →