We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning. When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.
核心贡献 · Key contributions
提出 Whisper，一个在 68 万小时弱监督多语言音频上训练的语音处理系统。 Proposes Whisper, a speech processing system trained on 680,000 hours of weakly supervised multilingual audio.
实现零样本迁移，无需微调即可与先前全监督结果竞争。 Achieves zero-shot transfer competitive with prior fully supervised results without fine-tuning.
证明扩展弱监督预训练可提升鲁棒性和泛化能力。 Demonstrates that scaling weakly supervised pre-training improves robustness and generalization.
引入使用解码器输入令牌的多任务和多语言训练格式。 Introduces a multitask and multilingual training format using decoder input tokens.
表明更大模型从跨任务和语言的联合训练中受益。 Shows that larger models benefit from joint training across tasks and languages.
发布模型和推理代码以促进鲁棒语音处理的进一步研究。 Releases models and inference code to facilitate further research on robust speech processing.
局限 · Limitations
Whisper 仅限于 30 秒音频片段，长音频需要缓冲转录。 Whisper is limited to 30-second audio segments and requires buffered transcription for longer audio.
由于训练数据以英语为主，低资源语言性能较差。 Performance on low-resource languages is poor due to English-heavy training data.
剩余错误包括长转录中的重复循环、漏词和幻觉。 Remaining errors include repetition loops, missing words, and hallucinations in long-form transcription.
零样本评估可能无法反映微调可达到的最佳性能。 Zero-shot evaluation may not reflect optimal performance achievable with fine-tuning.
文本标准化器可能过拟合 Whisper 的转录风格，影响 WER 比较。 Text normalizer may overfit to Whisper's transcription style, affecting WER comparisons.
论文章节 · Sections（共 23）
摘要Abstract
1 引言1 Introduction
2.1 数据处理2.1 Data Processing
2.2 模型2.2 Model
2.3 多任务格式2.3 Multitask Format
2.4 训练细节2.4 Training Details
3.1 零样本评估3.1 Zero-shot Evaluation
3.2 评估指标3.2 Evaluation Metrics
3.3 英语语音识别3.3 English Speech Recognition
3.4 多语言语音识别3.4 Multi-lingual Speech Recognition
3.5 翻译3.5 Translation
3.6 语言识别3.6 Language Identification
3.7 对加性噪声的鲁棒性3.7 Robustness to Additive Noise
3.8 长文本转录3.8 Long-form Transcription
3.9 与人类表现比较3.9 Comparison with Human Performance
4.1 模型缩放4.1 Model Scaling
4.2 数据集缩放4.2 Dataset Scaling
4.3 多任务与多语言迁移4.3 Multitask and Multilingual Transfer
4.4 文本规范化4.4 Text Normalization
4.5 可靠长文本转录策略4.5 Strategies for Reliable Long-form Transcription