Inductive transfer learning has greatly impacted computer vision, but existing approaches in NLP still require task-specific modifications and training from scratch. We propose Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduce techniques that are key for fine-tuning a language model. Our method significantly outperforms the state-of-the-art on six text classification tasks, reducing the error by 18-24% on the majority of datasets. Furthermore, with only 100 labeled examples, it matches the performance of training from scratch on 100x more data. We open-source our pretrained models and code.
核心贡献 · Key contributions
提出通用语言模型微调（ULMFiT），一种可应用于任何 NLP 任务的迁移学习方法。 Proposes Universal Language Model Fine-tuning (ULMFiT), a transfer learning method applicable to any NLP task.
引入判别式微调、倾斜三角学习率和逐步解冻，以防止灾难性遗忘。 Introduces discriminative fine-tuning, slanted triangular learning rates, and gradual unfreezing to prevent catastrophic forgetting.
在六个文本分类任务上达到最先进水平，在大多数数据集上错误率降低 18-24%。 Achieves state-of-the-art on six text classification tasks, reducing error by 18-24% on most datasets.
仅用 100 个标注样本即可匹配在 100 倍数据上从头训练的性能。 Matches performance of training from scratch on 100x more data with only 100 labeled examples.
开源预训练模型和代码以促进更广泛采用。 Open-sources pretrained models and code to enable wider adoption.
局限 · Limitations
方法仅在文本分类任务上评估，未在序列标注或其他 NLP 任务上测试。 Method evaluated only on text classification tasks, not on sequence labeling or other NLP tasks.
需要大型通用领域语料库（Wikitext-103）进行预训练，并非所有语言都可用。 Requires a large general-domain corpus (Wikitext-103) for pretraining, which may not be available for all languages.
在非常小的数据集（如 TREC-6）上的性能可能因测试集规模有限而不具有统计显著性。 Performance on very small datasets (e.g., TREC-6) may not be statistically significant due to limited test size.
方法依赖 LSTM 架构；使用 Transformer 等其他架构时性能可能不同。 Method relies on LSTM architecture; performance may vary with other architectures like Transformer.