With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.
核心贡献 · Key contributions
提出 XLNet，一种通过排列语言建模学习双向上下文的广义自回归预训练方法。 Proposes XLNet, a generalized autoregressive pretraining method that learns bidirectional contexts via permutation language modeling.
通过避免使用掩码破坏输入，克服了 BERT 的预训练-微调差异。 Overcomes BERT's pretrain-finetune discrepancy by avoiding input corruption with masks.
集成 Transformer-XL 的段循环和相对编码，改进长文本处理。 Integrates Transformer-XL's segment recurrence and relative encoding for improved long-text handling.
引入双流自注意力机制，解决排列建模中的目标预测歧义。 Introduces two-stream self-attention to resolve target prediction ambiguity in permutation modeling.
在 20 个 NLP 任务上取得最先进结果，包括问答、自然语言推理、情感分析和文档排序。 Achieves state-of-the-art results on 20 NLP tasks, including QA, NLI, sentiment analysis, and document ranking.
证明排列语言模型和 Transformer-XL 共同促成 XLNet 优于 BERT。 Demonstrates that permutation LM and Transformer-XL both contribute to XLNet's superiority over BERT.
局限 · Limitations
排列语言建模增加优化难度，需要部分预测才能收敛。 Permutation language modeling increases optimization difficulty, requiring partial prediction for convergence.
XLNet 的性能提升部分归因于更多训练数据和算力，而非仅目标函数。 XLNet's performance gain is partly due to more training data and compute, not solely the objective.
双流注意力机制增加了架构复杂性和内存开销。 The two-stream attention mechanism adds architectural complexity and memory overhead.
消融实验表明，与 BERT 不同，下一句预测并未持续改进 XLNet。 Ablation shows next-sentence prediction does not consistently improve XLNet, unlike BERT.
XLNet 对 Transformer-XL 的依赖可能限制其在极短序列任务上的适用性。 XLNet's reliance on Transformer-XL may limit applicability to tasks with very short sequences.
论文章节 · Sections（共 13）
摘要Abstract
1 引言1 Introduction
2.1 背景2.1 Background
2.2 目标：排列语言建模2.2 Objective: Permutation Language Modeling
2.3 架构：用于目标感知表示的双流自注意力2.3 Architecture: Two-Stream Self-Attention for Target-Aware Representations
2.4 融合 Transformer-XL 的思想2.4 Incorporating Ideas from Transformer-XL
2.5 多段建模2.5 Modeling Multiple Segments
2.6 讨论2.6 Discussion
3.1 预训练与实现3.1 Pretraining and Implementation
3.2 与 BERT 的公平比较3.2 Fair Comparison with BERT
3.3 与 RoBERTa 的比较：规模扩展3.3 Comparison with RoBERTa: Scaling Up