GLM：基于自回归空白填充的通用语言模型预训练

GLM: General Language Model Pretraining with Autoregressive Blank Infilling

唐杰 Jie Tang · Tsinghua University · 2021-03-18 · arXiv:2103.10360 ↗ · 被引 1944

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

现有的预训练架构包括自编码模型（如 BERT）、自回归模型（如 GPT）和编码器-解码器模型（如 T5），但没有一种框架能在自然语言理解（NLU）、无条件生成和条件生成这三类主要任务中表现最佳。我们提出了一种基于自回归空白填充的通用语言模型（GLM）来解决这一挑战。GLM 通过添加二维位置编码并允许以任意顺序预测片段，改进了空白填充预训练，从而在 NLU 任务上相比 BERT 和 T5 取得了性能提升。同时，通过改变空白的数量和长度，GLM 可以针对不同类型的任务进行预训练。在 NLU、条件生成和无条件生成的广泛任务中，在相同模型大小和数据条件下，GLM 优于 BERT、T5 和 GPT，并且仅使用参数为 BERT Large 1.25 倍的单一预训练模型就取得了最佳性能，展示了其对不同下游任务的泛化能力。

There have been various types of pretraining architectures including autoencoding models (e.g., BERT), autoregressive models (e.g., GPT), and encoder-decoder models (e.g., T5). However, none of the pretraining frameworks performs the best for all tasks of three main categories including natural language understanding (NLU), unconditional generation, and conditional generation. We propose a General Language Model (GLM) based on autoregressive blank infilling to address this challenge. GLM improves blank filling pretraining by adding 2D positional encodings and allowing an arbitrary order to predict spans, which results in performance gains over BERT and T5 on NLU tasks. Meanwhile, GLM can be pretrained for different types of tasks by varying the number and lengths of blanks. On a wide range of tasks across NLU, conditional and unconditional generation, GLM outperforms BERT, T5, and GPT given the same model sizes and data, and achieves the best performance from a single pretrained model with 1.25x parameters of BERT Large , demonstrating its generalizability to different downstream tasks.

核心贡献 · Key contributions

• 提出 GLM，一种基于自回归空白填充的通用语言模型。
  Proposes GLM, a general language model based on autoregressive blank infilling.
• 引入 2D 位置编码和跨度打乱以改进空白填充预训练。
  Introduces 2D positional encodings and span shuffling to improve blank filling pretraining.
• 在单一预训练框架下统一 NLU、条件生成和无条件生成任务。
  Unifies NLU, conditional, and unconditional generation tasks under a single pretraining framework.
• 在相似模型规模下，在 NLU 和生成任务上优于 BERT、T5 和 GPT。
  Outperforms BERT, T5, and GPT on NLU and generation tasks with similar model sizes.
• 在 Yahoo Answers 等文本填充任务上取得最先进结果。
  Achieves state-of-the-art results on text infilling tasks like Yahoo Answers.
• 展示了跨不同下游任务的有效参数共享。
  Demonstrates effective parameter sharing across diverse downstream tasks.

局限 · Limitations

• 需要为 NLU 任务手动设计完形填空模式。
  Requires manual crafting of cloze patterns for NLU tasks.
• 多任务预训练可能比单任务略微降低 NLU 性能。
  Multi-task pretraining may slightly degrade NLU performance compared to single-task.
• 无双向上下文时，语言建模困惑度落后于 GPT。
  Language modeling perplexity lags behind GPT without bidirectional context.
• 评估限于英语和 SuperGLUE 等特定数据集。
  Evaluation limited to English and specific datasets like SuperGLUE.
• 扩展到更大模型需要更多计算资源。
  Scaling to larger models requires more computational resources.

论文章节 · Sections（共 15）

• 摘要 Abstract
• 1 引言 1 Introduction
• 2 GLM 预训练框架 2 GLM Pretraining Framework
• 2.1 预训练目标 2.1 Pretraining Objective
• 2.2 模型架构 2.2 Model Architecture
• 2.3 微调 GLM 2.3 Finetuning GLM
• 2.4 讨论与分析 2.4 Discussion and Analysis
• 3 实验 3 Experiments
• 3.1 预训练设置 3.1 Pretraining Setup
• 3.2 SuperGLUE 3.2 SuperGLUE
• 3.3 多任务预训练 3.3 Multi-Task Pretraining
• 3.4 消融研究 3.4 Ablation Study
• 4 相关工作 4 Related Work
• 5 结论 5 Conclusions
• 致谢 Acknowledgements

阅读逐段中英对照全文 →