GLM-130B：一个开放的双语预训练模型

GLM-130B: An Open Bilingual Pre-trained Model

唐杰 Jie Tang · Zhipu AI · 2022-10-05 · arXiv:2210.02414 ↗ · 被引 1285

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

我们介绍了 GLM-130B，一个拥有 1300 亿参数的双语（英文和中文）预训练语言模型。这是开源一个至少与 GPT-3（davinci）一样好的百亿级模型的尝试，并揭示了如何成功预训练如此规模的模型。在此过程中，我们面临了许多意想不到的技术和工程挑战，特别是损失尖峰和发散问题。本文介绍了 GLM-130B 的训练过程，包括其设计选择、兼顾效率与稳定性的训练策略以及工程努力。最终的 GLM-130B 模型在广泛流行的英文基准上显著优于 GPT-3 175B（davinci），而 OPT-175B 和 BLOOM-176B 未观察到这一优势。它还在相关基准上持续且显著地优于最大的中文语言模型 ERNIE TITAN 3.0 260B。最后，我们利用 GLM-130B 独特的缩放特性，实现了无需后训练的 INT4 量化，且性能几乎无损，使其成为百亿级模型中的首个，更重要的是，允许其在 4×RTX 3090（24G）或 8×RTX 2080 Ti（11G）GPU 上进行有效推理，这是使用百亿级模型所需的最实惠的 GPU。GLM-130B 模型权重公开可访问，其代码、训练日志、相关工具包和经验教训已在 https://github.com/THUDM/GLM-130B/开源。

We introduce GLM-130B, a bilingual (English and Chinese) pre-trained language model with 130 billion parameters. It is an attempt to open-source a 100B-scale model at least as good as GPT-3 (davinci) and unveil how models of such a scale can be successfully pre-trained. Over the course of this effort, we face numerous unexpected technical and engineering challenges, particularly on loss spikes and divergence. In this paper, we introduce the training process of GLM-130B including its design choices, training strategies for both efficiency and stability, and engineering efforts. The resultant GLM-130B model offers significant outperformance over GPT-3 175B (davinci) on a wide range of popular English benchmarks while the performance advantage is not observed in OPT-175B and BLOOM-176B. It also consistently and significantly outperforms ERNIE TITAN 3.0 260B -- the largest Chinese language model -- across related benchmarks. Finally, we leverage a unique scaling property of GLM-130B to reach INT4 quantization without post training, with almost no performance loss, making it the first among 100B-scale models and more importantly, allowing its effective inference on 4$\times$RTX 3090 (24G) or 8$\times$RTX 2080 Ti (11G) GPUs, the most affordable GPUs required for using 100B-scale models. The GLM-130B model weights are publicly accessible and its code, training logs, related toolkit, and lessons learned are open-sourced at \url{https://github.com/THUDM/GLM-130B/}.

核心贡献 · Key contributions

• 开源了一个 130B 双语预训练模型，在多项英文基准上超越 GPT-3。
  Open-sourced a 130B bilingual pre-trained model outperforming GPT-3 on many English benchmarks.
• 实现无需后训练的 INT4 量化，可在 4×RTX 3090 等平价 GPU 上推理。
  Achieved INT4 quantization without post-training, enabling inference on affordable GPUs like 4×RTX 3090.
• 提出嵌入梯度收缩技术，稳定训练并防止损失尖峰。
  Proposed Embedding Gradient Shrink to stabilize training and prevent loss spikes.
• 采用 GLM 骨干网络，结合双向注意力和自回归空白填充，提升理解能力。
  Adopted GLM backbone with bidirectional attention and autoregressive blank infilling for superior comprehension.
• 引入多任务指令预训练（MIP），提升零样本性能。
  Introduced Multi-task Instruction Pre-training (MIP) to boost zero-shot performance.
• 展示了 GLM 架构中 INT4 权重量化的独特缩放定律。
  Demonstrated a unique scaling law for INT4 weight quantization in GLM architectures.

局限 · Limitations

• 少样本性能增长不如 GPT-3 显著，可能归因于 MIP 偏差。
  Performance growth with few-shot samples is less significant than GPT-3, possibly due to MIP bias.
• 训练需要 96 个 DGX-A100 节点和 60 天，限制了可复现性。
  Training required 96 DGX-A100 nodes and 60 days, limiting reproducibility.
• INT4 量化仅适用于权重，激活值仍为 FP16，限制了压缩程度。
  INT4 quantization only applies to weights; activations remain FP16, limiting compression.
• 中文基准评估仅限于零样本，未报告少样本结果。
  Evaluation on Chinese benchmarks is limited to zero-shot; few-shot results not reported.
• 开源强大 LLM 存在潜在伦理风险，但透明度可缓解。
  Potential ethical risks from open-sourcing a powerful LLM, though mitigated by transparency.

论文章节 · Sections（共 11）

• 摘要 Abstract
• 1 引言 1 Introduction
• 2 GLM-130B 的设计选择 2 The Design Choices of GLM-130B
• 3 GLM-130B 的训练稳定性 3 The Training Stability of GLM-130B
• 4 在 RTX 2080 Ti 上推理 GLM-130B 4 GLM-130B Inference on RTX 2080 Ti
• 5 实验结果 5 The Results
• 6 相关工作 6 Related Work
• 7 结论与经验 7 Conclusion and Lessons
• 致谢 Acknowledgement
• 伦理声明 Ethics Statement
• 可复现性 Reproducibility

阅读逐段中英对照全文 →