OPT：开放预训练 Transformer 语言模型

OPT: Open Pre-trained Transformer Language Models

Aaron Grattafiori Aaron Grattafiori · Meta AI · 2022-05-02 · arXiv:2205.01068 ↗ · 被引 4802

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

大型语言模型通常需要数十万计算天的训练，在零样本和少样本学习方面表现出色。由于计算成本高昂，没有大量资金很难复现这些模型。少数通过 API 提供的模型也无法获取完整权重，难以研究。我们提出了开放预训练 Transformer（OPT），一系列从 125M 到 175B 参数的仅解码器预训练 Transformer，旨在与感兴趣的研究人员全面且负责任地共享。我们展示了 OPT-175B 与 GPT-3 性能相当，但开发碳足迹仅为后者的七分之一。我们还发布了记录基础设施挑战的日志，以及用于实验所有已发布模型的代码。

Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning. Given their computational cost, these models are difficult to replicate without significant capital. For the few that are available through APIs, no access is granted to the full model weights, making them difficult to study. We present Open Pre-trained Transformers (OPT), a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters, which we aim to fully and responsibly share with interested researchers. We show that OPT-175B is comparable to GPT-3, while requiring only 1/7th the carbon footprint to develop. We are also releasing our logbook detailing the infrastructure challenges we faced, along with code for experimenting with all of the released models.

核心贡献 · Key contributions

• 发布了 OPT，一套从 125M 到 175B 参数的仅解码器预训练 Transformer，完全且负责任地与研究人员共享。
  Released OPT, a suite of decoder-only pre-trained transformers from 125M to 175B parameters, fully and responsibly shared with researchers.
• 证明 OPT-175B 与 GPT-3 性能相当，但碳足迹仅为后者的七分之一。
  Showed OPT-175B is comparable to GPT-3 while requiring only 1/7th the carbon footprint.
• 提供了基础设施挑战的详细日志和代码库，支持可重复研究。
  Provided detailed logbook of infrastructure challenges and codebase for reproducible research.
• 在 16 个 NLP 任务、对话、偏见和毒性基准上进行了广泛评估。
  Conducted extensive evaluations on 16 NLP tasks, dialogue, bias, and toxicity benchmarks.
• 在大多数任务上与 GPT-3 表现持平，但在毒性和仇恨言论检测上存在差异。
  Demonstrated parity with GPT-3 on most tasks, with variations in toxicity and hate speech detection.
• 使更广泛的研究社区能够访问大型语言模型，以研究其局限性和风险。
  Enabled broader research community access to large language models for studying limitations and risks.

局限 · Limitations

• OPT-175B 不擅长处理陈述性指令或直接问句。
  OPT-175B does not work well with declarative instructions or point-blank interrogatives.
• 倾向于重复，即使使用采样也可能陷入循环。
  Tends to be repetitive and can get stuck in loops, even with sampling.
• 产生事实错误的陈述，尤其在医疗等关键领域。
  Produces factually incorrect statements, especially in critical domains like healthcare.
• 高度倾向于生成有毒语言并强化有害刻板印象。
  High propensity to generate toxic language and reinforce harmful stereotypes.
• 评估结果随提示风格和样本数量变化，影响可重复性。
  Evaluation results vary with prompting style and number of shots, affecting replicability.

论文章节 · Sections（共 20）

• 摘要 Abstract
• 1 引言 1 Introduction
• 2.1 模型 2.1 Models
• 2.2 训练设置 2.2 Training Setup
• 2.3 预训练语料 2.3 Pre-training Corpus
• 2.4 训练效率 2.4 Training Efficiency
• 2.5 训练过程 2.5 Training Processes
• 3.1 提示与少样本 3.1 Prompting & Few-Shot
• 3.2 对话 3.2 Dialogue
• 4 偏见与毒性评估 4 Bias & Toxicity Evaluations
• 4.1 仇恨言论检测 4.1 Hate Speech Detection
• 4.2 CrowS-Pairs 4.2 CrowS-Pairs
• 4.3 StereoSet 4.3 StereoSet
• 4.4 RealToxicityPrompts 4.4 RealToxicityPrompts
• 4.5 对话安全评估 4.5 Dialogue Safety Evaluations
• 5 局限性 5 Limitations
• 6 发布考虑 6 Considerations for Release
• 7 相关工作 7 Related Work
• 8 结论 8 Conclusion
• 致谢 Acknowledgements

阅读逐段中英对照全文 →