Llama 3 模型群

The Llama 3 Herd of Models

Aaron Grattafiori Aaron Grattafiori · · 2024-07-31 · arXiv:2407.21783 ↗ · 被引 17476

摘要 · Abstract

现代人工智能系统由基础模型驱动。本文介绍了一组新的基础模型，称为 Llama 3。这是一个语言模型群，原生支持多语言、编码、推理和工具使用。我们的最大模型是一个稠密 Transformer，拥有 405B 参数和高达 128K token 的上下文窗口。本文对 Llama 3 进行了广泛的实证评估。我们发现，Llama 3 在众多任务上提供了与 GPT-4 等领先语言模型相当的质量。我们公开发布了 Llama 3，包括 405B 参数语言模型的预训练和后训练版本，以及用于输入和输出安全的 Llama Guard 3 模型。本文还介绍了通过组合方法将图像、视频和语音能力集成到 Llama 3 中的实验结果。我们观察到，这种方法在图像、视频和语音识别任务上表现出与最先进技术相当的水平。由此产生的模型尚未广泛发布，因为它们仍在开发中。

Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.

核心贡献 · Key contributions

推出 Llama 3 系列模型，包括 8B、70B 和 405B 参数的稠密语言模型，上下文窗口高达 128K。
Introduce Llama 3 herd of 8B, 70B, and 405B dense language models with up to 128K context window.
开发了一种数据整理流程，使用 15T 词元进行训练，提高了数据质量与多样性。
Develop a data curation pipeline that trains on 15T tokens, improving quality and diversity.
建立了能够准确预测下游基准性能的缩放定律，为模型规模选择提供指导。
Establish scaling laws that accurately predict downstream benchmark performance, guiding model size selection.
采用监督微调（SFT）、拒绝采样（RS）和直接偏好优化（DPO）的后训练方案，使模型性能与 GPT-4 相当。
Post-training recipe using SFT, rejection sampling, and DPO yields competitive performance with GPT-4.
公开了预训练和后训练版本的 405B 模型，以及用于输入输出安全的 Llama Guard 3 模型。
Release pre-trained and post-trained 405B models plus Llama Guard 3 safety model.
展示了通过组合方式集成图像、视频和语音能力的多模态扩展，其性能可与现有最优方法相媲美。
Demonstrate compositional multimodal integration (image, video, speech) competitive with state-of-the-art.

局限 · Limitations

多模态扩展仍在开发中，尚未公开发布。
Multimodal extensions are still under development and not publicly released.
尽管有安全缓解措施，模型仍可能产生有害或有偏见的输出。
Despite safety mitigations, models may produce harmful or biased outputs.
训练需要巨大的算力基础设施，限制了中小机构的可复现性。
Training requires massive compute infrastructure, limiting reproducibility to large organizations.
缩放定律预测略微低估了模型性能，且在未来更大规模上可能出现偏差。
Scaling-law predictions slightly underestimate performance; may diverge for future scales.
基于 Llama 社区许可发布，并非完全开源，存在使用限制。
Released under Llama Community License, not fully open; usage restrictions apply.

论文章节 · Sections（共 14）

摘要 Abstract
1 引言 1 Introduction
2 总体概述 2 General Overview
3 预训练 3 Pre-Training
3.1 预训练数据 3.1 Pre-Training Data
3.2 模型架构 3.2 Model Architecture
3.3 基础设施、规模扩张与效率 3.3 Infrastructure, Scaling, and Efficiency
训练方案 3.4 Training Recipe
4 后训练 4 Post-Training
4.1 建模 4.1 Modeling
4.2 后训练数据 4.2 Post-training Data
4.3 能力 4.3 Capabilities
5 结果 5 Results
5.1 预训练语言模型 5.1 Pre-trained Language Model

阅读逐段中英对照全文 →