We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.
核心贡献 · Key contributions
证明仅使用公开数据集即可训练出最先进的大语言模型，无需专有数据。 Demonstrates that state-of-the-art LLMs can be trained using only publicly available datasets, without proprietary data.
LLaMA-13B 在多数基准上超越 GPT-3（175B），体积却小 10 倍。 LLaMA-13B outperforms GPT-3 (175B) on most benchmarks while being 10x smaller.
LLaMA-65B 达到与 Chinchilla-70B 和 PaLM-540B 相当的性能。 LLaMA-65B achieves competitive performance with Chinchilla-70B and PaLM-540B.
向研究社区发布所有模型规模（7B 至 65B），促进大语言模型的普及。 Releases all model sizes (7B to 65B) to the research community to democratize LLM access.
表明使用比缩放定律推荐更多的 token 进行训练可提升推理效率。 Shows that training on more tokens than recommended by scaling laws improves inference efficiency.
局限 · Limitations
模型未针对代码或数学推理进行微调，限制了在专门任务上的性能。 Models are not fine-tuned for code or mathematical reasoning, limiting performance on specialized tasks.
对毒性和偏见的评估仅限于少数基准，未能全面捕捉风险。 Evaluation of toxicity and bias is limited to a few benchmarks, not fully capturing risks.
训练数据包含的书籍和学术论文有限，可能影响 MMLU 等知识密集型任务。 Training data includes limited books and academic papers, potentially affecting knowledge-intensive tasks like MMLU.
指令微调仅进行了初步探索，性能仍低于最先进水平。 Instruction fine-tuning was only briefly explored, with performance still below state-of-the-art.
训练过程的碳足迹巨大，估计为 1,015 吨二氧化碳当量。 Carbon footprint of training is substantial, estimated at 1,015 tCO2eq.
论文章节 · Sections（共 25）
摘要Abstract
1 引言1 Introduction
2 方法2 Approach
2.1 预训练数据2.1 Pre-training Data
2.2 架构2.2 Architecture
2.3 优化器2.3 Optimizer
2.4 高效实现2.4 Efficient implementation
3 主要结果3 Main results
3.1 常识推理3.1 Common Sense Reasoning
3.2 闭卷问答3.2 Closed-book Question Answering
3.3 阅读理解3.3 Reading Comprehension
3.4 数学推理3.4 Mathematical reasoning
3.5 代码生成3.5 Code generation
3.6 大规模多任务语言理解3.6 Massive Multitask Language Understanding
3.7 训练过程中的性能演变3.7 Evolution of performance during training