In this work, we introduce Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models, ranging in scale from 2 billion to 27 billion parameters. In this new version, we apply several known technical modifications to the Transformer architecture, such as interleaving local-global attentions (Beltagy et al., 2020a) and group-query attention (Ainslie et al., 2023). We also train the 2B and 9B models with knowledge distillation (Hinton et al., 2015) instead of next token prediction. The resulting models deliver the best performance for their size, and even offer competitive alternatives to models that are 2-3 times bigger. We release all our models to the community.
核心贡献 · Key contributions
推出 Gemma 2，一个参数规模从 2B 到 27B 的开放语言模型系列。 Introduces Gemma 2, a family of open language models from 2B to 27B parameters.
在 Transformer 架构中应用交错局部-全局注意力和分组查询注意力。 Applies interleaved local-global attention and grouped-query attention to Transformer architecture.
使用知识蒸馏而非下一个词预测训练 2B 和 9B 模型。 Trains 2B and 9B models using knowledge distillation instead of next-token prediction.
在模型规模上实现最先进性能，与 2-3 倍大的模型竞争。 Achieves state-of-the-art performance for model size, competitive with models 2-3x larger.
证明蒸馏优于原始文本训练，尤其对小模型有益。 Demonstrates distillation benefits over raw text training, especially for small models.
向社区发布所有模型以促进开放研究与开发。 Releases all models to the community for open research and development.
局限 · Limitations
模型非多模态，未针对多语言能力优化。 Models are not multimodal and not optimized for multilingual capabilities.
蒸馏带来的性能提升可能无法泛化到所有任务或领域。 Performance gains from distillation may not generalize to all tasks or domains.
安全评估无法覆盖所有可能的应用程序和场景。 Safety evaluations cannot cover all possible applications and scenarios.
尽管很低，但对某些数据源仍存在训练数据记忆。 Memorization of training data, though low, still occurs for certain data sources.
模型可能对对抗性攻击和说服任务表现出脆弱性。 Models may exhibit vulnerabilities to adversarial attacks and persuasion tasks.
论文章节 · Sections（共 22）
摘要Abstract
1 引言1 Introduction
2 模型架构2 Model Architecture
3 预训练3 Pre-training
3.1 训练数据3.1 Training Data
3.2 知识蒸馏3.2 Knowledge Distillation
3.3 计算基础设施3.3 Compute Infrastructure
3.4 碳足迹3.4 Carbon Footprint
4 后训练4 Post-Training
5 消融实验5 Ablations
6 评估6 Evaluation
6.1 预训练评估6.1 Pre-training Evaluations
6.2 后训练评估6.2 Post-training Evaluations
7 记忆与隐私7 Memorization and Privacy
8 责任、安全与保障8 Responsibility, Safety, Security
8.1 影响评估8.1 Impact assessment
8.2 安全策略与训练时缓解措施8.2 Safety policies and train-time mitigations
8.3 外部基准评估8.3 External benchmark evaluations
8.4 保障评估8.4 Assurance Evaluations
8.5 我们负责任开放模型的方法8.5 Our approach to responsible open models