We introduce Gemma 3, a multimodal addition to the Gemma family of lightweight open models, ranging in scale from 1 to 27 billion parameters. This version introduces vision understanding abilities, a wider coverage of languages and longer context - at least 128K tokens. We also change the architecture of the model to reduce the KV-cache memory that tends to explode with long context. This is achieved by increasing the ratio of local to global attention layers, and keeping the span on local attention short. The Gemma 3 models are trained with distillation and achieve superior performance to Gemma 2 for both pre-trained and instruction finetuned versions. In particular, our novel post-training recipe significantly improves the math, chat, instruction-following and multilingual abilities, making Gemma3-4B-IT competitive with Gemma2-27B-IT and Gemma3-27B-IT comparable to Gemini-1.5-Pro across benchmarks. We release all our models to the community.
核心贡献 · Key contributions
推出 Gemma 3，一种具备视觉理解、128K 上下文和多语言支持的多模态模型。 Introduces Gemma 3, a multimodal model with vision understanding, 128K context, and multilingual support.
提出 5:1 交错局部与全局注意力层，以减少长上下文的 KV 缓存内存。 Proposes a 5:1 interleaving of local and global attention layers to reduce KV-cache memory for long contexts.
通过蒸馏和新颖的后训练方案，在数学、对话和多语言能力上超越 Gemma 2。 Achieves superior performance over Gemma 2 via distillation and a novel post-training recipe improving math, chat, and multilingual abilities.
证明 Gemma 3 27B IT 在基准测试中与 Gemini 1.5 Pro 竞争。 Demonstrates that Gemma 3 27B IT is competitive with Gemini 1.5 Pro on benchmarks.
引入 Pan & Scan 方法，在推理时灵活处理图像分辨率。 Introduces Pan & Scan for flexible image resolution handling during inference.
向社区发布所有模型，包括通过量化感知训练获得的量化版本。 Releases all models to the community, including quantized versions via QAT.
局限 · Limitations
视觉编码器冻结且固定 896x896 分辨率，限制了适应性。 Vision encoder is frozen and operates at fixed 896x896 resolution, limiting adaptability.
长上下文性能在 128K 词元后下降，无法保证外推。 Long context performance degrades beyond 128K tokens, with no extrapolation guarantee.
尽管去污染，记忆风险仍然存在，尤其是罕见序列。 Memorization risk remains despite decontamination, especially for rare sequences.
安全评估依赖合成查询，可能无法覆盖所有现实危害。 Safety evaluations rely on synthetic queries and may not cover all real-world harms.
多语言覆盖有所改善，但语言间仍不平衡。 Multilingual coverage is improved but still imbalanced across languages.
论文章节 · Sections（共 24）
摘要Abstract
1 引言1 Introduction
2 模型架构2 Model Architecture
2.1 视觉模态2.1 Vision modality
2.2 预训练2.2 Pre-training
2.3 量化感知训练2.3 Quantization Aware Training
2.4 计算基础设施2.4 Compute Infrastructure
3 指令微调3 Instruction-Tuning
4 最终模型评估4 Evaluation of final models
4.1 LMSYS 聊天机器人竞技场4.1 LMSYS Chatbot Arena
4.2 标准基准测试4.2 Standard benchmarks
5 消融实验5 Ablations
5.1 预训练能力探测5.1 Pre-training ability probing
5.2 局部:全局注意力层5.2 Local:Global attention layers
5.3 启用长上下文5.3 Enabling long context
5.4 小教师与大教师5.4 Small versus large teacher
5.5 视觉编码器5.5 Vision encoder
6 记忆与隐私6 Memorization and Privacy
7 责任、安全与保障7 Responsibility, Safety, Security
7.1 治理与评估7.1 Governance & Assessment
7.2 安全策略与训练时缓解措施7.2 Safety policies and train-time mitigations
7.3 保障评估7.3 Assurance Evaluations
7.4 我们负责任开放模型的方法7.4 Our approach to responsible open models