We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family. Designed to advance compute efficiency and reasoning, the Gemma 4 model suite features dense and Mixture-of-Experts architectures, ranging from 2.3B to 31B parameters. Alongside improved vision and audio encoders for all model sizes, we propose a unified, encoder-free architecture for our 12B model, which ingests raw audio and image patches. Furthermore, we integrate a thinking mode, enabling Gemma models to generate reasoning traces prior to responding. We improve inference speed, memory, and compute efficiency, as well as long-context abilities through critical design choices. Gemma 4 establishes a leap in performance across STEM, multimodal, and long-context benchmarks, and rivals larger, frontier open models in human-rated tasks.
核心贡献 · Key contributions
提出了 Gemma 4，包含开放权重的原生多模态模型，采用稠密和 MoE 架构，参数规模高达 31B。 Introduces Gemma 4 with open-weight natively multimodal models and dense/MoE architectures up to 31B parameters.
集成了思考模式，在响应前生成推理轨迹，提升了数学和代码任务上的性能。 Integrates a thinking mode that generates reasoning traces before responding, improving performance on math and code.
为 12B 模型提出了统一的免编码器架构，直接处理原始音频片段和图像块。 Proposes a unified encoder-free architecture for the 12B model, processing raw audio patches and image patches directly.
通过局部-全局注意力比例、KV 缓存共享和 p-RoPE 提升了长上下文效率。 Enhances long-context efficiency via local-to-global attention ratios, KV cache sharing, and p-RoPE.
通过多词元预测草稿器和量化感知训练提升了计算和内存效率。 Improves compute and memory efficiency with multi-token prediction drafters and quantization-aware training.
在人工评分的任务上达到与更大规模前沿开放模型相当的性能，相比 Gemma 3 有显著飞跃。 Achieves performance rivaling larger frontier open models on human-rated tasks, with a leap over Gemma 3.
局限 · Limitations
免编码器架构仅应用于 12B 模型，未推广到其他尺寸。 The encoder-free architecture is only applied to the 12B model, not generalized to other sizes.
思考模式增加了响应延迟，可能阻碍实时应用。 Thinking mode increases response latency, which may hinder real-time applications.
模型比最大的前沿模型（如 GPT-4）小，可能限制了某些任务的性能。 Models are smaller than the largest frontier models (e.g., GPT-4), potentially limiting performance on some tasks.
安全评估基于内部测试，仍需独立验证。 Safety evaluations are based on internal testing; independent validation remains needed.
MoE 变体在训练和推理中可能遇到负载均衡挑战。 The MoE variant may encounter load balancing challenges during training and inference.
论文章节 · Sections（共 23）
摘要Abstract
Gemma 4 技术报告Gemma 4 Technical Report
1 引言1 Introduction
2 模型架构2 Model Architecture
2.1 视觉模态2.1 Vision modality
2.2 音频模态2.2 Audio modality
2.3 无编码器架构2.3 Encoder-free architecture
2.4 预训练2.4 Pre-training
2.5 量化感知训练2.5 Quantization-Aware Training
2.6 多词元预测草稿器2.6 Multi-Token Prediction Drafter
高效的 MTP 解码Efficient MTP Decoding.
2.7 算力基础设施2.7 Compute Infrastructure
3 指令调优3 Instruction Tuning
最终模型评估4 Evaluation of final models
4.1 人类评估4.1 Human evaluation
4.2 静态基准测试4.2 Static benchmarks
责任、安全、保障5 Responsibility, Safety, Security
治理与评估5.1 Governance & Assessment
5.2 安全策略与训练时缓解措施5.2 Safety Policies and Train-Time Mitigations
5.3 安全评估5.3 Safety Evaluations
5.4 伦理考量与风险缓解5.4 Ethical Considerations and Risk Mitigation
5.5 我们的负责任的开放模型方法5.5 Our Approach to Responsible Open Models