We introduce DiffusionGemma, an experimental open-weight language model that uses discrete diffusion to generate text at exceptionally high speed. Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, avoiding the sequential decoding bottleneck of conventional autoregressive (AR) large language models. Instead of training from scratch, we obtain DiffusionGemma by fine-tuning the mixture-of-experts Gemma 4 model with 3.8B activated and 25.2B total parameters. Our compute-efficient two-stage training pipeline uses fewer than 10% of the starting AR model's total training token budget. The first stage uses supervised fine-tuning to teach bidirectional denoising, while the second stage combines reinforcement learning with sampler distillation to jointly improve generation quality and inference efficiency. DiffusionGemma establishes a new Pareto frontier for the trade-off between generation speed and model capability. Averaged across our full evaluation suite, it generates around 20 tokens per forward pass and achieves roughly 1,500 output tokens per second on a single NVIDIA H100 GPU, which is substantially faster than AR models even with state-of-the-art speculative decoding. DiffusionGemma also retains the starting model's support for thinking mode, multimodal inputs, and long contexts. Despite diffusion fine-tuning, it remains capable of AR generation with only minor performance degradation, suggesting a path toward hybrid diffusion-AR decoding.
核心贡献 · Key contributions
提出了 DiffusionGemma，一个从 Gemma 4 微调而来的开放权重文本扩散模型，实现了高速并行生成。 Introduces DiffusionGemma, an open-weight text diffusion model fine-tuned from Gemma 4, achieving high-speed parallel generation.
采用两阶段训练流程（SFT 和 SD·RL），使用不到 AR 模型 10%的训练 token，提升了质量和效率。 Uses a two-stage training pipeline (SFT and SD·RL) with less than 10% of the AR model's training tokens, improving quality and efficiency.
在单个 H100 GPU 上实现每次前向约 20 个 token 和每秒约 1500 个 token，优于带投机解码的 AR 模型。 Achieves around 20 tokens per forward pass and ~1,500 tokens per second on a single H100 GPU, outperforming AR models with speculative decoding.
保留 AR 生成能力，支持混合扩散-AR 解码，并根据延迟和任务复杂度进行动态路由。 Retains AR generation capability, enabling hybrid diffusion-AR decoding and dynamic routing based on latency and task complexity.
在推理、编程和多模态基准上表现强劲，为速度-智能权衡设定了新的帕累托前沿。 Demonstrates strong performance on reasoning, coding, and multimodal benchmarks, setting a new Pareto frontier for speed-intelligence trade-off.
提供开源微调工具包和 LoRA 配方，支持在消费级硬件上进行领域特定适配。 Provides open-source finetuning toolkit and LoRA recipes, enabling domain-specific adaptation on consumer hardware.
局限 · Limitations
与原始 AR 模型相比，文本扩散模式存在性能下降，以质量换取速度。 Performance degradation compared to the original AR model in text-diffusion mode, trading quality for speed.
由于去噪步骤中的条件独立性，模型可能产生局部不一致或语法冲突的预测。 The model may produce local inconsistencies or conflicting grammatical predictions due to conditional independence in denoising steps.
SD·RL 目标鼓励简洁输出，可能放弃扩展推理带来的能力提升。 The SD·RL objective encourages concise outputs, potentially forgoing capability gains from extended reasoning.
模型的速度优势在低批量时最明显；在中等批量时，AR 模型获得吞吐量优势。 The model's speed advantage is most pronounced at low batch sizes; AR models gain throughput advantage at moderate batch sizes.
研究缺乏在真实流量条件下的全面实证分析，且未充分探索更大批量的内核优化。 The study lacks thorough empirical analysis under realistic traffic conditions, and kernel optimizations for larger batch sizes are not fully explored.
论文章节 · Sections（共 23）
摘要Abstract
1 引言1 Introduction
2 基于离散扩散的生成式文本建模2 Generative Text Modeling with Discrete Diffusion
2.1 概率路径与去噪2.1 Probability Paths and Denoising
3 DiffusionGemma 架构3 The DiffusionGemma Architecture
3.1 块自回归生成3.1 Block-Autoregressive Generation
3.2 推理阶段的去噪框架3.2 The Denoising Framework at Inference