Large language models (LLMs) face low hardware efficiency during decoding, especially for long-context reasoning tasks. This paper introduces Step-3, a 321B-parameter VLM with hardware-aware model-system co-design optimized for minimizing decoding costs. Step-3 innovates in two key dimensions: (1) A novel Multi-Matrix Factorization Attention (MFA) mechanism that significantly reduces both KV cache size and computation while maintaining high attention expressiveness, and (2) Attention-FFN Disaggregation (AFD), a distributed inference system that decouples attention and Feed-Forward Network (FFN) layers into specialized subsystems. This co-design achieves unprecedented cost efficiency: Step-3 significantly reduces theoretical decoding costs compared with models like DeepSeek-V3 and Qwen3 MoE 235B, with the gains widening at longer context. Step-3 achieves low cost while activating 38B parameters per token (more than DeepSeek-V3 and Qwen3 MoE 235B), demonstrating that hardware-aligned attention arithmetic intensity, MoE sparsity, and AFD are critical to cost-effectiveness. We perform a head-to-head comparison with DeepSeek-V3 in its favorable scenarios. Our implementation on Hopper GPUs achieves a decoding throughput of up to 4,039 tokens per second per GPU under 50ms TPOT SLA (4K context, FP8, no MTP). It is higher than DeepSeek-V3's 2,324 in the same setup and sets a new Pareto frontier for LLM decoding.
核心贡献 · Key contributions
提出 Step-3，一个 321B 参数的视觉语言模型，采用硬件感知的模型-系统协同设计，实现经济高效的解码。 Proposes Step-3, a 321B-parameter VLM with hardware-aware model-system co-design for cost-effective decoding.
引入多矩阵分解注意力机制，在保持表达能力的同时减少 KV 缓存和计算量。 Introduces Multi-Matrix Factorization Attention (MFA) to reduce KV cache and computation while maintaining expressiveness.
提出注意力-前馈网络分离的分布式推理系统，将注意力层和前馈网络层解耦为专用子系统。 Proposes Attention-FFN Disaggregation (AFD) for distributed inference, decoupling attention and FFN into specialized subsystems.
在 50ms TPOT 服务等级协议下实现高达 4039 tokens/s/GPU 的解码吞吐量，比 DeepSeek-V3 高 74%。 Achieves decoding throughput up to 4,039 tokens/s/GPU under 50ms TPOT SLA, outperforming DeepSeek-V3 by 74%.
证明硬件对齐的注意力算术强度、MoE 稀疏性和 AFD 对成本效益至关重要。 Demonstrates that hardware-aligned attention arithmetic intensity, MoE sparsity, and AFD are critical to cost-effectiveness.
通过理论成本分析表明，Step-3 在解码成本上低于 DeepSeek-V3 和 Qwen3 MoE 235B，尤其在长上下文场景下优势更大。 Provides theoretical cost analysis showing Step-3 achieves lower decoding costs than DeepSeek-V3 and Qwen3 MoE 235B, especially at longer contexts.
局限 · Limitations
评估仅限于解码阶段；预填充和训练成本未优化。 Evaluation limited to decoding phase; prefill and training costs not optimized.
结果基于特定硬件（Hopper GPU），可能不适用于其他加速器。 Results based on specific hardware (Hopper GPUs) and may not generalize to other accelerators.
理论成本分析假设理想条件（峰值 FLOPs、最大带宽），忽略实际开销。 Theoretical cost analysis assumes ideal conditions (peak FLOPs, maximum bandwidth) and ignores real-world overheads.
与 DeepSeek-V3 的比较采用对其有利的场景；其他模型可能有不同优势。 Comparison with DeepSeek-V3 uses favorable scenarios for DeepSeek-V3; other models may have different strengths.
模型-系统协同设计可能无法直接应用于现有模型，需重大修改。 Model-system co-design may not be directly applicable to existing models without significant modifications.
论文章节 · Sections（共 20）
摘要Abstract
1 引言1 Introduction
2 Step-3 模型卡2 Step-3 Model Card
3 注意力-FFN 分离3 Attention-FFN Disaggregation
3.1 设计目标3.1 Design Goals
3.2 与相关工作的比较3.2 Comparisons with Related Work
4 LLM 解码成本分析4 Cost Analysis for LLM Decoding
4.1 理论计算量和内存访问4.1 Theoretical Flops and Memory Access
4.2 理论解码成本（美元）4.2 Theoretical Decoding Cost in USD
4.3 揭示模型设计选择4.3 Demystifying Model Design Choices
5.1 匹配注意力算术强度与硬件5.1 Matching Attention Arithmetic Intensity with Hardware
5.2 讨论：量化和 MTP5.2 Discussion: Quantization and MTP
5.3 FFN 的高 MFU 批量需求5.3 FFN’s Batch Requirement for High MFU
5.4 最优 MoE 稀疏度 vs. 硬件5.4 Optimal MoE Sparsity vs. Hardware
5.5 讨论：过度稀疏的解决方案5.5 Discussion: Workaround for Over-sparsity
6 非旗舰硬件支持6 Non-Flagship Hardware Support
7.1 系统工作流和优化7.1 System Workflow and Optimizations
7.2 StepMesh：AFD 通信库7.2 StepMesh: AFD Communication Library