Scaling inference-time compute has emerged as an important driver of LLM performance, making inference efficiency a central focus of model design alongside model quality. While the current Transformer-based models deliver strong model quality, their quadratic compute and linear memory make inference expensive. This has spurred the development of sub-quadratic models with reduced linear compute and constant memory requirements. However, many recent linear models trade off model quality and capability for algorithmic efficiency, failing on tasks such as state tracking. Moreover, their theoretically linear inference remains hardware-inefficient in practice. Guided by an inference-first perspective, we introduce three core methodological improvements inspired by the state space model (SSM) viewpoint of linear models. We combine: (1) a more expressive recurrence derived from SSM discretization, (2) a complex-valued state update rule that enables richer state tracking, and (3) a multi-input, multi-output (MIMO) formulation for better model performance without increasing decode latency. Together with architectural refinements, our Mamba-3 model achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks. At the 1.5B scale, Mamba-3 improves average downstream accuracy by 0.6 percentage points compared to the next best model (Gated DeltaNet), with Mamba-3's MIMO variant further improving accuracy by another 1.2 points for a total 1.8 point gain. Across state-size experiments, Mamba-3 achieves comparable perplexity to Mamba-2 despite using half of its predecessor's state size. Our evaluations demonstrate Mamba-3's ability to advance the performance-efficiency Pareto frontier.
核心贡献 · Key contributions
引入指数-梯形离散化方法，改进了状态-输入积分的近似精度。 Introduces exponential-trapezoidal discretization for SSMs, improving state-input integral approximation.
通过数据依赖的 RoPE 提出复数值状态转移，实现了奇偶性等状态追踪任务。 Proposes complex-valued state transitions via data-dependent RoPE, enabling state-tracking tasks like parity.
开发了 SSM 的多输入多输出（MIMO）公式，在不增加解码延迟的情况下提高推理算力。 Develops MIMO formulation for SSMs, increasing inference FLOPs without raising decode latency.
在 1.5B 规模的下游任务上，相比 Gated DeltaNet 准确率提升 1.8 个百分点。 Achieves 1.8 point accuracy gain over Gated DeltaNet at 1.5B scale on downstream tasks.
以一半的状态大小达到 Mamba-2 的困惑度，推进了性能-效率帕累托前沿。 Matches Mamba-2 perplexity with half the state size, advancing the performance-efficiency Pareto frontier.
发布了优化的 Triton 和 CuTe 内核，支持 Mamba-3 的快速训练和推理。 Releases optimized Triton and CuTe kernels for fast training and inference of Mamba-3.
局限 · Limitations
由于固定大小的状态压缩，Mamba-3 在检索任务上仍不如 Transformer。 Mamba-3 still underperforms Transformers on retrieval tasks due to fixed-size state compression.
MIMO 变体将训练算力增加 R 倍，需要更多预训练计算资源。 MIMO variant increases training FLOPs by factor R, requiring more compute for pre-training.
复数值转移可能不适用于所有模态；在视觉/音频上的有效性尚未测试。 Complex-valued transitions may not benefit all modalities; effectiveness on vision/audio untested.