Kimi K3 架构图是针对昨天发布的大型开放权重模型的，附带一些观察和思考。1. 是的，它看起来相对复杂，但本质上它是他们去年发布的 Kimi Linear 模型的规模化生产版本（从 48B 扩展到 2.8T；K3 是目前最大的开放权重模型）。2. 与 Kimi Linear 相比，唯一的新组件是 LatentMoE。我在下图中省略了它，因为图已经很拥挤了，但它本质上与 Nemotron 3 Ultra 中的 LatentMoE 相同（如果你好奇，可以在我的 LLM 架构画廊中找到）。这里的想法是压缩（下投影）大型线性层，类似于多头潜在注意力。
The Kimi K3 architecture figure for yesterday’s big open-weight model release, along with some observations and thoughts. 1. Yes, it looks relatively complicated, but it’s essentially a scaled-up production version of their Kimi Linear model they released last year (scaled up from 48B to 2.8T; K3 is by far the biggest open-weight model right now) 2. The one new component compared to Kimi Linear is the LatentMoE. I omitted it in the figure below since it's already very crowded, but that's essentially the same LatentMoE as in Nemotron 3 Ultra (you can find it in my LLM Architecture Gallery if you are curious). The idea here is to compress (down-project) large linear layers similar to multi-head latent attention.
核心贡献 · Key contributions
Kimi K3 是 Kimi Linear 的规模化生产版本，参数从 48B 扩展到 2.8T，成为目前最大的开源权重模型。 Kimi K3 is a scaled-up production version of Kimi Linear, scaled from 48B to 2.8T parameters, making it the largest open-weight model.
引入 LatentMoE，通过下投影压缩大型线性层，类似于多头潜在注意力，提升推理效率。 Introduces LatentMoE, which compresses large linear layers via down-projection, similar to multi-head latent attention, improving inference efficiency.
采用注意力残差改进跨层残差路径，持续改善验证损失和下游性能，且开销适中。 Adopts attention residuals to improve residual paths across layers, consistently improving validation loss and downstream performance with modest overhead.
移除所有 RoPE 层，全面使用 NoPE（无位置嵌入），这在前沿级模型中尚属首次。 Removes all RoPE layers, using NoPE (No Positional Embeddings) throughout, a first for a frontier-level model.
架构复杂，可能增加实现和维护难度。 The architecture is complex, potentially increasing implementation and maintenance difficulty.
注意力残差增加约 4% 训练成本和 2% 推理成本，对大规模部署可能影响显著。 Attention residuals add about 4% training cost and 2% inference cost, which may be significant for large-scale deployment.
与 RoPE 相比，NoPE 可能限制外推到更长上下文的能力，尽管未明确测试。 NoPE may limit extrapolation to longer contexts compared to RoPE, though not explicitly tested.
模型性能在发布时进行基准测试，长期泛化性和稳健性仍有待观察。 The model's performance is benchmarked at release time; long-term generalization and robustness remain to be seen.