2. Local and global grouped-query attention Thinking Machines Lab released Inkling, a 975B-parameter open-weight Mixture-of-Experts (MoE) model. It activates 41B parameters per token and supports a context window of up to 1,048,576 tokens. Those numbers put Inkling in the same general size class as Kimi K2.5 and GLM-5.2. Yet the architecture has several details I have seen less often, including short convolutions inside every decoder block and a learned relative-position bias in place of RoPE.
核心贡献 · Key contributions
推出了 Inkling，一个 9750 亿参数的开源 MoE 模型，激活参数 410 亿，上下文窗口达 100 万 token。 Introduces Inkling, a 975B-parameter open-weight MoE model with 41B active parameters and a 1M-token context window.
在每个解码器块中使用短卷积，并用学习到的相对位置偏置替代 RoPE。 Uses short convolutions in every decoder block and a learned relative-position bias instead of RoPE.
采用 5:1 的局部-全局注意力模式，结合滑动窗口和全局注意力层。 Employs a 5-to-1 local-global attention pattern with sliding-window and global attention layers.
在包含文本、图像、音频和视频的 45T token 上训练，后训练使用强化学习和 3000 万次 rollout。 Trains on 45T tokens across text, images, audio, and video, with post-training using RL and 30M rollouts.
实现了一种努力控制机制，无需单独检查点即可调整输出长度和基准性能。 Implements an effort control mechanism that adjusts output length and benchmark performance without separate checkpoints.
报告了具有竞争力的基准，包括在 IFBench 和 SimpleQA Verified 上高于 GLM-5.2 的分数。 Reports competitive benchmarks, including higher IFBench and SimpleQA Verified scores than GLM-5.2.
局限 · Limitations
未提供短卷积、嵌入归一化或相对位置偏置的消融实验，其影响尚不明确。 No ablations are provided for short convolutions, embedding normalization, or relative-position bias, leaving their impact unclear.
基准比较是发布时的快照，混合了外部和内部测试工具，小差距难以解释。 Benchmark comparisons are release-time snapshots with mixed external and internal harnesses, making small gaps hard to interpret.
与 Kimi K2.5 和 GLM-5.2 的吞吐量比较需要相同的量化、批大小和硬件，但未提供。 Throughput comparisons with Kimi K2.5 and GLM-5.2 require identical quantization, batch size, and hardware, which are not provided.
全局层中的相对位置偏置限制在 1,024 token 内，可能影响超出该范围的长上下文注意力。 The relative-position bias is limited to 1,024 tokens in global layers, potentially affecting long-context attention beyond that span.
模型对开源教师模型合成数据的依赖可能继承其偏见或局限。 The model's reliance on synthetic data from open-weight teachers may inherit their biases or limitations.
论文章节 · Sections（共 6）
概述Overview
41B 活跃参数规模The 41B active footprint
局部与全局分组查询注意力Local and global grouped-query attention