缩放嵌入在语言模型中优于缩放专家

Scaling Embeddings Outperforms Scaling Experts in Language Models

美团 LongCat 团队 Meituan LongCat Team · Meituan · 2026-01-29 · arXiv:2601.21204 ↗ · 被引 10

摘要 · Abstract

尽管混合专家（MoE）架构已成为大型语言模型中稀疏性缩放的标准，但它们日益面临收益递减和系统级瓶颈。在这项工作中，我们探索了嵌入缩放作为稀疏性缩放的一个强大且正交的维度。通过全面的分析和实验，我们确定了嵌入缩放相比专家缩放能够实现更优帕累托前沿的特定场景。我们系统地描述了影响这种效能的关键架构因素——从参数预算到与模型宽度和深度的相互作用。此外，通过整合定制的系统优化和推测解码，我们有效地将这种稀疏性转化为实际的推理加速。在这些见解的指导下，我们提出了 LongCat-Flash-Lite，一个 685 亿参数、约 30 亿激活的模型，从头开始训练。尽管将超过 300 亿参数分配给嵌入，LongCat-Flash-Lite 不仅超越了参数等效的 MoE 基线，而且在代理和编码领域与现有同等规模模型相比表现出卓越的竞争力。

While Mixture-of-Experts (MoE) architectures have become the standard for sparsity scaling in large language models, they increasingly face diminishing returns and system-level bottlenecks. In this work, we explore embedding scaling as a potent, orthogonal dimension for scaling sparsity. Through a comprehensive analysis and experiments, we identify specific regimes where embedding scaling achieves a superior Pareto frontier compared to expert scaling. We systematically characterize the critical architectural factors governing this efficacy -- ranging from parameter budgeting to the interplay with model width and depth. Moreover, by integrating tailored system optimizations and speculative decoding, we effectively convert this sparsity into tangible inference speedups. Guided by these insights, we introduce LongCat-Flash-Lite, a 68.5B parameter model with ~3B activated trained from scratch. Despite allocating over 30B parameters to embeddings, LongCat-Flash-Lite not only surpasses parameter-equivalent MoE baselines but also exhibits exceptional competitiveness against existing models of comparable scale, particularly in agentic and coding domains.

核心贡献 · Key contributions

识别出嵌入缩放优于专家缩放的帕累托前沿区域。
Identifies regimes where embedding scaling achieves superior Pareto frontier over expert scaling.
系统刻画了影响嵌入缩放效果的架构因素。
Systematically characterizes architectural factors governing embedding scaling efficacy.
提出 N-gram 缓存和同步内核以减少 I/O 瓶颈。
Proposes N-gram Cache and synchronized kernels to reduce I/O bottlenecks.
引入 LongCat-Flash-Lite，一个 68.5B 参数、约 3B 激活参数的模型。
Introduces LongCat-Flash-Lite, a 68.5B model with ~3B activated parameters.
在智能体和编码任务上展现出优于同类模型的性能。
Demonstrates superior performance in agentic and coding tasks over comparable models.
验证了在高稀疏度下嵌入缩放优于专家缩放。
Validates that embedding scaling outperforms expert scaling at high sparsity levels.

局限 · Limitations

嵌入缩放超过总参数最优比例后收益递减。
Embedding scaling shows diminishing returns beyond optimal proportion of total parameters.
N-gram 嵌入存在哈希冲突，阻碍学习效率。
N-gram Embedding suffers from hash collisions that impede learning efficiency.
嵌入缩放的效果随模型深度增加而减弱。
Effectiveness of embedding scaling diminishes with increasing model depth.
需要大有效批量才能充分利用 GPU 内存带宽。
Requires large effective batch size to fully utilize GPU memory bandwidth.
需要内核融合等系统优化来减轻内核启动开销。
System optimizations like kernel fusion are needed to mitigate kernel launch overhead.

论文章节 · Sections（共 21）

摘要 Abstract
1 引言 1 Introduction
2 N 元嵌入层 2 N-gram Embedding Layer
3 专家与嵌入扩展的对比分析 3 Comparative Analysis of Expert and Embedding Scaling
3.1 N 元嵌入集成的最佳时机 3.1 Optimal Timing for N-gram Embedding Integration
3.2 集成策略 3.2 Integration Strategy
3.3 模型宽度与深度的扩展特性 3.3 Scaling Properties across Model Width and Depth
4.1 减少 MoE 激活参数 4.1 Reduction of MoE Activation Parameters
4.2 优化嵌入查找 4.2 Optimized Embedding Lookup
4.3 重新思考 N 元嵌入优化：推测解码的作用 4.3 Rethinking N-gram Embedding Optimization: The Role of Speculative Decoding
5 与逐层嵌入的集成 5 Integration with Per-Layer Embedding
5.1 逐层嵌入 5.1 Per-Layer Embedding
5.2 逐层 N 元嵌入 5.2 Per-Layer N-gram Embedding
5.3 实证比较 5.3 Empirical Comparison
6 LongCat-Flash-Lite 6 LongCat-Flash-Lite
6.1 模型信息 6.1 Model Information
6.2 基础模型评估 6.2 Base Model Evaluation
6.3 对话模型评估 6.3 Chat Model Evaluation
6.4 优化内核的快速推理 6.4 Fast Inference with Optimized Kernels
7 结论 7 Conclusions
8 致谢 8 Acknowledgement

阅读逐段中英对照全文 →