Hunyuan-TurboS：通过 Mamba-Transformer 协同与自适应思维链推进大语言模型

Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thought

孙兴武 Xingwu Sun · Tencent Hunyuan · 2025-05-21 · arXiv:2505.15431 ↗ · 被引 24

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

随着大语言模型（LLM）的快速发展，我们推出了 Hunyuan-TurboS，一种新颖的大型混合 Transformer-Mamba 混合专家（MoE）模型。它协同结合了 Mamba 在长序列处理上的高效性与 Transformer 在上下文理解上的卓越能力。Hunyuan-TurboS 具备自适应长-短思维链（CoT）机制，可针对简单查询动态切换快速响应模式，针对复杂问题切换深度“思考”模式，从而优化计算资源。在架构上，这个拥有 560B 总参数（56B 激活参数）的模型采用了 128 层（Mamba2、Attention、FFN）结构，并创新性地使用了 AMF/MF 块模式。更快的 Mamba2 保证了线性复杂度，分组查询注意力（GQA）最小化了 KV 缓存，而 FFN 则采用了 MoE 结构。该模型在 16T 高质量 token 上进行了预训练，支持 256K 上下文长度，是业界首个大规模部署的 Mamba 模型。我们全面的后训练策略通过监督微调（300 万条指令）、新颖的自适应长-短 CoT 融合方法、用于迭代改进的多轮审议学习，以及针对 STEM 和通用指令遵循的两阶段大规模强化学习过程，增强了模型能力。评估结果显示其性能强劲：在 LMSYS Chatbot Arena 上总体排名第七，得分 1356，优于 Gemini-2.0-Flash-001（1352）和 o4-mini-2025-04-16（1345）等领先模型。TurboS 在 23 个自动化基准测试中平均得分 77.9%。Hunyuan-TurboS 在高效能与低推理成本之间取得了平衡，以低于许多推理模型的成本提供了强大的能力，为高效大规模预训练模型树立了新范式。

As Large Language Models (LLMs) rapidly advance, we introduce Hunyuan-TurboS, a novel large hybrid Transformer-Mamba Mixture of Experts (MoE) model. It synergistically combines Mamba's long-sequence processing efficiency with Transformer's superior contextual understanding. Hunyuan-TurboS features an adaptive long-short chain-of-thought (CoT) mechanism, dynamically switching between rapid responses for simple queries and deep "thinking" modes for complex problems, optimizing computational resources. Architecturally, this 56B activated (560B total) parameter model employs 128 layers (Mamba2, Attention, FFN) with an innovative AMF/MF block pattern. Faster Mamba2 ensures linear complexity, Grouped-Query Attention minimizes KV cache, and FFNs use an MoE structure. Pre-trained on 16T high-quality tokens, it supports a 256K context length and is the first industry-deployed large-scale Mamba model. Our comprehensive post-training strategy enhances capabilities via Supervised Fine-Tuning (3M instructions), a novel Adaptive Long-short CoT Fusion method, Multi-round Deliberation Learning for iterative improvement, and a two-stage Large-scale Reinforcement Learning process targeting STEM and general instruction-following. Evaluations show strong performance: overall top 7 rank on LMSYS Chatbot Arena with a score of 1356, outperforming leading models like Gemini-2.0-Flash-001 (1352) and o4-mini-2025-04-16 (1345). TurboS also achieves an average of 77.9% across 23 automated benchmarks. Hunyuan-TurboS balances high performance and efficiency, offering substantial capabilities at lower inference costs than many reasoning models, establishing a new paradigm for efficient large-scale pre-trained models.

核心贡献 · Key contributions

• 提出 Hunyuan-TurboS，首个工业部署的大规模混合 Transformer-Mamba MoE 模型，激活参数 560 亿。
  Proposes Hunyuan-TurboS, the first industry-deployed large-scale hybrid Transformer-Mamba MoE model with 56B activated parameters.
• 引入自适应长短思维链融合，动态切换推理深度以优化算力。
  Introduces adaptive long-short chain-of-thought fusion, dynamically switching reasoning depth to optimize compute.
• 设计新颖的 AMF/MF 块模式，结合 Mamba2、分组查询注意力和 MoE 以提高效率。
  Designs novel AMF/MF block pattern combining Mamba2, Grouped-Query Attention, and MoE for efficiency.
• 开发多轮审议学习，结合 LLM 评审团和人工监督进行迭代改进。
  Develops multi-round deliberation learning with LLM judge ensemble and human oversight for iterative improvement.
• 实施两阶段大规模强化学习（GRPO），针对 STEM 推理和通用指令遵循。
  Implements two-stage large-scale reinforcement learning (GRPO) targeting STEM reasoning and general instruction-following.
• 在 LMSYS Chatbot Arena 取得第 7 名（1356 分），23 项基准平均 77.9%，推理成本更低。
  Achieves top 7 LMSYS Chatbot Arena score (1356) and 77.9% average on 23 benchmarks with lower inference cost.

局限 · Limitations

• 评估限于 23 项自动化基准；实际部署场景可能揭示额外局限。
  Evaluation limited to 23 automated benchmarks; real-world deployment scenarios may reveal additional limitations.
• 自适应思维链机制可能难以泛化到难度模糊或新颖推理模式的领域。
  Adaptive CoT mechanism may not generalize well to domains with ambiguous difficulty or novel reasoning patterns.
• 依赖高质量预训练数据（16T 词元）可能限制资源受限团队的可复现性。
  Dependence on high-quality pre-training data (16T tokens) may limit reproducibility for resource-constrained teams.
• 混合 Mamba-Transformer 架构增加了推理优化和部署的工程复杂性。
  Hybrid Mamba-Transformer architecture introduces engineering complexity for inference optimization and deployment.
• 两阶段 RL 需要大量奖励工程和领域特定数据，可能限制可扩展性。
  Two-stage RL requires extensive reward engineering and domain-specific data, potentially limiting scalability.

论文章节 · Sections（共 19）

• 摘要 Abstract
• 1 引言 1 Introduction
• 2 预训练 2 Pre-Training
• 2.1 预训练数据 2.1 Data for Pre-training
• 2.2 模型架构 2.2 Model Architecture
• 2.3 退火 2.3 Annealing
• 2.4 长上下文扩展 2.4 Long-Context Extension
• 2.5 预训练模型评估 2.5 Evaluations on Pre-Trained Model
• 3 后训练 3 Post-training
• 3.1 监督微调 3.1 Supervised Fine-tuning
• 3.2 自适应长短思维链融合 3.2 Adaptive Long-short Chain-of-Thought Fusion
• 3.3 深思学习 3.3 Deliberation Learning
• 3.4 通用奖励系统 3.4 General Reward System
• 3.5 强化学习训练 3.5 RL Training
• 4.1 LMSYS 聊天机器人竞技场 4.1 LMSYS Chatbot Arena
• 4.2 自动评估结果 4.2 Auto Evaluation Results
• 5 基础设施 5 Infrastructures
• 6 结论 6 Conclusion
• 7 作者 7 Authors

阅读逐段中英对照全文 →