In this paper, we introduce Hunyuan-Large, which is currently the largest open-source Transformer-based mixture of experts model, with a total of 389 billion parameters and 52 billion activation parameters, capable of handling up to 256K tokens. We conduct a thorough evaluation of Hunyuan-Large's superior performance across various benchmarks including language understanding and generation, logical reasoning, mathematical problem-solving, coding, long-context, and aggregated tasks, where it outperforms LLama3.1-70B and exhibits comparable performance when compared to the significantly larger LLama3.1-405B model. Key practice of Hunyuan-Large include large-scale synthetic data that is orders larger than in previous literature, a mixed expert routing strategy, a key-value cache compression technique, and an expert-specific learning rate strategy. Additionally, we also investigate the scaling laws and learning rate schedule of mixture of experts models, providing valuable insights and guidances for future model development and optimization. The code and checkpoints of Hunyuan-Large are released to facilitate future innovations and applications. Codes: https://github.com/Tencent/Hunyuan-Large Models: https://huggingface.co/tencent/Tencent-Hunyuan-Large
核心贡献 · Key contributions
开源了最大的基于 Transformer 的 MoE 模型，总参数量 389B，激活参数量 52B。 Open-sourced the largest Transformer-based MoE model with 389B total and 52B activated parameters.
提出了混合专家路由策略，包含共享专家和专用专家，以及回收路由以减少 token 丢弃。 Proposed mixed expert routing with shared and specialized experts, and recycle routing to reduce token dropping.
基于有效批次大小差异，开发了专家特定的学习率缩放策略。 Developed expert-specific learning rate scaling based on effective batch size differences.
研究了 MoE 缩放定律，指导最优模型规模和训练数据量。 Investigated MoE scaling laws to guide optimal model size and training data volume.
在多种基准测试中取得优异性能，超越 LLama3.1-70B，与 LLama3.1-405B 相当。 Achieved superior performance on diverse benchmarks, outperforming LLama3.1-70B and matching LLama3.1-405B.
局限 · Limitations
评估仅限于中英文基准测试，多语言泛化能力未充分评估。 Evaluation limited to English and Chinese benchmarks; multilingual generalization not fully assessed.
合成数据质量依赖于种子数据和生成模型，可能引入偏差。 Synthetic data quality depends on seed data and generation models, which may introduce biases.
长上下文能力测试至 256K token，超出此长度的性能未知。 Long-context capability tested up to 256K tokens; performance beyond this length is unknown.
后训练对齐依赖 DPO 和离线/在线数据，奖励黑客问题未完全缓解。 Post-training alignment relies on DPO with offline and online data; potential reward hacking not fully mitigated.
缩放定律实验基于小模型（10M-1B），外推至 52B 可能存在不准确性。 Scaling law experiments conducted on small models (10M-1B); extrapolation to 52B may have inaccuracies.
论文章节 · Sections（共 15）
摘要Abstract
1 引言1 Introduction
2 预训练2 Pre-Training
2.1 数据与分词器2.1 Data and Tokenizer
2.2 模型结构2.2 Model Structure
2.3 预训练方案2.3 Pre-Training Recipes
3 后训练3 Post-Training
3.1 监督微调3.1 Supervised Fine-Tuning
3.2 基于人类反馈的强化学习3.2 Reinforcement Learning from Human Feedback