We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e. experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs. Even though each token only sees two experts, the selected experts can be different at each timestep. As a result, each token has access to 47B parameters, but only uses 13B active parameters during inference. Mixtral was trained with a context size of 32k tokens and it outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks. We also provide a model fine-tuned to follow instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both the base and instruct models are released under the Apache 2.0 license.
核心贡献 · Key contributions
提出 Mixtral 8x7B，一种稀疏专家混合（SMoE）语言模型，总参数 47B 但每 token 仅 13B 活跃参数。 Introduces Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model with 47B total parameters but only 13B active per token.
在所有基准测试中超越或持平 Llama 2 70B 和 GPT-3.5，尤其在数学、代码和多语言任务上。 Outperforms or matches Llama 2 70B and GPT-3.5 across all benchmarks, especially in math, code, and multilingual tasks.
提供经 SFT 和 DPO 微调的 Mixtral 8x7B-Instruct，在人类评估中超越 GPT-3.5 Turbo、Claude-2.1、Gemini Pro 和 Llama 2 70B 聊天模型。 Provides Mixtral 8x7B-Instruct, fine-tuned with SFT and DPO, surpassing GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B chat on human evaluations.
在 32k token 上下文长度内实现 100%的密钥检索准确率，展示了有效的长上下文处理能力。 Achieves 100% passkey retrieval accuracy across context lengths up to 32k tokens, demonstrating effective long-context handling.
在 Apache 2.0 许可下发布基础和指令模型，通过 vLLM 和 Megablocks 提供开源推理栈。 Releases both base and instruct models under Apache 2.0 license, with open-source inference stack via vLLM and Megablocks.
与 Llama 2 相比，在 BBQ 和 BOLD 基准上偏见减少，情感分布更积极。 Shows reduced biases on BBQ and BOLD benchmarks compared to Llama 2, with more positive sentiment profiles.
局限 · Limitations
内存成本与 47B 稀疏参数成正比，服务时需要大量 GPU 内存。 Memory costs are proportional to 47B sparse parameters, requiring significant GPU memory for serving.
SMoE 层引入路由开销和内存负载增加，小批量时效率降低。 SMoE layers introduce routing overhead and increased memory loads, reducing efficiency for small batch sizes.
专家分配未显示明确的领域专业化，限制了针对性专家优化的潜力。 Expert assignment shows no clear domain specialization, limiting potential for targeted expert optimization.
专家选择的高局部性可能导致专家并行中的负载不均，需要仔细调度。 High locality in expert selection may cause load imbalance in Expert Parallelism, requiring careful scheduling.
与 Llama 2 论文的评估差异（如 MBPP 子集、无上下文的 TriviaQA）可能影响直接比较。 Evaluation differences from Llama 2 paper (e.g., MBPP subset, TriviaQA without context) may affect direct comparisons.