This report introduces the Qwen2 series, the latest addition to our large language models and large multimodal models. We release a comprehensive suite of foundational and instruction-tuned language models, encompassing a parameter range from 0.5 to 72 billion, featuring dense models and a Mixture-of-Experts model. Qwen2 surpasses most prior open-weight models, including its predecessor Qwen1.5, and exhibits competitive performance relative to proprietary models across diverse benchmarks on language understanding, generation, multilingual proficiency, coding, mathematics, and reasoning. The flagship model, Qwen2-72B, showcases remarkable performance: 84.2 on MMLU, 37.9 on GPQA, 64.6 on HumanEval, 89.5 on GSM8K, and 82.4 on BBH as a base language model. The instruction-tuned variant, Qwen2-72B-Instruct, attains 9.1 on MT-Bench, 48.1 on Arena-Hard, and 35.7 on LiveCodeBench. Moreover, Qwen2 demonstrates robust multilingual capabilities, proficient in approximately 30 languages, spanning English, Chinese, Spanish, French, German, Arabic, Russian, Korean, Japanese, Thai, Vietnamese, and more, underscoring its versatility and global reach. To foster community innovation and accessibility, we have made the Qwen2 model weights openly available on Hugging Face and ModelScope, and the supplementary materials including example code on GitHub. These platforms also include resources for quantization, fine-tuning, and deployment, facilitating a wide range of applications and research endeavors.
核心贡献 · Key contributions
推出 Qwen2 系列，参数范围 0.5B 至 72B，包含密集模型和 MoE 模型。 Introduces Qwen2 series with 0.5B to 72B parameters, including dense and MoE models.
Qwen2-72B 在 MMLU（84.2）、HumanEval（64.6）、GSM8K（89.5）和 BBH（82.4）上达到 SOTA。 Qwen2-72B achieves SOTA on MMLU (84.2), HumanEval (64.6), GSM8K (89.5), and BBH (82.4).
支持约 30 种语言，多语言性能强劲，与 GPT-4 和 Claude-3 竞争。 Supports ~30 languages with robust multilingual performance, competitive with GPT-4 and Claude-3.
采用 GQA、DCA、YARN 和细粒度 MoE 专家，实现高效和长达 128K token 的上下文。 Employs GQA, DCA, YARN, and fine-grained MoE experts for efficiency and long context up to 128K tokens.
后训练采用可扩展对齐（SFT 和 DPO），最小化人工标注。 Post-training with scalable alignment using SFT and DPO, minimizing human annotation.
在 Hugging Face 和 ModelScope 开源模型权重，提供量化和微调资源。 Open-source model weights on Hugging Face and ModelScope with quantization and fine-tuning resources.
局限 · Limitations
在英语理解和编码任务上与 Llama-3-70B 存在性能差距。 Performance gap with Llama-3-70B in English comprehension and coding tasks.
Qwen2-7B-Instruct 在指令遵循上落后于 Llama-3-8B-Instruct。 Qwen2-7B-Instruct lags behind Llama-3-8B-Instruct in instruction following.
MoE 模型（57B-A14B）可能需要更多预训练 token 才能匹配密集模型。 MoE model (57B-A14B) may need more pre-training tokens to match dense models.
安全评估显示仍有改进空间，尤其在色情内容检测方面。 Safety evaluation shows room for improvement, especially in pornography detection.
小模型（0.5B、1.5B）在复杂推理任务上受限，不如大模型。 Smaller models (0.5B, 1.5B) limited in complex reasoning tasks compared to larger ones.
论文章节 · Sections（共 16）
摘要Abstract
1 引言1 Introduction
2 分词器与模型2 Tokenizer & Model
2.1 分词器2.1 Tokenizer
2.2 模型架构2.2 Model Architecture
3 预训练3 Pre-training
3.1 预训练数据3.1 Pre-training Data
3.2 长上下文训练3.2 Long-context Training
4 后训练4 Post-training
4.1 后训练数据4.1 Post-training Data
4.2 监督微调4.2 Supervised Fine-tuning
4.3 基于人类反馈的强化学习4.3 Reinforcement Learning from Human Feedback