Large language models (LLMs) have revolutionized the field of artificial intelligence, enabling natural language processing tasks that were previously thought to be exclusive to humans. In this work, we introduce Qwen, the first installment of our large language model series. Qwen is a comprehensive language model series that encompasses distinct models with varying parameter counts. It includes Qwen, the base pretrained language models, and Qwen-Chat, the chat models finetuned with human alignment techniques. The base language models consistently demonstrate superior performance across a multitude of downstream tasks, and the chat models, particularly those trained using Reinforcement Learning from Human Feedback (RLHF), are highly competitive. The chat models possess advanced tool-use and planning capabilities for creating agent applications, showcasing impressive performance even when compared to bigger models on complex tasks like utilizing a code interpreter. Furthermore, we have developed coding-specialized models, Code-Qwen and Code-Qwen-Chat, as well as mathematics-focused models, Math-Qwen-Chat, which are built upon base language models. These models demonstrate significantly improved performance in comparison with open-source models, and slightly fall behind the proprietary models.
核心贡献 · Key contributions
Qwen 基础模型使用高达 3 万亿 tokens 预训练，在多项基准上超越更大规模模型。 Qwen base models outperform larger counterparts on multiple benchmarks, using up to 3 trillion tokens for pretraining.
采用 RLHF 训练的 Qwen-Chat 模型在人类评估中表现具竞争力，紧追 GPT-4。 Qwen-Chat models trained with RLHF achieve competitive performance, closely trailing GPT-4 in human evaluation.
Code-Qwen 和 Math-Qwen-Chat 专用模型大幅优于开源同类，在代码和数学任务上接近 GPT-3.5。 Code-Qwen and Math-Qwen-Chat specialized models significantly outperform open-source peers, approaching GPT-3.5 on code and math tasks.
Qwen 系列支持多语言、高效分词（152K 词表），并通过动态 NTK 感知插值扩展上下文长度。 Qwen series includes multilingual support, efficient tokenization (152K vocab), and context length extension via dynamic NTK-aware interpolation.
局限 · Limitations
尽管基准表现强劲，Qwen 在复杂推理和创造性任务上仍落后于 GPT-4 等专有模型。 Qwen still lags behind proprietary models like GPT-4 on complex reasoning and creative tasks despite strong benchmarks.
RLHF 对齐的有效性存在争议；可能带来多样性降低或过拟合等缺陷。 The effectiveness of RLHF for alignment is debated; potential drawbacks like reduced diversity or overfitting remain.
上下文长度扩展技术（NTK、LogN）无需训练，但超过 8K tokens 时可能性能下降。 Context length extension techniques (NTK, LogN) are training-free but may degrade performance on longer sequences beyond 8K tokens.
专用模型（Code-Qwen、Math-Qwen）因领域预训练可能损失通用能力。 Specialized models (Code-Qwen, Math-Qwen) may lose general-purpose capability due to domain-specific pretraining.
基准评估可能无法完全反映真实场景下的对齐与安全性；人类评估仅限于中文。 Benchmark evaluations may not fully capture real-world alignment and safety; human evaluation limited to Chinese only.
论文章节 · Sections（共 4）
摘要Abstract
3 对齐3 Alignment
1 尽快适应大学生活1 Adjusting to university life as soon as possible.
5 为未来打下坚实基础5 Building a solid foundation for the future.