We introduce Mistral 7B v0.1, a 7-billion-parameter language model engineered for superior performance and efficiency. Mistral 7B outperforms Llama 2 13B across all evaluated benchmarks, and Llama 1 34B in reasoning, mathematics, and code generation. Our model leverages grouped-query attention (GQA) for faster inference, coupled with sliding window attention (SWA) to effectively handle sequences of arbitrary length with a reduced inference cost. We also provide a model fine-tuned to follow instructions, Mistral 7B -- Instruct, that surpasses the Llama 2 13B -- Chat model both on human and automated benchmarks. Our models are released under the Apache 2.0 license.
核心贡献 · Key contributions
提出 Mistral 7B，一个 70 亿参数模型，在所有基准测试中超越 Llama 2 13B，并在推理、数学和代码方面超越 Llama 1 34B。 Introduces Mistral 7B, a 7B-parameter model outperforming Llama 2 13B on all benchmarks and Llama 1 34B on reasoning, math, and code.
利用分组查询注意力（GQA）加速推理，滑动窗口注意力（SWA）高效处理长序列。 Leverages grouped-query attention (GQA) for faster inference and sliding window attention (SWA) for efficient long sequence handling.
证明较小模型能达到与更大模型相当的性能，知识压缩程度超出先前认知。 Demonstrates that a smaller model can achieve performance comparable to much larger models, compressing knowledge more than previously thought.
提供 Mistral 7B – Instruct 微调聊天模型，在人工和自动基准上超越 Llama 2 13B – Chat。 Provides Mistral 7B – Instruct, a fine-tuned chat model surpassing Llama 2 13B – Chat on human and automated benchmarks.
引入滚动缓冲区缓存和预填充分块，减少内存使用并提升推理效率。 Introduces rolling buffer cache and pre-fill chunking to reduce memory usage and improve inference efficiency.
以 Apache 2.0 许可证发布模型，附带参考实现便于部署。 Releases models under Apache 2.0 license with reference implementation for easy deployment.
局限 · Limitations
知识基准性能压缩率较低（1.9 倍），受限于参数数量。 Performance on knowledge benchmarks shows lower compression rate (1.9x) due to limited parameter count.
评估协议差异可能影响与 Llama 2 在某些基准上的直接比较。 Evaluation protocol differences may affect direct comparison with Llama 2 results on some benchmarks.
安全评估仅使用 175 个不安全提示，实际鲁棒性可能不同。 Safety evaluation uses a limited set of 175 unsafe prompts; real-world robustness may vary.
滑动窗口注意力理论跨度 131K 词元在实践中可能无法充分利用。 Sliding window attention theoretical span of 131K tokens may not be fully utilized in practice.
微调指令模型仅为初步演示，未使用专有数据或训练技巧。 Fine-tuned instruct model is a preliminary demonstration; no proprietary data or training tricks used.
论文章节 · Sections（共 10）
摘要Abstract
1 引言1 Introduction
2 架构细节2 Architectural details
3 结果3 Results
4 指令微调4 Instruction Finetuning
5 为前端应用添加护栏5 Adding guardrails for front-facing applications
5.1 系统提示强制执行护栏5.1 System prompt to enforce guardrails
5.2 通过自我反思进行内容审核5.2 Content moderation with self-reflection