Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→我们推出了 phi-3-mini，一个拥有 38 亿参数的语言模型，在 3.3 万亿个 token 上训练而成。根据学术基准和内部测试，其整体性能可与 Mixtral 8x7B 和 GPT-3.5 等模型相媲美（例如，phi-3-mini 在 MMLU 上达到 69%，在 MT-bench 上达到 8.38），尽管它小到可以部署在手机上。我们的训练数据集是 phi-2 所用数据集的放大版本，由经过严格筛选的公开网络数据和合成数据组成。该模型还进一步针对鲁棒性、安全性和聊天格式进行了对齐。我们还提供了参数缩放结果，包括在 4.8 万亿 token 上训练的 7B 和 14B 模型，分别称为 phi-3-small 和 phi-3-medium，两者都比 phi-3-mini 强大得多（例如，在 MMLU 上分别达到 75% 和 78%，在 MT-bench 上分别达到 8.7 和 8.9）。为了增强多语言、多模态和长上下文能力，我们推出了 phi-3.5 系列中的三个模型：phi-3.5-mini、phi-3.5-MoE 和 phi-3.5-Vision。phi-3.5-MoE 是一个 16x3.8B 的 MoE 模型，拥有 66 亿活跃参数，在语言推理、数学和代码任务上优于其他类似规模的开源模型，如 Llama 3.1 和 Mixtral 系列，并与 Gemini-1.5-Flash 和 GPT-4o-mini 相当。同时，phi-3.5-Vision 是一个源自 phi-3.5-mini 的 42 亿参数模型，在推理任务中表现出色，擅长处理单图像和文本提示，以及多图像和文本提示。
We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. Our training dataset is a scaled-up version of the one used for phi-2, composed of heavily filtered publicly available web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide parameter-scaling results with a 7B, 14B models trained for 4.8T tokens, called phi-3-small, phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75%, 78% on MMLU, and 8.7, 8.9 on MT-bench). To enhance multilingual, multimodal, and long-context capabilities, we introduce three models in the phi-3.5 series: phi-3.5-mini, phi-3.5-MoE, and phi-3.5-Vision. The phi-3.5-MoE, a 16 x 3.8B MoE model with 6.6 billion active parameters, achieves superior performance in language reasoning, math, and code tasks compared to other open-source models of similar scale, such as Llama 3.1 and the Mixtral series, and on par with Gemini-1.5-Flash and GPT-4o-mini. Meanwhile, phi-3.5-Vision, a 4.2 billion parameter model derived from phi-3.5-mini, excels in reasoning tasks and is adept at handling both single-image and text prompts, as well as multi-image and text prompts.