Yi: Open Foundation Models by 01.AI
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→我们介绍了 Yi 模型系列，这是一系列语言和多模态模型，展现出强大的多维能力。Yi 模型系列基于 6B 和 34B 的预训练语言模型，然后我们将其扩展为聊天模型、200K 长上下文模型、深度上采样模型和视觉语言模型。我们的基础模型在 MMLU 等广泛基准测试中取得了强劲性能，而微调后的聊天模型在 AlpacaEval 和 Chatbot Arena 等主要评估平台上获得了较高的人类偏好率。基于我们可扩展的超级计算基础设施和经典的 Transformer 架构，我们将 Yi 模型的性能主要归因于数据工程努力所带来的数据质量。在预训练方面，我们使用级联数据去重和质量过滤管道构建了 3.1 万亿个 token 的英文和中文语料库。在微调方面，我们通过多次迭代优化了一个小规模（少于 10K）的指令数据集，确保每个实例都经过机器学习工程师的直接验证。对于视觉语言，我们将聊天语言模型与视觉 Transformer 编码器结合，训练模型将视觉表示与语言模型的语义空间对齐。我们通过轻量级持续预训练将上下文长度进一步扩展到 200K，并展示了强大的大海捞针检索性能。我们表明，通过持续预训练增加预训练检查点的深度可以进一步提高性能。我们相信，基于当前的结果，使用经过充分优化的数据继续扩大模型参数将带来更强大的前沿模型。
We introduce the Yi model family, a series of language and multimodal models that demonstrate strong multi-dimensional capabilities. The Yi model family is based on 6B and 34B pretrained language models, then we extend them to chat models, 200K long context models, depth-upscaled models, and vision-language models. Our base models achieve strong performance on a wide range of benchmarks like MMLU, and our finetuned chat models deliver strong human preference rate on major evaluation platforms like AlpacaEval and Chatbot Arena. Building upon our scalable super-computing infrastructure and the classical transformer architecture, we attribute the performance of Yi models primarily to its data quality resulting from our data-engineering efforts. For pretraining, we construct 3.1 trillion tokens of English and Chinese corpora using a cascaded data deduplication and quality filtering pipeline. For finetuning, we polish a small scale (less than 10K) instruction dataset over multiple iterations such that every single instance has been verified directly by our machine learning engineers. For vision-language, we combine the chat language model with a vision transformer encoder and train the model to align visual representations to the semantic space of the language model. We further extend the context length to 200K through lightweight continual pretraining and demonstrate strong needle-in-a-haystack retrieval performance. We show that extending the depth of the pretrained checkpoint through continual pretraining further improves performance. We believe that given our current results, continuing to scale up model parameters using thoroughly optimized data will lead to even stronger frontier models.