While large language models have facilitated breakthroughs in many applications of artificial intelligence, their inherent largeness makes them computationally expensive and challenging to deploy in resource-constrained settings. In this paper, we document the development of SmolLM2, a state-of-the-art "small" (1.7 billion parameter) language model (LM). To attain strong performance, we overtrain SmolLM2 on ~11 trillion tokens of data using a multi-stage training process that mixes web text with specialized math, code, and instruction-following data. We additionally introduce new specialized datasets (FineMath, Stack-Edu, and SmolTalk) at stages where we found existing datasets to be problematically small or low-quality. To inform our design decisions, we perform both small-scale ablations as well as a manual refinement process that updates the dataset mixing rates at each stage based on the performance at the previous stage. Ultimately, we demonstrate that SmolLM2 outperforms other recent small LMs including Qwen2.5-1.5B and Llama3.2-1B. To facilitate future research on LM development as well as applications of small LMs, we release both SmolLM2 as well as all of the datasets we prepared in the course of this project.
核心贡献 · Key contributions
开发了 SmolLM2，一个 1.7B 参数的小型语言模型，在同类模型中达到了最先进的性能。 Developed SmolLM2, a 1.7B parameter small language model, achieving state-of-the-art performance among similarly sized models.
引入了三个新的专用数据集：FineMath、Stack-Edu 和 SmolTalk，弥补了现有公共数据集的不足。 Introduced three new specialized datasets: FineMath, Stack-Edu, and SmolTalk, addressing limitations in existing public datasets.
设计了一个多阶段训练流程，根据性能反馈手动调整数据混合比例，针对小模型容量进行了优化。 Designed a multi-stage training process with manual rebalancing of data mixtures based on performance feedback, optimizing for small model capacity.
证明了过度训练的小模型可以获取通常与大型模型相关的能力，例如在 MMLU 上的多项选择形式。 Demonstrated that overtrained small models can acquire abilities typically associated with larger models, such as multiple-choice formulation on MMLU.
发布了 SmolLM2 模型和所有准备的数据集，以促进未来对小型语言模型的研究。 Released SmolLM2 models and all prepared datasets to facilitate future research on small language models.
通过有针对性的数据整理和退火策略，在数学和代码基准测试上取得了强劲性能。 Achieved strong performance on math and code benchmarks through targeted data curation and annealing strategies.
局限 · Limitations
在数学和代码基准测试上的性能仍落后于 Qwen2.5-1.5B 等较大模型。 Performance on math and code benchmarks still lags behind larger models like Qwen2.5-1.5B.
多阶段训练方法计算成本高，可能并非所有研究团队都能负担。 Multi-stage training approach is computationally expensive and may not be feasible for all research groups.
手动调整数据混合比例依赖经验洞察，可能无法推广到其他模型规模或领域。 Manual rebalancing of data mixtures relies on empirical insights and may not generalize to other model sizes or domains.
训练过程中损失尖峰的确切原因尚未确定，表明存在潜在的不稳定性。 The exact cause of loss spikes during training remains undetermined, indicating potential instability.
评估仅限于英语基准测试；未评估其他语言的性能。 Evaluation is limited to English-language benchmarks; performance in other languages is not assessed.