We present Llemma, a large language model for mathematics. We continue pretraining Code Llama on the Proof-Pile-2, a mixture of scientific papers, web data containing mathematics, and mathematical code, yielding Llemma. On the MATH benchmark Llemma outperforms all known open base models, as well as the unreleased Minerva model suite on an equi-parameter basis. Moreover, Llemma is capable of tool use and formal theorem proving without any further finetuning. We openly release all artifacts, including 7 billion and 34 billion parameter models, the Proof-Pile-2, and code to replicate our experiments.
核心贡献 · Key contributions
训练并发布了 Llemma 7B 和 34B 模型，在 MATH 上达到开源基础模型新最优。 Trained and released Llemma 7B and 34B models, new state-of-the-art open base models on MATH.
发布了 Proof-Pile-2，一个 550 亿词元的科学论文、数学网页数据和代码数据集。 Released Proof-Pile-2, a 55B-token dataset of scientific papers, math web data, and code.
展示了无需微调的少样本工具使用（Python）和形式定理证明。 Demonstrated few-shot tool use (Python) and formal theorem proving without finetuning.
开源了所有工件：模型、数据集和可复现代码。 Open-sourced all artifacts: models, dataset, and code for reproducibility.
提供了数据混合、记忆化和监督微调的分析。 Provided analysis of data mixture, memorization, and supervised finetuning.
局限 · Limitations
Llemma 在 MATH 上的表现可能部分受训练数据记忆化的影响。 Llemma's performance on MATH may be partially influenced by memorization of training data.
34B 模型训练因算力限制仅进行了 500 亿词元。 The 34B model training was limited to 50B tokens due to compute constraints.
形式定理证明评估仅限于无微调的少样本设置。 Formal theorem proving evaluation is limited to few-shot settings without finetuning.
该研究未探索数学领域的缩放定律或最优数据组成。 The study does not explore scaling laws or optimal data composition for math domain.