Language models (LMs) have become ubiquitous in both NLP research and in commercial product offerings. As their commercial importance has surged, the most powerful models have become closed off, gated behind proprietary interfaces, with important details of their training data, architectures, and development undisclosed. Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs. To this end, we have built OLMo, a competitive, truly Open Language Model, to enable the scientific study of language models. Unlike most prior efforts that have only released model weights and inference code, we release OLMo alongside open training data and training and evaluation code. We hope this release will empower the open research community and inspire a new wave of innovation.
核心贡献 · Key contributions
发布了 OLMo，一个真正开放的语言模型，包含完整的训练数据、代码和检查点。 Releases OLMo, a truly open language model with full training data, code, and checkpoints.
提供了 Dolma，一个来自多种来源的 3T 词元开放预训练数据集。 Provides Dolma, a 3T-token open pretraining dataset from diverse sources.
开源了 Catwalk 和 Paloma 评估框架，用于下游任务和困惑度评估。 Open-sources evaluation frameworks Catwalk and Paloma for downstream and perplexity evaluation.
展示了 OLMo-7B 在核心任务上与其他开放模型相比具有竞争力的性能。 Demonstrates competitive performance of OLMo-7B against other open models on core tasks.
发布了中间检查点和训练日志，以支持语言模型的科学研究。 Releases intermediate checkpoints and training logs to enable scientific study of language models.
通过指令微调和 DPO 提供了适配模型，展示了改进的安全性和性能。 Provides adapted models via instruction tuning and DPO, showing improved safety and performance.
局限 · Limitations
OLMo-7B 在整体能力上仍落后于 GPT-4 等专有模型。 OLMo-7B still lags behind proprietary models like GPT-4 in overall capabilities.
研究集中在 1B 和 7B 规模；65B 等更大模型尚未发布。 The study focuses on 1B and 7B scales; larger models like 65B are not yet released.
碳排放估计为下限，不包括硬件制造和其他生命周期影响。 Carbon emission estimates are lower bounds, excluding hardware manufacturing and other lifecycle impacts.
适配模型与 Tülu 2 相比存在性能差距，可能源于数据污染或混合设计。 Adapted models show a performance gap compared to Tülu 2, possibly due to data contamination or mix design.
研究未涉及多模态能力或超过 2048 词元的长上下文场景。 The study does not address multimodal capabilities or long-context scenarios beyond 2048 tokens.
论文章节 · Sections（共 22）
摘要Abstract
1 引言1 Introduction
2 OLMo 框架2 OLMo Framework
2.1 OLMo 模型与架构2.1 OLMo Model and Architecture
2.2 预训练数据：Dolma2.2 Pretraining Data: Dolma
2.3 适配2.3 Adaptation
2.4 评估2.4 Evaluation
3 训练 OLMo3 Training OLMo
3.1 分布式训练框架3.1 Distributed Training Framework
3.2 优化器3.2 Optimizer
3.3 数据3.3 Data
3.4 硬件3.4 Hardware
4 结果4 Results
4.1 下游评估4.1 Downstream evaluation
4.2 内在语言建模评估4.2 Intrinsic language modeling evaluation
4.3 适配评估4.3 Adaptation Evaluation
4.4 功耗与碳足迹4.4 Power Consumption and Carbon Footprint