Information overload is a major obstacle to scientific progress. The explosive growth in scientific literature and data has made it ever harder to discover useful insights in a large mass of information. Today scientific knowledge is accessed through search engines, but they are unable to organize scientific knowledge alone. In this paper we introduce Galactica: a large language model that can store, combine and reason about scientific knowledge. We train on a large scientific corpus of papers, reference material, knowledge bases and many other sources. We outperform existing models on a range of scientific tasks. On technical knowledge probes such as LaTeX equations, Galactica outperforms the latest GPT-3 by 68.2% versus 49.0%. Galactica also performs well on reasoning, outperforming Chinchilla on mathematical MMLU by 41.3% to 35.7%, and PaLM 540B on MATH with a score of 20.4% versus 8.8%. It also sets a new state-of-the-art on downstream tasks such as PubMedQA and MedMCQA dev of 77.6% and 52.9%. And despite not being trained on a general corpus, Galactica outperforms BLOOM and OPT-175B on BIG-bench. We believe these results demonstrate the potential for language models as a new interface for science. We open source the model for the benefit of the scientific community.
核心贡献 · Key contributions
提出 Galactica，一个在 1060 亿 token 的精选科学语料上训练的大型语言模型。 Introduces Galactica, a large language model trained on a curated scientific corpus of 106 billion tokens.
在 LaTeX 方程知识探测上超越 GPT-3（68.2% 对 49.0%），在数学 MMLU 上超越 Chinchilla（41.3% 对 35.7%）。 Outperforms GPT-3 on LaTeX equation knowledge probes (68.2% vs 49.0%) and Chinchilla on mathematical MMLU (41.3% vs 35.7%).
在 PubMedQA（77.6%）和 MedMCQA dev（52.9%）上达到最先进水平，无需微调。 Achieves state-of-the-art on PubMedQA (77.6%) and MedMCQA dev (52.9%) without fine-tuning.
证明在精选数据上重复 token 可同时提升上游和下游性能。 Demonstrates that repeated tokens improve both upstream and downstream performance on curated data.
引入工作记忆 token（<work>）用于逐步推理，在 MATH 上超越更大模型。 Introduces a working memory token (<work>) for step-by-step reasoning, outperforming larger models on MATH.
展示多模态任务能力，包括 SMILES 到 IUPAC 命名和蛋白质功能注释。 Shows capability in multi-modal tasks including SMILES-to-IUPAC naming and protein function annotation.
局限 · Limitations
语料限于开放获取资源；封闭知识可能提升性能。 Corpus limited to open-access resources; closed-access knowledge could improve performance.
对分子（200 万）和蛋白质（50 万）的自设限制限制了模态性能。 Self-imposed constraints on molecules (2 million) and proteins (0.5 million) limit modality performance.
提示预训练与核心语料的效果未分离；通用能力可能依赖提示。 Effects of prompt pre-training vs core corpus not disentangled; general capabilities may rely on prompts.
在 120B 规模下，引文预测仍偏向热门论文。 Bias towards popular papers remains in citation prediction at 120B scale.
缺乏几何概念（如 3D 坐标）限制了化学和生物学理解。 No notion of geometry (e.g., 3D coordinates) limits chemical and biological understanding.
论文章节 · Sections（共 25）
摘要Abstract
1 引言1 Introduction
1.1 我们的贡献1.1 Our Contribution
2 相关工作2 Related Work
3 数据集3 Dataset
3.1 分词3.1 Tokenization
3.2 提示预训练3.2 Prompt Pre-Training
4.1 架构4.1 Architecture
4.2 模型4.2 Models
4.3 库与基础设施4.3 Libraries and Infrastructure
5 结果5 Results
5.1 重复标记无害5.1 Repeated Tokens Considered Not Harmful