In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.
核心贡献 · Key contributions
提出 Gemini 1.5 系列模型（Pro 和 Flash），采用 MoE 架构，在长达 1000 万词元的上下文中实现近完美的召回能力。 Introduces Gemini 1.5 models (Pro & Flash) with MoE architecture, achieving near-perfect recall on long-context retrieval up to 10M tokens.
在长文档问答、长视频问答和长上下文语音识别上达到新 SOTA，超越 Claude 2.1 和 GPT-4 Turbo 等先前模型。 Sets new state-of-the-art on long-document QA, long-video QA, and long-context ASR, outperforming prior models like Claude 2.1 and GPT-4 Turbo.
展示了仅凭一本语法手册进行上下文语言学习的能力，在仅 25 万词元的上下文中达到人类水平的 Kalamang 语翻译。 Demonstrates in-context language learning from a single grammar manual for Kalamang, matching human-level translation with only 250k tokens of context.
在不损失核心能力（数学、科学、代码）的前提下，以更少算力匹配或超越 Gemini 1.0 Ultra 的性能。 Achieves strong core capabilities (math, science, code) without compromise, matching or surpassing Gemini 1.0 Ultra despite using less compute.
在实际应用中实现 26%至 75%的时间节省，覆盖 10 种职业类别的人机协作任务。 Shows real-world productivity gains: 26–75% time savings across 10 job categories in professional collaboration tasks.
局限 · Limitations
长上下文检索在多针或相似干扰项下性能下降，在多针检索和共指消解任务中准确率降低。 Long-context retrieval degrades with multiple needles or similar distractors; performance drops in multi-needle and co-reference resolution tasks.
语言翻译中规则应用不一致，例如尽管语法明确说明，仍错误应用 Kalamang 语的音变规则。 Inconsistent rule application in language translation, e.g., misapplying phonological alternations in Kalamang despite explicit grammar rules.
在强依赖 OCR 的图像任务上表现不佳；假阴性可能低估真实能力，提示评估方法需改进。 Underperforms on OCR-heavy image tasks; false negatives may underestimate true capability, suggesting evaluation improvements needed.
长上下文能力可能放大已有安全风险（如长输入中的有害内容），需要新的缓解策略。 Long-context capability may amplify existing safety risks (e.g., harmful content in long inputs), requiring new mitigation strategies.
现有评估基准可能存在数据污染或不足以衡量超长上下文中的高级推理能力。 Evaluation benchmarks may be contaminated or insufficient for measuring advanced reasoning over extremely long contexts.
论文章节 · Sections（共 64）
Gemini 1.5：解锁跨百万级上下文令牌的多模态理解Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
1 引言1 Introduction
2 模型架构2 Model Architecture
训练基础设施与数据集3 Training Infrastructure and Dataset
4 长上下文评估4 Long-context Evaluation
4.1 多模态长上下文能力的定性示例4.1 Qualitative Examples of Multimodal Long-Context Capabilities