Gemini：一个高度能干的多模态模型家族

Gemini: A Family of Highly Capable Multimodal Models

诺姆·沙泽尔 Noam Shazeer · · 2023-12-19 · arXiv:2312.11805 ↗

摘要 · Abstract

本报告介绍了一个新的多模态模型家族 Gemini，它在图像、音频、视频和文本理解方面表现出非凡的能力。Gemini 家族包括 Ultra、Pro 和 Nano 三种尺寸，适用于从复杂推理任务到内存受限的端侧用例等多种应用。在广泛的基准测试评估中，我们最强大的 Gemini Ultra 模型在 32 项基准测试中的 30 项上取得了最优性能——值得注意的是，它是首个在广受研究的考试基准 MMLU 上达到人类专家水平的模型，并且在我们考察的 20 个多模态基准测试中每一项都提升了最优水平。我们相信，Gemini 家族在跨模态推理和语言理解方面的新能力将能够支持多种多样的用例。我们讨论了如何通过包括 Gemini、Gemini Advanced、Google AI Studio 和 Cloud Vertex AI 在内的服务，负责任地对 Gemini 模型进行后训练和部署，并交付给用户。

This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultra model advances the state of the art in 30 of 32 of these benchmarks - notably being the first model to achieve human-expert performance on the well-studied exam benchmark MMLU, and improving the state of the art in every one of the 20 multimodal benchmarks we examined. We believe that the new capabilities of the Gemini family in cross-modal reasoning and language understanding will enable a wide variety of use cases. We discuss our approach toward post-training and deploying Gemini models responsibly to users through services including Gemini, Gemini Advanced, Google AI Studio, and Cloud Vertex AI.

核心贡献 · Key contributions

提出 Gemini 模型系列（Ultra、Pro、Nano），在 30/32 个基准测试中达到最先进水平，涵盖文本、图像、音频和视频。
Introduces Gemini family (Ultra, Pro, Nano) with SOTA on 30/32 benchmarks across text, image, audio, video.
首个在 MMLU（90.04%）上超越人类专家表现的模型，这是一个全面的考试基准。
First model to surpass human-expert performance on MMLU (90.04%), a comprehensive exam benchmark.
展示了强大的跨模态推理能力，例如理解潦草笔迹并解决物理问题。
Demonstrates strong cross-modal reasoning, e.g., understanding messy handwriting and solving physics problems.
通过 Gemini Nano 实现高效的设备端部署，在摘要和阅读理解方面表现优异。
Achieves efficient on-device deployment with Gemini Nano, excelling in summarization and reading comprehension.
提出了全面的后训练策略（SFT、RLHF）以实现对齐和安全，部署在 Gemini 和 Gemini Advanced 中。
Presents comprehensive post-training strategy (SFT, RLHF) for alignment and safety, deployed in Gemini and Gemini Advanced.
推进多模态能力：视频理解、音频 ASR、图像生成和多语言翻译。
Advances multimodal capabilities: video understanding, audio ASR, image generation, and multilingual translation.

局限 · Limitations

基准测试结果可能受数据污染影响；HellaSwag 结果需要去污染处理。
Benchmark results may be affected by data contamination; HellaSwag results required decontamination.
Gemini Ultra 在音频任务上的评估有限；最大模型的音频性能尚未报告。
Limited evaluation of Gemini Ultra on audio tasks; audio performance not yet reported for the largest model.
依赖大规模计算基础设施（TPUv4 SuperPod）和复杂的分布式训练，限制了可重复性。
Relies on massive compute infrastructure (TPUv4 superpods) and sophisticated distributed training, limiting reproducibility.
安全缓解依赖于后训练方法；尽管努力，仍可能产生有害输出。
Safety mitigation relies on post-training methods; potential for harmful outputs remains despite efforts.
上下文窗口限制为 32K 个词元，可能限制长文档或长视频理解。
Context window limited to 32K tokens, which may restrict long-document or long-video understanding.

论文章节 · Sections（共 31）

摘要 Abstract
1 引言 1 Introduction
2 模型架构 2 Model Architecture
训练基础设施 3 Training Infrastructure
4 预训练数据集 4 Pre-Training Dataset
5 评估 5 Evaluation
5.1 文本 5.1 Text
在线演示 3 See demos on website https://deepmind.google/gemini.
5.2 多模态 5.2 Multimodal
6 后训练模型 6 Post-Training Models
6.1 Gemini 应用：Gemini 与 Gemini Advanced 6.1 Gemini Apps: Gemini and Gemini Advanced
6.2 Gemini API：Google AI Studio 和 Cloud Vertex AI 6.2 Gemini APIs: Google AI Studio and Cloud Vertex AI
6.3 后训练方法与数据 6.3 Post-Training Methods & Data
6.4 评估 6.4 Evaluation
6.5 模型能力 6.5 Model Capabilities
7 负责任部署 7 Responsible Deployment
7.1 影响评估 7.1 Impact Assessment
7.2 安全策略 7.2 Safety Policies
7.3 缓解措施 7.3 Mitigations
7.4 安全评估 7.4 Safety Evaluations
7.5 部署 7.5 Deployment
8 讨论与结论 8 Discussion and Conclusion
9 贡献与致谢 9 Contributions and Acknowledgments
致谢 8 Lead, Gemini App Core Modeling, Eval, Data
10.1 Gemini Ultra 模型卡 10.1 Gemini Ultra Model Card
10.2 MMLU 基准上的思维链比较 10.2 Chain-of-Thought Comparisons on MMLU benchmark
10.3 能力与基准测试任务 10.3 Capabilities and Benchmarking Tasks
10.4 定性示例 10.4 Qualitative Examples
10.5 多语言环境下的常识推理 10.5 Commonsense reasoning in a multilingual setting
2 视觉上应与谷歌搜索相似 2 It should be visually similar to Google search,
10.6 多步推理与数学 10.6 Multi-step reasoning and mathematics

阅读逐段中英对照全文 →