Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily interleaved visual and textual data, and (iii) seamlessly ingest images or videos as inputs. Thanks to their flexibility, Flamingo models can be trained on large-scale multimodal web corpora containing arbitrarily interleaved text and images, which is key to endow them with in-context few-shot learning capabilities. We perform a thorough evaluation of our models, exploring and measuring their ability to rapidly adapt to a variety of image and video tasks. These include open-ended tasks such as visual question-answering, where the model is prompted with a question which it has to answer; captioning tasks, which evaluate the ability to describe a scene or an event; and close-ended tasks such as multiple-choice visual question-answering. For tasks lying anywhere on this spectrum, a single Flamingo model can achieve a new state of the art with few-shot learning, simply by prompting the model with task-specific examples. On numerous benchmarks, Flamingo outperforms models fine-tuned on thousands of times more task-specific data.
核心贡献 · Key contributions
提出 Flamingo 系列视觉语言模型，用于多模态任务的少样本学习。 Introduces Flamingo, a family of Visual Language Models for few-shot learning on multimodal tasks.
提出架构创新：桥接冻结的视觉和语言模型、处理交错数据、使用感知器重采样器。 Proposes architectural innovations: bridging frozen vision and language models, handling interleaved data, and using a Perceiver Resampler.
在 16 个多模态基准上取得少样本学习最优性能，常超越微调模型。 Achieves state-of-the-art few-shot performance on 16 multimodal benchmarks, often surpassing fine-tuned models.
展示了在开放任务（如描述和视觉问答）上的有效上下文学习。 Demonstrates effective in-context learning for open-ended tasks like captioning and visual question-answering.
表明在交错网络数据上训练对少样本能力至关重要。 Shows that training on interleaved web data is crucial for few-shot capabilities.
扩展模型规模和样本数量以提升性能，类似 GPT-3。 Scales model size and number of shots to improve performance, similar to GPT-3.
局限 · Limitations
继承预训练语言模型的弱点，包括幻觉和对长序列泛化差。 Inherits weaknesses from pretrained LMs, including hallucinations and poor generalization to long sequences.
分类性能落后于对比模型（如 CLIP）。 Classification performance lags behind contrastive models like CLIP.
上下文学习对示例选择敏感，且随样本数量增多扩展性差。 In-context learning is sensitive to demonstration selection and scales poorly with many shots.
训练需要大规模网络数据，可能包含偏见和有害内容。 Training requires large-scale web data, which may contain biases and toxic content.
冻结语言模型可防止灾难性遗忘，但限制了适应性。 Freezing the language model prevents catastrophic forgetting but limits adaptation.
论文章节 · Sections（共 14）
摘要Abstract
1 引言1 Introduction
2 方法2 Approach
2.1 视觉处理与感知器重采样器2.1 Visual processing and the Perceiver Resampler
2.2 基于视觉表示的条件化冻结语言模型2.2 Conditioning frozen language models on visual representations