We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions significantly outperform retrieval baselines on both full images and on a new dataset of region-level annotations.
核心贡献 · Key contributions
提出一种深度神经网络模型，通过多模态嵌入推断句子片段与图像区域之间的潜在对齐。 Proposes a deep neural network model that infers latent alignments between sentence segments and image regions via a multimodal embedding.
引入一种多模态循环神经网络架构，利用推断的对齐生成图像区域的新颖描述。 Introduces a Multimodal Recurrent Neural Network architecture that generates novel descriptions of image regions from inferred alignments.
在 Flickr8K、Flickr30K 和 MSCOCO 数据集上的图像-句子检索实验中取得最先进结果。 Achieves state-of-the-art results on image-sentence retrieval experiments on Flickr8K, Flickr30K, and MSCOCO datasets.
证明生成的描述在完整图像和区域级标注上均显著优于检索基线。 Demonstrates that generated descriptions significantly outperform retrieval baselines on both full images and region-level annotations.
相比先前工作简化了排序目标，无需额外超参数即可提升性能。 Simplifies the ranking objective compared to prior work, improving performance without additional hyperparameters.
使用双向循环神经网络计算词表示，无需依赖解析即可捕获无界上下文。 Uses a Bidirectional Recurrent Neural Network to compute word representations, capturing unbounded context without dependency parsing.
局限 · Limitations
多模态 RNN 只能为固定分辨率的单个输入区域生成描述，缺乏多扫视探索。 The Multimodal RNN can only generate descriptions for a single input region at fixed resolution, lacking multi-saccade exploration.
图像信息仅通过加性偏置交互传递，其表达能力弱于乘性交互。 Image information is communicated only through additive bias interactions, which are less expressive than multiplicative interactions.
该方法由两个独立模型组成；从图像-句子对到区域描述的端到端训练仍是开放问题。 The approach consists of two separate models; end-to-end training from image-sentence pairs to region descriptions remains an open problem.
模型可能生成训练数据中已有的句子（束搜索大小为 7 时占 60%），限制了新颖性。 The model may generate sentences found in training data (60% with beam size 7), limiting novelty.
性能对束搜索大小以及稀有词与常见词之间的词频差异敏感。 Performance is sensitive to beam size and word frequency disparity between rare and common words.
论文章节 · Sections（共 13）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Work
3 我们的模型3 Our Model
3.1 学习对齐视觉与语言数据3.1 Learning to align visual and language data
3.2 用于生成描述的多模态循环神经网络3.2 Multimodal Recurrent Neural Network for generating descriptions