Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. For instance, while the current state-of-the-art BLEU-1 score (the higher the better) on the Pascal dataset is 25, our approach yields 59, to be compared to human performance around 69. We also show BLEU-1 score improvements on Flickr30k, from 56 to 66, and on SBU, from 19 to 28. Lastly, on the newly released COCO dataset, we achieve a BLEU-4 of 27.7, which is the current state-of-the-art.
核心贡献 · Key contributions
提出端到端神经网络（NIC），结合 CNN 和 LSTM 生成图像描述。 Proposes an end-to-end neural network (NIC) combining CNN and LSTM for image caption generation.
在 Pascal（59）、Flickr30k（66）和 SBU（28）上取得 BLEU 最优分数。 Achieves state-of-the-art BLEU scores on Pascal (59), Flickr30k (66), and SBU (28).
证明模型仅从图像描述中学习流畅语言。 Demonstrates that the model learns fluent language solely from image descriptions.
展示跨不同规模和噪声水平数据集的迁移学习能力。 Shows transfer learning capability across datasets with different sizes and noise levels.
提供定性和定量分析，包括人工评估和生成描述的多样性。 Provides qualitative and quantitative analysis including human evaluation and diversity of generated captions.
局限 · Limitations
模型性能受限于小规模训练数据集，导致过拟合。 Model performance is limited by small training dataset sizes, leading to overfitting.
BLEU 分数可能无法完全反映描述质量；人工评估得分较低。 BLEU score may not fully capture caption quality; human evaluation shows lower scores.
当源域和目标域差异显著时，迁移学习效果下降。 Transfer learning degrades when source and target domains differ significantly.
模型在最佳候选描述中常检索训练句子而非生成新句子。 Model often retrieves training sentences rather than generating novel ones for the top candidate.
弱标注数据（如 SBU）因噪声和词汇不匹配降低性能。 Weakly labeled data (e.g., SBU) reduces performance due to noise and vocabulary mismatch.