Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering. Cognition is core to tasks that involve not just recognizing, but reasoning about our visual world. However, models used to tackle the rich content in images for cognitive tasks are still being trained using the same datasets designed for perceptual tasks. To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in an image. When asked "What vehicle is the person riding?", computers will need to identify the objects in an image as well as the relationships riding(man, carriage) and pulling(horse, carriage) in order to answer correctly that "the person is riding a horse-drawn carriage". In this paper, we present the Visual Genome dataset to enable the modeling of such relationships. We collect dense annotations of objects, attributes, and relationships within each image to learn these models. Specifically, our dataset contains over 100K images where each image has an average of 21 objects, 18 attributes, and 18 pairwise relationships between objects. We canonicalize the objects, attributes, relationships, and noun phrases in region descriptions and questions answer pairs to WordNet synsets. Together, these annotations represent the densest and largest dataset of image descriptions, objects, attributes, relationships, and question answers.
核心贡献 · Key contributions
提出 Visual Genome 数据集，包含 10.8 万张图像，每张图像平均标注 21 个物体、18 个属性和 18 个关系。 Introduces Visual Genome, a dataset with 108K images, each annotated with 21 objects, 18 attributes, and 18 relationships on average.
提供密集的区域描述（每张图像 42 条）和 170 万个问答对，支持推理等认知任务。 Provides dense region descriptions (42 per image) and 1.7M question-answer pairs, enabling cognitive tasks like reasoning.
将物体、属性和关系规范化到 WordNet 同义词集，实现跨图像的一致查询。 Canonicalizes objects, attributes, and relationships to WordNet synsets, enabling consistent cross-image querying.
构建场景图，将物体、属性和关系组合成结构化图，形式化表示图像。 Constructs scene graphs that formally represent images, combining objects, attributes, and relationships into a structured graph.
通过众包收集数据，使用 BLEU 分数确保多样性，并通过多数投票进行验证。 Collects data via crowdsourcing with diversity enforcement using BLEU scores and verification via majority voting.
相比 MS-COCO 和 ImageNet 等现有数据集，实现了更高的物体密度和语义多样性。 Achieves higher object density and semantic diversity than existing datasets like MS-COCO and ImageNet.
局限 · Limitations
数据集偏向人物和运动图像，限制了向其他领域的泛化能力。 Dataset is biased toward images of people and sports, limiting generalizability to other domains.
尽管经过验证，众包标注仍可能包含错误，影响可靠性。 Crowdsourced annotations may contain errors despite verification, affecting reliability.
WordNet 规范化可能对多义词引入歧义，需要手动规则修正。 WordNet canonicalization may introduce ambiguity for polysemous words, requiring manual rules.
场景图构建依赖启发式重叠阈值，可能合并不同物体。 Scene graph construction relies on heuristic overlap thresholds, potentially merging distinct objects.
区域描述和问答仅限于英语，限制了多语言适用性。 Region descriptions and QAs are limited to English, restricting multilingual applicability.
论文章节 · Sections（共 43）
摘要Abstract
1 引言1 Introduction
2 视觉基因组数据表示2 Visual Genome Data Representation
2.1 多个区域及其描述2.1 Multiple regions and their descriptions
2.2 多个对象及其边界框2.2 Multiple objects and their bounding boxes
2.3 一组属性2.3 A set of attributes
2.4 一组关系2.4 A set of relationships
2.5 一组区域图2.5 A set of region graphs
2.6 一个场景图2.6 One scene graph
2.7 一组问答对2.7 A set of question answer pairs
3 相关工作3 Related Work
3.1 数据集3.1 Datasets
3.2 图像描述3.2 Image Descriptions
3.3 对象3.3 Objects
3.4 属性3.4 Attributes
3.5 关系3.5 Relationships
3.6 问答3.6 Question Answering
3.7 知识表示3.7 Knowledge Representation
4 众包策略4 Crowdsourcing Strategies
4.1 众包工作者4.1 Crowd Workers
4.2 区域描述4.2 Region Descriptions
4.3 对象4.3 Objects
4.4 属性、关系和区域图4.4 Attributes, Relationships, and Region Graphs