TL;DR
This paper tackles the issue of limited and biased vocabularies in existing scene graph generation methods. They propose Entity Synset Alignment (ESA), a novel approach to integrate multiple scene graphs from different datasets into a more general and comprehensive representation by aligning entity labels using WordNet and IoU. This integrated scene graph shows improved performance on downstream tasks like image-caption retrieval.
What Problem Does This Paper Address?
Scene graphs are powerful tools for representing visual scenes by capturing objects, their attributes, and relationships. However, current methods for generating scene graphs often suffer from a limited vocabulary, which is dictated by the specific dataset and the hypotheses of the researchers. This leads to biased information and makes the generated scene graphs difficult to generalize to new images or downstream tasks. For instance, if an object or relationship isn't present in the predefined vocabulary of a particular scene graph generation model, it will be omitted, even if it's clearly visible in the image. Furthermore, the same entity can be represented by different labels across different datasets (e.g., "man" vs. "person"), further fragmenting the semantic knowledge.
Methods
The core of this paper is the Entity Synset Alignment (ESA) method. ESA aims to create a unified scene graph by integrating information from multiple existing scene graphs. The process involves:
- Scene Graph Generation: They first generate scene graphs from raw images using existing models. For object and attribute extraction, they use a bottom-up attention model (Anderson et al., 2018). For relationship generation, they employ a model called CompTransR, which uses compositional translational embedding to learn semantic relationships between entities.
- Entity Synset Alignment: This is the novel contribution. ESA aligns entity labels from different scene graphs. It leverages:
- WordNet: A large lexical database that groups words into sets of synonyms (synsets). ESA uses synsets to identify if entity labels from different graphs refer to the same concept, even if the exact wording differs.
- Intersection over Union (IoU): After identifying potential label matches using synsets, IoU is used to quantitatively assess the spatial overlap of the corresponding bounding boxes. A threshold (0.3 in this paper) is used to confirm that the aligned entities indeed refer to the same object in the image.
Datasets
The paper utilizes the following datasets:
- Visual Genome (VG): A large-scale dataset for scene graph generation.
- VG200: A filtered version of the original VG dataset.
- BottomUp-VG: A dataset generated by the bottom-up attention model.
- VrR-VG (Visually-Relevant Relationships): A dataset focused on visually-relevant relationships.
- VG-COCO: A subset of images that belong to both the MS COCO dataset (for captions) and the Visual Genome dataset (for scene graphs), used for the image-caption retrieval task.
Key Insights and Ideas
- The problem of vocabulary bias is significant: The paper clearly demonstrates how limited vocabularies in individual scene graph datasets lead to incomplete representations of images.
- WordNet is a powerful tool for semantic alignment: Leveraging WordNet's synsets allows for a more robust matching of entity labels across different datasets, going beyond simple string matching.
- IoU provides a crucial spatial validation: Combining semantic alignment with spatial overlap (IoU) is essential to ensure that aligned entities are indeed the same object in the image, preventing spurious matches.
- Integration leads to richer scene graphs: The quantitative and qualitative results strongly suggest that combining scene graphs from multiple sources results in a more comprehensive and informative representation of an image.
Key Results
- Qualitative Improvement: Figure 3 shows that the integrated scene graph captures more entities and relationships compared to individual scene graphs, and it aligns entities like "person" and "woman" to a single representation.
- Quantitative Improvement in Image-Caption Retrieval: When applying the integrated scene graph to the image-caption retrieval task, the graph-based method (using GCN) shows superior performance compared to a standard CNN-based model (ResNet-152).
- Integrated Scene Graphs Outperform Default Ones: Table 2 shows that using integrated scene graphs (e.g., VG200 + VrR-VG + BU-VG) generally leads to better performance in image-caption retrieval compared to using individual scene graphs. For instance, the integrated approach achieves higher Recall@K values.
Limitations and Future Work
- Limitations: The paper acknowledges that the current ESA method relies on pre-generated scene graphs. The quality of the integrated scene graph is thus dependent on the quality of the input scene graphs.
- Future Work: The authors plan to integrate more diverse visual semantic knowledge, such as human-object interactions, into their framework. They also suggest exploring more advanced methods for scene graph integration.
Summary updated