Zero-Shot Scene Graph Relation Prediction Through Commonsense Knowledge Integration

Xuan Kan, Hejie Cui, Carl Yang

TL;DR

This paper tackles the critical challenge of "zero-shot" relation prediction in scene graphs, where models must infer relationships between objects they haven't seen during training. They introduce COACHER, a novel framework that integrates commonsense knowledge from ConceptNet to improve generalization. By leveraging semantic similarities between entities and relationships found in external knowledge graphs, COACHER significantly boosts performance on unseen triplets.

What Problem Does This Paper Address?

Scene graph generation (SGG) aims to create structured representations of images by identifying objects and the relationships between them. While SGG has advanced significantly, a major hurdle remains: predicting relationships for object pairs that were not present in the training data (zero-shot triplets). Existing SGG models often struggle with this, exhibiting a dramatic performance drop. This limitation hinders their real-world applicability, as the world is full of an inexhaustible number of possible object-relation combinations. The authors argue that this deficiency stems from a lack of commonsense reasoning – the human ability to infer relationships based on general world knowledge.

Methods

The core of the proposed solution, COACHER, lies in integrating commonsense knowledge from ConceptNet, a large, crowd-sourced semantic knowledge graph. COACHER builds upon a state-of-the-art SGG framework (Neural Motifs) and introduces three key "integrators" for commonsense knowledge:

  • Neighbor Integrator: This method leverages the semantic similarity between entities. In ConceptNet, entities with similar neighbors are considered semantically similar. For example, "child" and "man" might share neighbors like "fun" and "sleep," suggesting they can participate in similar relationships. COACHER calculates this similarity using the Jaccard index of their neighbors in ConceptNet.
  • Path Integrator: This approach exploits the relationships encoded in paths within ConceptNet. If two entity pairs share common intermediate paths, they are likely to have similar relations. COACHER defines "MidPaths" (sequences of edges and nodes excluding the start and end) and uses a message-passing mechanism to learn embeddings from these paths.
  • Fused Integrator: This combines the strengths of both neighbor and path integrators. It initializes the path-based message passing with a combination of neighbor embeddings and original node features.

These commonsense embeddings are then integrated into the SGG pipeline to enhance relation prediction, particularly for zero-shot triplets.

Datasets

The primary dataset used is Visual Genome [8], a widely adopted benchmark for SGG. The authors follow a common split strategy, using 70% of images for training and 30% for testing, with a 5k validation set for tuning.

To specifically test and amplify the zero-shot generalization capabilities, they also create a "zero-shot amplified dataset." This dataset is constructed by removing images with less common relations from the training data, thereby exacerbating the zero-shot challenge and making it harder for models to rely on memorization.

Key Insights and Ideas

  • Commonsense is Key for Generalization: The paper strongly posits that the inability to handle zero-shot triplets in SGG is directly linked to the absence of commonsense reasoning. Humans effortlessly use their understanding of the world to make such predictions.
  • ConceptNet as a Rich Source: ConceptNet's vast network of concepts and relations provides a valuable external knowledge source that can bridge the gap between visual data and abstract reasoning.
  • Neighbor Similarity Reflects Entity Similarity: The overlap in neighbors between two entities in ConceptNet is a strong indicator of their semantic similarity, which can be leveraged to infer similar relational behaviors.
  • Path Similarity Reflects Relation Similarity: Shared paths between entity pairs in ConceptNet can reveal underlying relational patterns, allowing for better prediction of unseen relations.
  • Zero-Shot Triplets are Common and Important: The paper highlights that many "zero-shot" triplets are actually quite common in the real world, making their accurate prediction crucial for practical SGG applications.

Key Results

COACHER, particularly the COACHER-N+P (fused integrator) variant, consistently outperforms baseline methods on both the original and the zero-shot amplified Visual Genome datasets.

  • On the original dataset, COACHER-N+P achieves the highest zero-shot recall (zR@K) and no-graph-constraint recall (ng-zR@K) across various K values. For instance, it achieves a zR@100 of 22.22% and ng-zR@100 of 41.39%, surpassing strong baselines like NeuralMotifs (NM+) and TDE.
  • On the zero-shot amplified dataset, the performance gap widens, with COACHER-N+P again showing superior generalization. For example, it achieves a zR@100 of 20.26% and ng-zR@100 of 39.57%, significantly outperforming others.
  • Qualitative case studies demonstrate that COACHER can correctly predict relations for zero-shot triplets where baselines fail, showcasing the practical benefit of its commonsense integration.

Limitations and Future Work

  • Relation Types in ConceptNet vs. Scene Graphs: The authors acknowledge that ConceptNet's relations are often more factual (e.g., "has property," "synonym") compared to the action-oriented and spatial relations found in scene graphs (e.g., "riding," "on," "near"). This suggests a potential mismatch that could be addressed.
  • Future Work - Scene-Oriented Knowledge Graph: A promising direction is to construct a scene-oriented commonsense knowledge graph specifically tailored for visual tasks. This could bridge the gap between visual reasoning and knowledge extraction more effectively.
  • Future Work - Refining Graph Construction: The downstream training process could be used to further refine the constructed knowledge graph, making it more relevant to visual contexts.
  • Path Length Limitations: Due to GPU memory constraints, the authors primarily used 1 and 2-hop paths. Exploring longer paths could potentially capture more nuanced relationships, though this may introduce noise.

Summary updated

Abstract

Relation prediction among entities in images is an important step in scene graph generation (SGG), which further impacts various visual understanding and reasoning tasks. Existing SGG frameworks, however, require heavy training yet are incapable of modeling unseen (i.e., zero-shot) triplets. In this work, we stress that such incapability is due to the lack of commonsense reasoning, i.e., the ability to associate similar entities and infer similar relations based on general understanding of the world. To fill this gap, we propose CommOnsense-integrAted sCene grapH rElation pRediction (COACHER), a framework to integrate commonsense knowledge for SGG, especially for zero-shot relation prediction. Specifically, we develop novel graph mining pipelines to model the neighborhoods and paths around entities in an external commonsense knowledge graph, and integrate them on top of state-of-the-art SGG frameworks. Extensive quantitative evaluations and qualitative case studies on both original and manipulated datasets from Visual Genome demonstrate the effectiveness of our proposed approach. The code is available at https://github.com/Wayfear/Coacher .
BibTeX
@inproceedings{kan2021zero,
  author = {Xuan Kan and Hejie Cui and Carl Yang},
  title = {{Zero-Shot Scene Graph Relation Prediction Through Commonsense Knowledge Integration}},
  booktitle = {ECML/PKDD},
  year = {2021},
  doi = {10.1007/978-3-030-86520-7\_29},
}
Figure