Zero-Shot Scene Graph Relation Prediction Through Commonsense Knowledge Integration

Xuan Kan, Hejie Cui, Carl Yang

TL;DR

This paper tackles the challenge of zero-shot relation prediction in scene graphs, where models struggle to identify relationships between objects not seen during training. They introduce COACHER, a framework that integrates commonsense knowledge from ConceptNet to improve generalization, achieving significant performance gains, especially in challenging zero-shot scenarios.

What Problem Does This Paper Address?

Scene graph generation (SGG) aims to create structured representations of images by identifying objects and the relationships between them. While SGG has advanced significantly, a major hurdle remains: predicting relationships that were not explicitly present in the training data (zero-shot triplets). Existing SGG models often rely on memorizing seen triplets, leading to a dramatic performance drop when encountering unseen combinations. This limitation hinders their real-world applicability, as the vast majority of possible object-relation combinations are impossible to cover in a training dataset. The paper argues that this incapability stems from a lack of commonsense reasoning – the human ability to infer relationships based on general world knowledge.

Methods

The core of this work is the COACHER framework, which integrates commonsense knowledge to enhance zero-shot relation prediction. COACHER builds upon a state-of-the-art SGG pipeline (NeuralMotifs) and introduces three novel "integrators" that leverage external commonsense knowledge from ConceptNet:

  • Neighbor Integrator: This module exploits the semantic similarity between entities. It leverages ConceptNet's graph structure to find common neighbors between entity classes. If two entities share many neighbors in ConceptNet (e.g., "child" and "man" both have neighbors like "fun," "sleep," "boy"), it suggests they are semantically similar and likely to participate in similar relations. The similarity is quantified using Jaccard similarity of their neighbors.
  • Path Integrator: This module captures relational similarity by analyzing paths in ConceptNet. If two entity pairs share common intermediate paths (sequences of nodes and edges) in ConceptNet, it implies they might have similar relationships. The paper defines "MidPaths" (paths excluding start and end nodes) and uses a message-passing mechanism to learn embeddings from these paths.
  • Fused Integrator: This module combines the strengths of both the neighbor and path integrators, initializing the path-based message passing with a combination of neighbor embeddings and original node features.

These integrators generate commonsense knowledge embeddings that are then fused with visual features to improve the relation prediction module.

Datasets

The primary dataset used is Visual Genome [8], a widely adopted benchmark for SGG. The authors follow a standard split strategy, using 70% of images for training, 30% for testing, and a 5k validation set for tuning.

To specifically test generalization on zero-shot triplets, they create a "zero-shot amplified dataset." This dataset is constructed by reducing the information available for less common relations in the training data, thereby exacerbating the zero-shot challenge and making it harder for models to rely on memorization.

Key Insights and Ideas

  • Commonsense is key for generalization: The paper strongly posits that the inability of current SGG models to handle zero-shot triplets is due to a lack of commonsense reasoning.
  • ConceptNet as a rich source: ConceptNet, a large, crowd-sourced semantic knowledge graph, provides a valuable resource for capturing world knowledge relevant to SGG.
  • Neighbor similarity implies entity similarity: The overlap of neighbors in ConceptNet between entity classes is a strong indicator of their semantic similarity, which can be leveraged for zero-shot prediction.
  • Path similarity implies relation similarity: Shared paths in ConceptNet between entity pairs can reveal underlying relational similarities, aiding in predicting unseen relations.
  • Zero-shot triplets are common and important: The paper highlights that many "zero-shot" triplets are actually quite common in the real world, making their accurate prediction crucial for practical SGG applications.

Key Results

COACHER, particularly the COACHER-N+P variant (fusing neighbor and path integrators), consistently outperforms baseline methods on both the original Visual Genome dataset and the more challenging zero-shot amplified dataset.

  • On the original dataset, COACHER-N+P achieves the highest zero-shot recall (zR@K) and no-graph-constraint recall (ng-zR@K) across different K values. For example, it achieves 13.42% zR@20 and 15.54% ng-zR@20, outperforming strong baselines like NeuralMotifs (NM) and TDE.
  • On the zero-shot amplified dataset, the performance gap widens, with COACHER-N+P showing even more significant gains, demonstrating its superior generalization ability in severe zero-shot settings.
  • Qualitative case studies (Figure 6) illustrate how neighbor and path-based commonsense knowledge directly contribute to correctly predicting zero-shot relations in scenarios where other methods fail.

Limitations and Future Work

  • Limitations of ConceptNet: The authors acknowledge that while ConceptNet is comprehensive, its relations are often more factual (e.g., "has property," "synonym") than the action-oriented or spatial relations common in scene graphs.
  • Future Work:
    • Investigating other external knowledge graphs for SGG.
  • Constructing a scene-oriented commonsense knowledge graph specifically tailored for visual tasks.
  • Further refining the graph construction process through downstream training.
  • Bridging the gap between visual reasoning and knowledge extraction communities.

Summary updated

BibTeX
@inproceedings{kan2021zero,
  author = {Xuan Kan and Hejie Cui and Carl Yang},
  title = {{Zero-Shot Scene Graph Relation Prediction Through Commonsense Knowledge Integration}},
  booktitle = {ECML/PKDD},
  year = {2021},
  doi = {10.1007/978-3-030-86520-7\_29},
}

Loading related content...

Figure