Symbolic Graph Inference for Compound Scene Understanding

Fnu Aryan, Simon Stepputtis, Sarthak Bhagat, Joseph Campbell, Kwonjoon Lee, Hossein Nourkhiz Mahjoub, Katia P. SycaraarXiv 2024 · DOI · arXiv

Introduction

Most current AI models treat scene understanding as a simple classification task: an image is processed, and a label like "bedroom" or "kitchen" is produced. However, real-world scenes are compound concepts—they are defined by the sum of their parts and the specific spatial relationships between them.

Researchers from Carnegie Mellon and Honda identify two major failure modes in existing end-to-end neural networks:

  1. The "Missing Piece" Problem: A kitchen remains a kitchen even if the oven is missing, provided other key elements like a sink and fridge are present. Standard models often struggle with such variance.
  2. The "Spatial Context" Problem: A dog and a sled only constitute a "dogsled" if they are arranged correctly. Similarly, a collection of boats isn't a "harbor" if there is no water present.

To address these, the authors developed a hybrid reasoning framework that combines Scene Graphs (SG) for spatial relationships with Knowledge Graphs (KG) for general domain logic.

Methodology

The proposed system bridges the gap between raw pixels and symbolic reasoning through a three-step pipeline:

  1. Graph Generation: The system uses a Faster R-CNN to detect objects (nodes) and a neural network to predict spatial relationships (edges). Each node is enriched with visual features using a Vision Transformer (ViT).
  2. Graph Merging: The Scene Graph (specific to the image) is linked to a hand-designed Knowledge Graph (containing general domain knowledge). This creates a "Merged Graph" where specific objects in the room are tied to abstract concepts.
  3. Dual-Graph Search: This core innovation performs an iterative search over the merged graph using a Propagation Network to identify neighborhoods of related concepts and an Importance Network to decide which nodes are relevant based on image context.
The proposed pipeline: from image to Scene Graph generation, merging with a Knowledge Graph, and final Dual-Graph Search.

To maintain efficiency, the authors introduced Dynamic Propagation, which halts the search once no new "important" information is discovered, significantly reducing runtime without sacrificing accuracy.

Experimental Evaluation

The method was evaluated using the ADE20K dataset, focusing on a subset of 20 compound classes (e.g., Bedroom, Mountain, Harbor). The final evaluation was conducted on a test set of 2,200 images, with a smaller 100-image subset used for direct comparison against high-cost models like GPT-4 Vision.

Key Results

  • Object-Level Reasoning: On the full test set, the proposed method achieved 74.52% accuracy, significantly outperforming the Knowledge Graph baseline (65.34%) and GPT-4 (61.89%).
  • Image-Level Reasoning: When conditioned on image features, the model reached 96.25% accuracy, approaching human-level performance (97%) and outperforming a standard ViT (85.82%).

Insights and Robustness

A primary advantage of this approach is its interpretability; the path the model takes through the graph to reach a conclusion is visible. Furthermore, the model excels in scenarios where visual-only models fail.

Qualitative comparison showing how the proposed model correctly identifies components (e.g., "Boats" instead of "Harbour") in tricky contexts where GPT-4 Vision is misled.

As shown in the comparison above, if an image shows boats on land (no water), the model correctly refuses to label it a "harbor," whereas GPT-4 Vision is often misled by the mere presence of boats. The spatial relationships in the Scene Graph act as the "glue" that allows the Knowledge Graph to correctly identify or reject a compound concept.

Conclusion and Future Work

While the current framework relies on a hand-designed Knowledge Graph for 20 classes, it demonstrates the power of hybrid symbolic-neural reasoning for robust scene understanding. Future work aims to scale this by automating KG construction from large language models and extending the framework to video understanding, where spatio-temporal reasoning will be critical for interpreting dynamic scenes.

Summary updated

Abstract

Scene understanding is a fundamental capability needed in many domains, ranging from question-answering to robotics. Unlike recent end-to-end approaches that must explicitly learn varying compositions of the same scene, our method reasons over their constituent objects and analyzes their arrangement to infer a scene's meaning. We propose a novel approach that reasons over a scene's scene- and knowledge-graph, capturing spatial information while being able to utilize general domain knowledge in a joint graph search. Empirically, we demonstrate the feasibility of our method on the ADE20K dataset and compare it to current scene understanding approaches.
BibTeX
@misc{aryan2024symbolic,
  author = {FNU Aryan and Simon Stepputtis and Sarthak Bhagat and Joseph Campbell and Kwonjoon Lee and Hossein Nourkhiz Mahjoub and Katia P. Sycara},
  title = {{Symbolic Graph Inference for Compound Scene Understanding}},
  year = {2024},
  eprint = {2410.22626},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.22626},
}
Figure