Object Detection Meets Knowledge Graphs

Yuan Fang, Kingsley Kuan, Jie Lin, Cheston Tan, Vijay Chandrasekhar

Abstract

Modern object detectors excel at pattern recognition but often lack the contextual "common sense" humans apply to scene interpretation. This research introduces a framework that integrates structured Knowledge Graphs (KGs), such as MIT ConceptNet, into the object detection pipeline. By employing a re-optimization technique, the framework ensures that detected objects are semantically consistent—for instance, recognizing that a cat is likely to be on a dining table, whereas a bear is not.

Introduction: The Semantic Gap in Object Detection

State-of-the-art object detection algorithms, like Faster R-CNN, primarily rely on visual features extracted from pixels. However, these models are often "context-blind." They do not inherently understand that certain objects frequently co-occur (e.g., a keyboard and a laptop) while others are mutually exclusive in typical settings (e.g., a bear on a dining table).

This lack of background knowledge creates a "semantic gap." A detector might fail to identify a blurry keyboard despite high confidence in a neighboring laptop, or it might hallucinate an improbable object due to ambiguous textures. This paper aims to bridge this gap by enabling detectors to leverage structured world knowledge from external KGs.

Methodology

The proposed knowledge-aware object detection framework functions as a post-processing enhancement layer applicable to any existing detector.

1. Quantifying Semantic Consistency

The core of the approach is the representation of relationships between object labels as numerical values in a consistency matrix $S$. The authors explored two primary methods for constructing this matrix:

  • Frequency-based: Utilizing Pointwise Mutual Information (PMI) to calculate co-occurrence statistics from training data.
  • Graph-based (KG-CNet): Applying Random Walk with Restart (RWR) on MIT ConceptNet. This method captures indirect relationships; for example, "person" and "plate" may be linked through "cat" (e.g., person pets cat, cat licks plate), indicating semantic relevance even without a direct edge.
Example of a Knowledge Graph representing semantic relationships between objects.

2. Re-optimization Framework

The framework takes raw detection probabilities from a base model and refines them through an iterative re-optimization process. The objective function balances two competing goals: maintaining fidelity to the original visual evidence and adjusting probabilities to maximize the semantic consistency of the final set of detected objects.

The proposed knowledge-aware object detection pipeline, integrating visual features with semantic consistency.

The optimization is computationally efficient, typically converging in fewer than 30 iterations, making it suitable for real-time applications.

Experimental Evaluation

The framework was evaluated on two major computer vision benchmarks:

  • MSCOCO15: A complex dataset featuring 80 categories and high scene density.
  • PASCAL07: A standard dataset with 20 categories.
  • External Knowledge Source: The authors utilized MIT ConceptNet, a crowdsourced KG containing over 4 million concepts and 9 million relationships.

Key Insights and Results

The integration of knowledge graphs yielded several significant findings:

  • Recall Enhancement: Background knowledge acts primarily as a recall booster. It allows the system to "rescue" objects that the visual detector might have missed due to low confidence, provided they are contextually appropriate. On MSCOCO, the KG-CNet variant improved Recall@100 by up to 4.6 points over the Faster R-CNN baseline, with a 6.3 point jump for large objects.
  • Superiority of Graph-based Knowledge: The KG-CNet approach outperformed simple frequency-based statistics, demonstrating that structured knowledge generalizes better to diverse scenes.
  • Complexity and Value: The benefits were more pronounced on the complex MSCOCO dataset than on PASCAL07, suggesting that the value of semantic knowledge increases with scene complexity.
  • Precision Stability: Crucially, the increase in recall did not degrade Mean Average Precision (mAP). The model became more effective at finding objects without increasing false positives.

Limitations and Future Outlook

While effective, the framework remains dependent on the quality of the underlying visual detector. If the base model provides extremely poor initial features, the KG enhancement has limited corrective power. Furthermore, while MIT ConceptNet is a powerful general-purpose tool, the authors suggest that KGs specifically tailored for visual-spatial relationships (e.g., "sits on" or "next to") might yield even stronger results. Future research will likely focus on optimizing knowledge graphs specifically for computer vision tasks.

Summary updated

Abstract

Object detection in images is a crucial task in computer vision, with important applications ranging from security surveillance to autonomous vehicles. Existing state-of-the-art algorithms, including deep neural networks, only focus on utilizing features within an image itself, largely neglecting the vast amount of background knowledge about the real world. In this paper, we propose a novel framework of knowledge-aware object detection, which enables the integration of external knowledge such as knowledge graphs into any object detection algorithm. The framework employs the notion of semantic consistency to quantify and generalize knowledge, which improves object detection through a re-optimization process to achieve better consistency with background knowledge. Finally, empirical evaluation on two benchmark datasets show that our approach can significantly increase recall by up to 6.3 points without compromising mean average precision, when compared to the state-of-the-art baseline.
BibTeX
@inproceedings{fang2017object,
  author = {Yuan Fang and Kingsley Kuan and Jie Lin and Cheston Tan and Vijay Chandrasekhar},
  title = {{Object Detection Meets Knowledge Graphs}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/230},
}
Figure