Knowledge Graph-Based Image Classification Refinement

Cheng Xie (谢诚), Dehai Zhang, Menglong Cui, Yun Yang, Po Yang, Cheng Xie, Di Liu, Beibei Yu, Zhibo Chen

Abstract

While modern Convolutional Neural Networks (CNNs) are masters of pixel-level patterns, they often lack the "common sense" context that humans use to classify images. This paper introduces the Image Knowledge Graph (IKG), a framework that refines CNN predictions by incorporating semantic and scene-based relationships. By leveraging external linguistic knowledge and internal statistical co-occurrences, the IKG framework boosts top-1 accuracy on ImageNet by over 2% across various architectures.

Introduction: The Context Gap in Computer Vision

Standard image classification algorithms treat categories as isolated labels. A CNN might struggle to distinguish between a "tiger cat" and a "tabby cat" because it focuses primarily on the pixels of the animal itself. However, a human observer uses context: a tiger cat is likely surrounded by grass and trees (wild), while a tabby is likely near a sofa or a keyboard (domestic).

The authors identify two main hurdles in current computer vision: Intraclass Diversity, where objects in the same category look different, and Interclass Confusion, where different categories look remarkably similar or share confusing backgrounds.

Figure 1: Examples of interclass confusion where visually similar categories like 'tabby' and 'tiger cat' or 'tape player' and 'cassette player' are difficult to distinguish without context.

By ignoring the semantic relationships between objects and the environments they inhabit, current models miss out on the "biological vision" advantage—the ability to use associations to aid recognition.

Methodology: The Image Knowledge Graph (IKG)

The proposed multi-step refinement process acts as a post-processing layer for existing high-performing CNNs:

  1. Entity Extraction: Images are processed through standard models (e.g., InceptionResNetV2) to extract the top-10 predicted labels.
  2. Building Associations: Two types of relationships are defined:
  3. Semantic Association: Using WordNet, the system calculates linguistic similarity between categories (e.g., taxonomic relations).
  4. Scene Association: A confusion matrix from training data determines which objects frequently co-occur or are commonly mistaken in specific scenes.
  5. IKG Construction: These associations are merged into a graph where categories are nodes and edges represent relationship strength.
  6. Refinement: The final classification score is recalculated based on the "neighborhood" in the IKG. If a predicted label has strong associations with other likely labels or detected background objects, its confidence is boosted.
Figure 2: The IKG construction process integrates semantic associations from the WordNet corpus with scene associations (like grass or wood) detected by CNNs to refine the final classification.

Key Insights and Results

The IKG framework treats the background as a feature rather than noise. For instance, a "computer desk" near a "mouse" provides vital evidence for the primary classification. The authors also discovered a "Filt" threshold: refinement is most effective when the CNN is uncertain (confidence scores between 0.4 and 0.5). If a model is already 90% sure, the IKG has less room to improve; if it is totally lost, the graph cannot recover the correct label.

The IKG consistently improved the performance of every CNN architecture tested on the ImageNet dataset:

  • InceptionResNetV2: 80.40% → 82.43% (+2.03% gain).
  • Xception: 79.0% → 80.43% (+1.43% gain).
  • VGG16: 71.5% → 73.27% (+1.77% gain).

In specific cases, such as identifying a "tiger cat," the IKG increased accuracy by approximately 10% by successfully leveraging "wild" vs. "indoor" scene associations.

Conclusion and Future Work

While the IKG provides a significant boost, it faces limitations in images with sparse context (plain backgrounds) or extreme semantic ambiguity (e.g., "wool" vs. "mitten"). Future research will extend the IKG framework beyond classification to more complex tasks like object detection and automated image captioning, where contextual reasoning is even more critical.

Summary updated

Abstract

Biologically inspired ideas are important in image processing. Not only does more than 80% of the information received by humans comes from the visual system, but the human visual system also gives its fast, accurate, and efficient image processing capability. In the current image classification task, convolutional neural networks (CNNs) focus on processing pixels and often ignore the semantic relationships and human brain mechanisms. With the development of image analysis and processing techniques, the information in the image is becoming increasingly complicated. Humans can learn about the characteristics of objects and the relationships that occur between them to classify the images. It is a significant characteristic that sets humans apart from the modern learning-based computer vision algorithms. How to make full use of the semantic relationships in categories and how to apply the knowledge of biological vision to image classification are our main concerns. In this view, we propose the concept of the image knowledge graph (IKG) to incorporate the semantic association and the scene association to fully consider the relations between objects (external and internal). We take full advantage of the reasoning model of the knowledge graph that is closer to the biological visual information-processing model. We conduct extensive experiments on large-scale image datasets (ImageNet), demonstrating the effectiveness of our approach. Furthermore, our method participates in ILSVRC 2017 challenges and obtains the new state-of-the-art results on the ImageNet (82.43%).
BibTeX
@article{2019knowledge,
  author = {Cheng Xie (谢诚) and Dehai Zhang and Menglong Cui and Yun Yang and Po Yang and Cheng Xie and Di Liu and Beibei Yu and Zhibo Chen},
  title = {{Knowledge Graph-Based Image Classification Refinement}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2912627},
}
Figure