Abstract
While modern Convolutional Neural Networks (CNNs) are masters of pixel-level patterns, they often lack the "common sense" context that humans use to classify images. This paper introduces the Image Knowledge Graph (IKG), a framework that refines CNN predictions by incorporating semantic and scene-based relationships. By leveraging external linguistic knowledge and internal statistical co-occurrences, the IKG framework boosts top-1 accuracy on ImageNet by over 2% across various architectures.
Introduction: The Context Gap in Computer Vision
Standard image classification algorithms treat categories as isolated labels. A CNN might struggle to distinguish between a "tiger cat" and a "tabby cat" because it focuses primarily on the pixels of the animal itself. However, a human observer uses context: a tiger cat is likely surrounded by grass and trees (wild), while a tabby is likely near a sofa or a keyboard (domestic).
The authors identify two main hurdles in current computer vision: Intraclass Diversity, where objects in the same category look different, and Interclass Confusion, where different categories look remarkably similar or share confusing backgrounds.
By ignoring the semantic relationships between objects and the environments they inhabit, current models miss out on the "biological vision" advantage—the ability to use associations to aid recognition.
Methodology: The Image Knowledge Graph (IKG)
The proposed multi-step refinement process acts as a post-processing layer for existing high-performing CNNs:
- Entity Extraction: Images are processed through standard models (e.g., InceptionResNetV2) to extract the top-10 predicted labels.
- Building Associations: Two types of relationships are defined:
- Semantic Association: Using WordNet, the system calculates linguistic similarity between categories (e.g., taxonomic relations).
- Scene Association: A confusion matrix from training data determines which objects frequently co-occur or are commonly mistaken in specific scenes.
- IKG Construction: These associations are merged into a graph where categories are nodes and edges represent relationship strength.
- Refinement: The final classification score is recalculated based on the "neighborhood" in the IKG. If a predicted label has strong associations with other likely labels or detected background objects, its confidence is boosted.
Key Insights and Results
The IKG framework treats the background as a feature rather than noise. For instance, a "computer desk" near a "mouse" provides vital evidence for the primary classification. The authors also discovered a "Filt" threshold: refinement is most effective when the CNN is uncertain (confidence scores between 0.4 and 0.5). If a model is already 90% sure, the IKG has less room to improve; if it is totally lost, the graph cannot recover the correct label.
The IKG consistently improved the performance of every CNN architecture tested on the ImageNet dataset:
- InceptionResNetV2: 80.40% → 82.43% (+2.03% gain).
- Xception: 79.0% → 80.43% (+1.43% gain).
- VGG16: 71.5% → 73.27% (+1.77% gain).
In specific cases, such as identifying a "tiger cat," the IKG increased accuracy by approximately 10% by successfully leveraging "wild" vs. "indoor" scene associations.
Conclusion and Future Work
While the IKG provides a significant boost, it faces limitations in images with sparse context (plain backgrounds) or extreme semantic ambiguity (e.g., "wool" vs. "mitten"). Future research will extend the IKG framework beyond classification to more complex tasks like object detection and automated image captioning, where contextual reasoning is even more critical.
Summary updated