Introduction
Modern computer vision models are remarkably proficient at identifying objects they have encountered during training. However, they remain "knowledge-poor," struggling with "undetectable words"—concepts present in user queries that were not part of the model's original training vocabulary. For instance, a system trained on general objects might recognize a "person" or a "stove" but fail to identify a "chef" because that specific label was absent from its training set. This paper explores a method to bridge this gap by integrating ConceptNet, a massive commonsense knowledge base, into deep learning-based image retrieval systems.
Bridging the Semantic Gap with ConceptNet
The core challenge in image retrieval is mapping natural language queries to visual features. When a query contains a term the system cannot directly detect, the authors propose using an ontology to find "proxy" concepts. By linking undetectable words to visually similar or related concepts that the system does recognize, search accuracy for complex queries can be significantly improved.
As shown in the figure above, the concept "chef" is semantically linked to "kitchen," "cook," and "dish." If the retrieval system has detectors for these related terms, it can infer the presence of a chef even without a dedicated "chef" detector.
Methodology
The authors developed a multi-step pipeline to enhance standard image retrieval:
- Baseline (MIL): A Multiple Instance Learning approach utilizing 1,000 pre-trained visual detectors serves as the foundation.
- Stemming: Natural Language Processing (NLP) is used to match word variations (e.g., "running" to "run"), expanding the usable vocabulary.
- ConceptNet Integration: For words lacking a direct detector, the system queries ConceptNet to identify related concepts (e.g., "tuxedo" → "jacket").
- Visual Grounding with ESPGAME: Because not all logical relations are visually relevant (e.g., "a pen is used for writing"), the authors use the ESPGAME dataset to filter the ontology. This step calculates the probability of concept co-occurrence in actual images, removing "noisy" or non-visual relations.
- Score Aggregation: Scores from these proxy detectors are combined using functions like MAX or MEAN to estimate the likelihood of the original undetectable word's presence.
Key Insights and Results
A critical finding is that ontologies alone are insufficient for vision tasks due to "noise." For example, the relation "pig –AtLocation→ pen" is logically sound but visually confusing. Visual grounding via secondary datasets like ESPGAME is essential to ensure the system relies only on relations with a visual footprint.
The system proved more effective for nouns than for verbs or adjectives, as commonsense knowledge about objects (composition, location) is more robustly represented. Interestingly, even antonyms (e.g., "cat" and "dog") were found to be useful because they often appear in similar visual contexts.
On the MS COCO 5K benchmark, the ConceptNet-enhanced model (using MAX aggregation) improved the mean rank of correct images from 82.2 to 60.6. Furthermore, it achieved a Recall@1 of 16.2%, outperforming several state-of-the-art approaches, including NeuralTalk and BRNN. Qualitative results highlighted the system's ability to retrieve images of "bagels" by leveraging detectors for "doughnut" and "bread."
Conclusion and Future Work
While the integration of commonsense ontologies significantly boosts retrieval performance, challenges remain. The current method's bias toward nouns suggests a need for better modeling of actions and attributes. Additionally, the authors suggest that future work should explore more nuanced ways to utilize specific relation types, such as MadeOf or LocatedNear, rather than treating them as simple co-occurrence indicators. Integrating such structured knowledge remains a promising path toward making computer vision systems more "knowledge-rich."
Summary updated