Seeing and Knowing in the Wild: Open-domain Visual Entity Recognition with Large-scale Knowledge Graphs via Contrastive Learning

Hongkuan Zhou, Lavdim Halilaj, Sebastian Monka, Stefan Schmid, Yuqicheng Zhu, Jingcheng Wu, Nadeem Nazer, Steffen Staab

Introduction

Traditional image classification systems are often constrained by fixed, predefined label sets (e.g., ImageNet's 1,000 categories). In contrast, real-world visual understanding requires linking images to a vast and dynamic universe of concepts. This paper introduces KnowCoL (Knowledge-Guided Contrastive Learning), a framework designed for open-domain visual entity recognition (OVEN). By projecting images, text queries, and structured knowledge graph (KG) data into a shared semantic space, KnowCoL achieves state-of-the-art performance on millions of entities, including those never seen during training.

The Challenge of Open-Domain Visual Entity Recognition

Open-domain visual entity recognition involves identifying a unique entity from a massive knowledge base like Wikidata (e.g., mapping a building to its specific ID, "Q41225") based on an image and a textual query. This task presents two primary hurdles:

  1. Scale and Sparsity: The label space contains millions of entities, many of which are rare or zero-shot.
  2. Semantic Ambiguity: Existing generative models often compress visual information into text labels (e.g., "Mercury"), losing the nuance required to distinguish between a planet and a chemical element.
Figure 1: (a) Examples of visual entity recognition queries in the OVEN benchmark. (b) Conceptual overview of the KnowCoL framework, aligning image, text, and semantic spaces.

Current approaches typically rely on dual-encoders (like CLIP) or two-step generative models. However, these methods often treat entities as isolated labels, ignoring the rich relational data—such as hierarchy and attributes—available in knowledge graphs.

Methodology: KnowCoL

KnowCoL addresses these limitations by integrating structured entity knowledge directly into the learning process. The framework utilizes three core components:

  • Input Fusion: Image and text query features are extracted via CLIP encoders and fused into a single multimodal representation.
  • Multimodal Entity Representation: Target entities are represented not just by text, but by their Wikipedia descriptions, lead images, and their structural positions within the Wikidata Knowledge Graph.
  • Knowledge Graph Embeddings (KGE): Using TransE-based methods adapted for cosine similarity, the model maps relationships like instance of or subclass of to ensure related entities cluster logically in the latent space.

The model is optimized using a tripartite contrastive loss: an Alignment Loss to match inputs to entity nodes, a Proxy Loss to align nodes with their descriptions/images, and a Knowledge Graph (KE) Loss to preserve the structural integrity of the KG.

Experimental Results

The authors evaluated KnowCoL on the OVEN Benchmark, which spans over 6 million training samples grounded in Wikidata.

State-of-the-Art Performance

KnowCoL-bigG (2.0B parameters) achieved a 38.8% Harmonic Mean (HM) accuracy, significantly outperforming the previous state-of-the-art, Auto-VER-14B (28.9% HM), despite being 7x smaller.

Zero-Shot Generalization

The framework's reliance on graph structure provides a powerful prior for unseen entities. The 0.4B parameter KnowCoL-L model achieved 29.1% accuracy on unseen entities, a 10.5% absolute improvement over the 14B parameter Auto-VER. Ablation studies confirmed that removing hierarchical knowledge (e.g., subclass of relations) resulted in a significant performance drop, proving that the graph structure is essential for effective disambiguation.

Conclusion and Future Work

KnowCoL demonstrates that structured knowledge is a vital component for scaling visual recognition to the open world. By moving away from flat label spaces and embracing the relational web of knowledge graphs, the model achieves superior accuracy and efficiency. Future research may explore non-Euclidean geometries, such as hyperbolic spaces, to better represent the naturally hierarchical structures found in global knowledge bases.

Summary updated

BibTeX
@inproceedings{zhou2026seeing,
  author = {Hongkuan Zhou and Lavdim Halilaj and Sebastian Monka and Stefan Schmid and Yuqicheng Zhu and Jingcheng Wu and Nadeem Nazer and Steffen Staab},
  title = {{Seeing and Knowing in the Wild: Open-domain Visual Entity Recognition with Large-scale Knowledge Graphs via Contrastive Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i16.38370},
}
Figure