Mapping of ImageNet and Wikidata for Knowledge Graphs Enabled Computer Vision

Dominik Filipiak, Anna Fensel, Agata FilipowskaBis 2021 · DOI

Introduction

While deep learning has fundamentally transformed computer vision, standard models remain largely "knowledge-blind." They treat labels like "husky" or "beagle" as independent categories, lacking the inherent understanding that both are types of dogs sharing biological and cultural traits. To address this, researchers are increasingly turning to Knowledge Graphs (KGs)—structured networks of real-world facts—to provide "prior knowledge."

This paper introduces a complete mapping between the 1,000 categories of the ILSVRC 2012 dataset (the "classic" ImageNet) and Wikidata entities. By bridging the world's most famous computer vision benchmark with a massive, structured knowledge graph, the authors provide a foundation for "knowledge-aware" AI models capable of leveraging external context for complex tasks like few-shot and zero-shot learning.

Methodology

The authors employed a multi-stage approach to ensure high-quality, verified links for every category in the standard ImageNet challenge:

  1. SPARQL Querying: Automated queries were used to identify existing links. Approximately 46% of ImageNet labels were already directly linked to Wikidata via the skos:exactMatch property.
  2. Manual Heuristic Search: For the remaining labels, authors performed manual searches using a specific heuristic: searching for the WordNet synset name, identifying the corresponding Wikipedia article, and extracting its linked Wikidata item. This covered an additional 46.7% of the dataset.
  3. Qualitative Analysis for "Hard" Cases: The final 7.2% (72 labels) presented issues due to mismatches in granularity or definition. The authors manually categorized these issues (e.g., hyponymy, ambiguity) and selected the semantically closest Wikidata entity.
  4. Automated Validation: To test scalability, the authors evaluated an automated Named Entity Recognition and Disambiguation (NERD) tool, entity-fishing, which correctly matched about 62% of synsets.

Challenges in Semantic Mapping

The primary obstacle in this mapping was the "Granularity Gap." WordNet, the lexical database underlying ImageNet, is often more granular than Wikidata. For instance, ImageNet distinguishes between a "magnetic compass" and other types, while Wikidata may only offer a general entry for "compass."

Biological entities presented unique linguistic quirks. ImageNet contains specific labels for "drake" (male duck) or "hen" (female chicken). Because Wikidata often lacks these gender-specific entities, these labels were mapped to the broader species level.

Figure 2: Examples of biological mappings illustrating the granularity gap where specific WordNet synsets (e.g., drake, hen) are mapped to broader Wikidata entities (e.g., duck, chicken).

Results and Metadata Enrichment

The project achieved 100% coverage, mapping all 1,000 ImageNet labels to Wikidata. This mapping unlocks significant metadata for each label. On average, each linked entity gains access to 28 direct properties, such as "subclass of," "image," or "instance of," which are crucial for calculating semantic distances between classes in hierarchical models.

Figure 5: Frequency of common Wikidata properties available for the mapped ImageNet entities, highlighting the rich metadata now accessible for computer vision tasks.

The analysis identified the most common properties available, including Freebase IDs (932 entities) and "subclass of" relationships (659 entities). These structured relationships allow models to understand the taxonomic hierarchy of the objects they are classifying.

Conclusion and Future Work

By linking ImageNet to Wikidata, this work provides a vital bridge between visual data and structured human knowledge. While the mapping inherits some of ImageNet's known biases and errors, it provides a robust starting point for the next generation of knowledge-aware vision models.

Future research will focus on empirically testing how this mapping improves performance in few-shot and zero-shot learning tasks and scaling the effort to the larger "ImageNet-6K" dataset. The full mapping is publicly available on GitHub to support the broader research community.

Summary updated

BibTeX
@inproceedings{filipiak2021mapping,
  author = {Dominik Filipiak and Anna Fensel and Agata Filipowska},
  title = {{Mapping of ImageNet and Wikidata for Knowledge Graphs Enabled Computer Vision}},
  booktitle = {Bis},
  year = {2021},
  doi = {10.52825/bis.v1i.65},
}
Figure