Building a Visual Semantics Aware Object Hierarchy

Xiaolei DiaoIJCAI 2022 · DOI

TL;DR

This paper tackles the "semantic gap" in computer vision, where linguistic labels for images don't always align with their visual content, leading to biased models. They propose a novel unsupervised method to build a "visual semantics aware object hierarchy" directly from visual features, aiming to create more robust and less biased object recognition models.

What Problem Does This Paper Address?

The core problem this paper addresses is the "semantic gap" in computer vision. This gap refers to the discrepancy between how humans describe an object linguistically and how that object is represented visually. For instance, a single car might be labeled as "car," "Ferrari," or even "LaFerrari" depending on the observer's knowledge and the image's context. This subjectivity in labeling is a major issue because current machine learning models, especially in computer vision, heavily rely on these linguistic annotations for training. This reliance introduces an inevitable bias into the models, leading to suboptimal performance on real-world tasks. The authors argue that this bias stems from the fact that linguistic representations are not always the most accurate or comprehensive way to describe visual concepts.

Methods

The authors propose a novel unsupervised method to construct a "visual semantics aware object hierarchy." Their approach is inspired by how knowledge is represented in the real world, where concepts are organized hierarchically and described by a set of features rather than just a linguistic label.

The core of their method involves modeling object categories as "substance concepts" organized into a visual hierarchy. Within this hierarchy, each node represents a concept and is described by a set of visual features derived from the visual objects stored in that node. They use an algorithm inspired by [Giunchiglia et al., 2021], which employs two pre-defined properties: "Genus" (a set of existing properties an object satisfies) and "Differentia" (a new set of properties that distinguish an object from others with the same genus). These properties help determine an object's specific position within the hierarchy, indicating whether it's an existing concept, a new one, or requires further refinement. The "memory" in their algorithm is a collection of concepts organized hierarchically, where child nodes inherit features from their parent nodes and add distinguishing details.

Datasets

The paper mentions using VGG-16 for pre-training on ImageNet and grouping similar frames using DeeperCluster. For evaluation, they construct a lexical hierarchy based on the WordNet database. They also mention exploring the advantages of ground truth based on visual Genus and Differentia by reorganizing and annotating visual data, implying the creation or adaptation of a dataset for this purpose, though specific dataset names are not detailed.

Key Insights and Ideas

  • Visual Semantics as the Foundation: The paper champions the idea that visual features themselves can form a robust and objective basis for understanding and organizing concepts, bypassing the subjectivity of linguistic labels.
  • Hierarchy for Robustness: Organizing visual concepts into a hierarchy, akin to human knowledge structures, can lead to more nuanced and accurate object recognition.
  • Dissipating Linguistic Bias: By learning directly from visual information, the proposed method aims to mitigate the inherent biases introduced by human-generated linguistic annotations.
  • Bridging the Gap: The ultimate goal is to build a classification model that is less susceptible to the semantic gap, leading to improved performance in computer vision tasks.

Key Results

Preliminary results indicate that the proposed method achieves a relative performance improvement of around 10% on all depth models when trained with the same depth models and datasets but with different ground truths. The authors state that their constructed visual hierarchy, when applied to an object recognition task, shows validity when compared to existing lexical hierarchies.

Limitations and Future Work

The authors acknowledge that their project is "well underway" and that preliminary experimental results have demonstrated superiority. They mention that the success of the bag-of-words model and data distillation model has provided viable solutions for feature analysis on visual object hierarchy.

Future work includes applying the visual object hierarchy to various tasks such as object recognition, content-based image retrieval, and scene understanding. They also highlight its potential role in machine learning interpretability by providing a macroscopic view of how models work, thus offering a solution for explaining machine decisions and fostering trust in AI.

Summary updated

BibTeX
@inproceedings{diao2022building,
  author = {Xiaolei Diao},
  title = {{Building a Visual Semantics Aware Object Hierarchy}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/826},
}
Figure