Learning Object Semantic Similarity with Self-Supervision

Arthur Aubret, Timothy Schaumlöffel, Gemma Roig, Jochen Triesch

Learning Object Semantic Similarity with Self-Supervision

Abstract

This paper investigates how neural networks can learn complex semantic relationships between objects using two bio-inspired principles: temporal slowness and visuo-language alignment. By simulating egocentric visual experiences, the researchers demonstrate that grouping objects seen close together in time (temporal slowness) and associating them with names (visuo-language alignment) allows a model to organize information into context-wise clusters, such as grouping "mugs" and "kettles" within a "kitchen" context. The study reveals that semantic hierarchies emerge across network layers, with higher layers capturing abstract environmental associations through increased sparsity.

Introduction

Humans possess a deep understanding of how objects relate to their environments. We intuitively associate a "fork" with a "plate" because they frequently co-occur in a kitchen, despite having distinct visual features. Traditional AI models often focus on object categorization or identification but struggle to capture these broader contextual associations. This research explores whether the statistical structure of daily life—specifically the temporal proximity of related objects—is sufficient to build sophisticated semantic maps from scratch.

Methods

The researchers proposed a model inspired by human developmental learning, combining two key strategies:

  1. Self-Supervised Learning Through Time (SSLTT): Based on the "slowness principle," this method assumes that the environment changes slowly. Objects seen in quick succession are likely part of the same context. The model uses a contrastive loss to align representations of images appearing close in time.
  2. Visuo-Language Alignment (VLA): Mimicking caregiver interaction, this strategy aligns visual representations with category labels (e.g., "mug").
The learning architecture integrates temporal alignment (SSLTT) and language alignment (VLA) to learn semantic relationships.

The model was trained on sequences of images from the MVImageNet dataset, simulating an observer moving through various rooms. These sequences included intra-context transitions (moving between objects in the same room) and inter-context transitions (moving between rooms). To test the impact of environmental stability, the researchers varied the probability of context switches ($p_c$).

Results and Discussion

The model's ability to learn context was evaluated using an Odd-One-Out (OOO) task. When temporal transitions were stable (low probability of context switching, $p_c = 0.1$), the model achieved high OOO context accuracy (~0.75), significantly outperforming chance levels.

Visualizing the learned representations using t-SNE confirms the formation of distinct context clusters. As shown in the visualization below, stable temporal sequences lead to clear groupings of objects by room (e.g., kitchen, bathroom), whereas random sequences ($p_c = 1.0$) cause these clusters to collapse, proving that environmental stability is a requirement for learning these relationships.

t-SNE visualizations demonstrate that stable temporal transitions lead to clear context-based clustering.

Analysis of the network layers revealed a semantic hierarchy. Lower layers primarily encode object identity, while middle layers capture categories. The highest layers excel at capturing abstract context. This transition to abstraction is accompanied by increased layer sparsity, suggesting that as the model compresses visual data, it naturally moves toward more efficient, context-driven representations.

Conclusion

This work demonstrates that temporal slowness is a powerful driver for learning semantic context. While language helps group objects by category, the temporal structure of our visual experience is what allows us to bridge the gap between visually dissimilar but contextually related objects. Future research will focus on more realistic, continuous video streams and incorporating gaze data to further mimic human infant learning.

Summary updated

BibTeX
@inproceedings{aubret2024learning,
  author = {Arthur Aubret and Timothy Schaumlöffel and Gemma Roig and Jochen Triesch},
  title = {{Learning Object Semantic Similarity with Self-Supervision}},
  booktitle = {ICDL},
  year = {2024},
  doi = {10.1109/icdl61372.2024.10644930},
}
Figure