Leveraging Hidden Structure in Self-Supervised Learning

Emanuele Sansone

Abstract

This work considers the problem of learning structured representations from raw images using self-supervised learning. We propose a principled framework based on a mutual information objective, which integrates self-supervised and structure learning. Furthermore, we devise a post-hoc procedure to interpret the meaning of the learnt representations. Preliminary experiments on CIFAR-10 show that the proposed framework achieves higher generalization performance in downstream classification tasks and provides more interpretable representations compared to the ones learnt through traditional self-supervised learning.

Introduction

Self-supervised learning (SSL) has revolutionized the training of AI models by enabling them to learn from vast amounts of unlabeled data. However, most current SSL methods learn "distributed" representations, where image information is compressed into a single feature vector. This approach ignores the inherent structure of the physical world, where scenes are composed of distinct entities (objects) and their relationships. By failing to account for this structure, models miss a powerful inductive bias, and the resulting "black box" representations remain difficult to interpret. This paper addresses the challenge of learning these entities and relations directly from pixels without human-provided labels or bounding boxes.

Methods

The core of the proposed framework is a principled approach based on Generalized Mutual Information. Instead of maximizing information solely between an image ($X$) and its representation ($Z$), the framework introduces a latent structure tensor ($A$) to represent relations, aiming to maximize the joint mutual information $I(X, Z, A)$.

To make the optimization tractable, the model assumes that entities and their relations are conditionally independent given the image. This allows the objective to be decomposed into two parts: $I(X, Z) + I(X, A)$. The architecture utilizes an encoder ($g_\theta$) to produce a global representation that is reshaped into a latent tensor $Z$ representing $S$ entities. A Message Passing Neural Network (MPNN) then processes these entities to predict the relational tensor $A$. Two "critic" functions, $T_\beta$ and $T_\omega$, are employed to estimate the mutual information bounds during training.

Architecture of the proposed SSL framework showing the two critic functions $T_\beta$ and $T_\omega$ used to estimate mutual information.

To understand what the model learns, a post-hoc interpretation procedure was developed. By minimizing an upper bound of the mutual information between a specific row of the latent tensor and the input image, the model generates a "soft mask" showing which pixels contribute to that specific entity representation.

The post-hoc interpretation procedure using a masking mechanism to visualize the correspondence between latent entities and image regions.

Experiments and Results

The framework was evaluated on two standard computer vision benchmarks: CIFAR-10 and CIFAR-100. On CIFAR-10, the structured approach achieved test accuracies of 56.3% (using a Conv-4 backbone) and 59.5% (using a Resnet-8 backbone), consistently outperforming non-structured baselines. Transfer learning experiments on CIFAR-100 further demonstrated the robustness of the learned features, with the model reaching 29.6% accuracy compared to the baseline's 28.2%.

Qualitative analysis revealed emergent specialization: the model naturally binds specific latent rows to distinct image regions (e.g., focusing on the wheels of a vehicle) without explicit supervision. This suggests that explicitly modeling relations acts as an effective inductive bias, forcing the latent space to organize in a way that facilitates relational reasoning.

Conclusion

This work demonstrates that explicitly incorporating hidden structure into self-supervised learning improves both downstream performance and model interpretability. While the results on small backbones are promising, future work will focus on scaling to larger architectures like ResNet-50 and implementing dynamic entity discovery to handle varying scene complexity. Additionally, exploring mechanisms like Slot Attention could further refine the model's ability to cluster input data into distinct objects.

Summary updated

Abstract

This work considers the problem of learning structured representations from raw images using self-supervised learning. We propose a principled framework based on a mutual information objective, which integrates self-supervised and structure learning. Furthermore, we devise a post-hoc procedure to interpret the meaning of the learnt representations. Preliminary experiments on CIFAR-10 show that the proposed framework achieves higher generalization performance in downstream classification tasks and provides more interpretable representations compared to the ones learnt through traditional self-supervised learning.
BibTeX
@misc{sansone2021leveraging,
  author = {Emanuele Sansone},
  title = {{Leveraging Hidden Structure in Self-Supervised Learning}},
  year = {2021},
  eprint = {2106.16060},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.16060},
}
Figure