State Representation Learning Using an Unbalanced Atlas

Li Meng, Morten Goodwin, Anis Yazidi, Paal E. Engelstad

Introduction

The manifold hypothesis posits that high-dimensional data, such as complex images, actually resides on a much lower-dimensional "surface" or manifold. Successfully learning this manifold allows models to create highly efficient and useful data representations.

While this concept is foundational in traditional dimensionality reduction, integrating it into modern self-supervised learning (SSL) has proven difficult. Previous manifold-based SSL attempts, such as MSimCLR, only performed well when the target encoding dimensions were extremely small (e.g., fewer than 8 dimensions). When scaled up to the hundreds or thousands of dimensions typically required in deep learning, these models suffered from high prediction uncertainty or "collapsed" into redundant, uninformative representations.

To overcome this scaling bottleneck, this paper introduces the Unbalanced Atlas (UA) paradigm, a novel approach to self-supervised learning that enables manifold-based models to scale to much higher dimensions and significantly outperform state-of-the-art methods.

The Unbalanced Atlas Paradigm

The authors draw inspiration from differential geometry, where an "atlas" is a collection of overlapping "charts" that collectively describe a complex manifold by mapping it to simpler Euclidean spaces.

A manifold Z covered by charts mapping to Euclidean spaces.

To adapt this concept for deep learning, the researchers introduce several key innovations:

  • Unbalanced Prior: Previous methods forced every chart to have an equal probability of representing a data point (a uniform prior). The paper shows that this forced uniformity leads to high entropy and uncertainty as dimensions grow. The UA paradigm deliberately trains the model to deviate from uniformity, which drastically reduces prediction uncertainty and increases model confidence.
  • Dilated Prediction Targets: To prevent multiple neural network "heads" (representing different charts) from converging to the same redundant solution, the authors use a Minkowski sum of chart outputs. This creates "dilated" targets that encourage diversity among the learned features.
  • DIM-UA Integration: The authors engineered a new algorithm by adapting the Spatiotemporal DeepInfomax (ST-DIM) framework to the UA paradigm, creating DIM-UA. They also utilize a Maximum Mean Discrepancy (MMD) objective to push the conditional membership distribution away from a uniform state.

Experimental Results

The researchers evaluated their approach on two main benchmarks: the Atari Annotated RAM Interface (AtariARI) (a suite of 19 Atari 2600 games that provides ground-truth labels for representation evaluation) and CIFAR10.

The UA paradigm consistently improved existing algorithms, particularly as the target encoding dimensions grew:

  • Scaling Success: While previous manifold-based SSL only worked for tiny dimensions, the UA paradigm thrives as the number of hidden units increases. Standard scaled-up models (like ST-DIM) eventually drop in performance due to overfitting on global features, but DIM-UA remains stable and continues to improve.
  • Atari Performance: When scaled to 16,384 hidden units, DIM-UA achieved a mean F1 score of ~75% across the 19 Atari games, compared to ~70% for the baseline ST-DIM.
Mean F1 and accuracy scores as the total number of hidden units increases. DIM-UA scales much better than ST-DIM.
  • CIFAR10 Accuracy: Integrating the UA paradigm into SimCLR (SimCLR-UA) achieved a top accuracy of 88.6%, outperforming both standard SimCLR (88.3%) and the previous manifold-based MSimCLR (85.9%) when using multiple heads.

Limitations and Future Work

While highly effective, the UA paradigm has a few limitations. The mathematical proofs for the "dilated targets" assume that the chart mappings are convex; while neural networks are universal approximators that handle non-convexity well in practice, this remains a theoretical gap. Additionally, training models with multiple output heads is inherently more computationally demanding than single-head models.

Finally, the authors note that there is likely an optimal number of heads related to the "intrinsic dimension" of the underlying data. Future work could involve developing techniques to automatically determine this optimal number during training.

Summary updated

Abstract

The manifold hypothesis posits that high-dimensional data often lies on a lower-dimensional manifold and that utilizing this manifold as the target space yields more efficient representations. While numerous traditional manifold-based techniques exist for dimensionality reduction, their application in self-supervised learning has witnessed slow progress. The recent MSimCLR method combines manifold encoding with SimCLR but requires extremely low target encoding dimensions to outperform SimCLR, limiting its applicability. This paper introduces a novel learning paradigm using an unbalanced atlas (UA), capable of surpassing state-of-the-art self-supervised learning approaches. We investigated and engineered the DeepInfomax with an unbalanced atlas (DIM-UA) method by adapting the Spatiotemporal DeepInfomax (ST-DIM) framework to align with our proposed UA paradigm. The efficacy of DIM-UA is demonstrated through training and evaluation on the Atari Annotated RAM Interface (AtariARI) benchmark, a modified version of the Atari 2600 framework that produces annotated image samples for representation learning. The UA paradigm improves existing algorithms significantly as the number of target encoding dimensions grows. For instance, the mean F1 score averaged over categories of DIM-UA is~75% compared to ~70% of ST-DIM when using 16384 hidden units.
BibTeX
@inproceedings{meng2024state,
  author = {Li Meng and Morten Goodwin and Anis Yazidi and Paal E. Engelstad},
  title = {{State Representation Learning Using an Unbalanced Atlas}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=cWdAYDLmPa},
}
Figure