Learning Representations by Maximizing Mutual Information Across Views

Philip Bachman, R. Devon Hjelm, William Buchwalter

Executive Summary

AMDIM (Augmented Multiscale Deep InfoMax) is a self-supervised learning framework that achieves state-of-the-art image representations by forcing a neural network to recognize that different "views" of the same image share the same underlying information. By comparing features across different scales and layers of the network, the model learns high-level concepts that are robust to noise, leading to a massive 12% jump in ImageNet linear evaluation accuracy over previous methods at the time of publication.

The Challenge of Self-Supervised Learning

In deep learning, labeled data is the gold standard but is incredibly expensive to produce at scale. Self-supervised learning aims to bypass this bottleneck by creating "pretext tasks" where the data provides its own supervision.

The primary challenge is designing a task that forces the model to learn useful high-level features (like shapes and objects) rather than "cheating" by focusing on low-level details (like pixel noise, textures, or edges). Earlier methods tried predicting the relative position of image patches or the rotation of an image, but these often failed to capture the complex, global information needed for downstream tasks like image classification.

The AMDIM Approach

The authors introduce Augmented Multiscale Deep InfoMax (AMDIM). The core idea is to maximize the Mutual Information (MI) between different features extracted from the same "context" (an image).

AMDIM Architecture

Key components of the framework include:

  • Multiple Views via Augmentation: For a single image, the model generates two different versions using stochastic data augmentation (e.g., random cropping, color jittering, and grayscaling). Instead of just expanding a dataset, augmentation creates different "perspectives" of the same underlying cause.
  • Multiscale MI Maximization: Instead of just comparing the final output of the network, AMDIM compares features across different layers. It compares a "global" feature (representing the whole image) from one view to "local" features (representing small patches) from the other view.
  • Noise-Contrastive Estimation (NCE): To maximize MI, the model uses a contrastive loss. Given a feature from View A, the model must pick its corresponding feature in View B from a "bag of distractors" (features from entirely different images).
  • Receptive Field Control: A subtle but vital contribution is the design of the encoder. The authors carefully manage the "receptive field" (the area of the original image a specific neuron "sees") to ensure that the features being compared don't overlap in pixel space. If they overlapped, the model could "cheat" by matching exact pixel values rather than learning abstract concepts.

Key Results and Insights

The model was evaluated on standard computer vision benchmarks, including ImageNet, CIFAR10/100, STL10, and Places205. AMDIM set a new bar for self-supervised learning:

  • ImageNet Dominance: Achieved 68.1% accuracy using a "linear evaluation" (freezing the encoder and only training a simple linear classifier on top). This was a 12% improvement over the previous state-of-the-art and beat concurrent high-profile models by 7%.
  • Strong Transferability: Representations learned on ImageNet translated exceptionally well to the Places205 dataset (55% accuracy), beating prior unsupervised methods by a significant margin. On STL10, it reached over 94% accuracy, performing on par with fully supervised models.
  • Emergent Segmentation: When extended to use "mixture-based representations," the model naturally started to cluster similar parts of an image together. It learned to segment objects from backgrounds without ever being explicitly taught what an "object" is.

Limitations and Future Work

While powerful, AMDIM has a few notable constraints:

  • Computational Cost: Training AMDIM is highly resource-intensive, requiring 4-8 Tesla V100 GPUs in the authors' experiments, which may be prohibitive for smaller research labs.
  • Hyperparameter Sensitivity: The mixture-based models, while promising for segmentation, are more sensitive to hyperparameters and require more tuning than the standard version.
  • Architectural Constraints: Because the custom encoder must strictly prevent "cheating" via overlapping receptive fields, it limits the use of standard, off-the-shelf architectures (like a vanilla ResNet-50) without modifications. Future work could explore how to apply these MI principles to standard architectures more seamlessly.

Summary updated

Abstract

We propose an approach to self-supervised representation learning based on maximizing mutual information between features extracted from multiple views of a shared context. For example, one could produce multiple views of a local spatio-temporal context by observing it from different locations (e.g., camera positions within a scene), and via different modalities (e.g., tactile, auditory, or visual). Or, an ImageNet image could provide a context from which one produces multiple views by repeatedly applying data augmentation. Maximizing mutual information between features extracted from these views requires capturing information about high-level factors whose influence spans multiple views – e.g., presence of certain objects or occurrence of certain events. Following our proposed approach, we develop a model which learns image representations that significantly outperform prior methods on the tasks we consider. Most notably, using self-supervised learning, our model learns representations which achieve 68.1% accuracy on ImageNet using standard linear evaluation. This beats prior results by over 12% and concurrent results by 7%. When we extend our model to use mixture-based representations, segmentation behaviour emerges as a natural side-effect. Our code is available online: https://github.com/Philip-Bachman/amdim-public.
BibTeX
@inproceedings{bachman2019learning,
  author = {Philip Bachman and R. Devon Hjelm and William Buchwalter},
  title = {{Learning Representations by Maximizing Mutual Information Across Views}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=QjkuO7uG0i},
}
Figure