Learning Representations by Maximizing Mutual Information Across Views

Philip Bachman, R. Devon Hjelm, William Buchwalter

Abstract

We propose an approach to self-supervised representation learning based on maximizing mutual information between features extracted from multiple views of a shared context. For example, one could produce multiple views of a local spatio-temporal context by observing it from different locations (e.g., camera positions within a scene), and via different modalities (e.g., tactile, auditory, or visual). Or, an ImageNet image could provide a context from which one produces multiple views by repeatedly applying data augmentation. Maximizing mutual information between features extracted from these views requires capturing information about high-level factors whose influence spans multiple views -- e.g., presence of certain objects or occurrence of certain events. Following our proposed approach, we develop a model which learns image representations that significantly outperform prior methods on the tasks we consider. Most notably, using self-supervised learning, our model learns representations which achieve 68.1% accuracy on ImageNet using standard linear evaluation. This beats prior results by over 12% and concurrent results by 7%. When we extend our model to use mixture-based representations, segmentation behaviour emerges as a natural side-effect. Our code is available online: https://github.com/Philip-Bachman/amdim-public.

Introduction

Deep learning usually relies on massive datasets labeled by humans, which is expensive and unscalable. Self-supervised learning aims to bypass this by creating "pretext tasks" where the data provides its own labels. However, many early pretext tasks were too low-level, allowing models to solve tasks like jigsaw puzzles by looking at edge alignments rather than understanding object semantics. The challenge addressed here is: how can we force a neural network to learn high-level, semantic representations without ever giving it a label?

The AMDIM Framework

We introduce Augmented Multiscale Deep InfoMax (AMDIM), a framework built on the core logic of maximizing mutual information (MI) across different augmented "views" of an image across multiple spatial scales. By forcing a model to recognize that two differently cropped and colored versions of an image share the same underlying content, the model learns features that are invariant to lighting or framing.

Multiscale Comparisons

The model doesn't just compare global summaries; it compares features from different layers of the network. For example, it might try to predict a local feature map in one view using a global vector from another view. This forces the model to relate local details to global context.

Overview of the AMDIM framework showing (a) global-to-local and (b) multi-level local-to-local prediction tasks across different augmented views.

Receptive Field Control and NCE

Crucially, we carefully control the "receptive field"—the area of the original image a specific feature sees. If the features being compared overlap too much, the task becomes a trivial pixel-matching exercise. By limiting overlap, the model is forced to learn more abstract relationships. The mathematical engine is Noise-Contrastive Estimation (NCE), where the model must pick its "true" match from a second view out of a bag of thousands of distractor features.

Algorithmic details for the memory-efficient Noise-Contrastive Estimation (NCE) and the ImageNet Encoder Architecture.

Key Results

The results represent a major leap forward for self-supervised visual recognition:

  • ImageNet Linear Evaluation: Using a frozen encoder and a simple linear classifier, AMDIM reached 68.1% top-1 accuracy, beating the previous state-of-the-art by over 12%.
  • STL-10: Achieved over 94% accuracy, competitive with models using significantly more labeled data.
  • Transfer Learning: Features learned on ImageNet achieved 55% accuracy on the Places205 dataset, proving the representations are generalizable.
  • Emergent Segmentation: When extended to mixture-based representations, the model naturally started to group pixels belonging to the same object without any pixel-level labels.

Limitations and Future Work

While powerful, the mixture-based version of the model is sensitive to hyperparameters and difficult to tune for large datasets. Additionally, hardware constraints limited ImageNet training to $128 \times 128$ resolution; higher resolutions would likely yield even better results. Future directions include extending this multi-view MI approach to other modalities, such as video, audio, and text.

Summary updated

Abstract

We propose an approach to self-supervised representation learning based on maximizing mutual information between features extracted from multiple views of a shared context. For example, one could produce multiple views of a local spatio-temporal context by observing it from different locations (e.g., camera positions within a scene), and via different modalities (e.g., tactile, auditory, or visual). Or, an ImageNet image could provide a context from which one produces multiple views by repeatedly applying data augmentation. Maximizing mutual information between features extracted from these views requires capturing information about high-level factors whose influence spans multiple views -- e.g., presence of certain objects or occurrence of certain events. Following our proposed approach, we develop a model which learns image representations that significantly outperform prior methods on the tasks we consider. Most notably, using self-supervised learning, our model learns representations which achieve 68.1% accuracy on ImageNet using standard linear evaluation. This beats prior results by over 12% and concurrent results by 7%. When we extend our model to use mixture-based representations, segmentation behaviour emerges as a natural side-effect. Our code is available online: https://github.com/Philip-Bachman/amdim-public.
BibTeX
@inproceedings{bachman2019learning,
  author = {Philip Bachman and R. Devon Hjelm and William Buchwalter},
  title = {{Learning Representations by Maximizing Mutual Information Across Views}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=QjkuO7uG0i},
}
Figure