Phase Transitions for the Information Bottleneck in Representation Learning

Tailin Wu, Ian S. Fischer

This paper explores a fascinating phenomenon in representation learning: the discrete "jumps" in model performance that occur as we tune the trade-off between compression and prediction. The authors demonstrate that learning within the Information Bottleneck (IB) framework does not happen smoothly. Instead, it progresses through discrete "phase transitions" where the model suddenly captures new features or classes.

Introduction: The Information Bottleneck and Phase Transitions

In representation learning, the Information Bottleneck (IB) objective is frequently used to find a representation $Z$ of an input $X$ that is as compressed as possible while remaining highly predictive of a target $Y$. This trade-off is expressed mathematically as:

$$L = I(X;Z) - \beta I(Y;Z)$$

Here, $\beta$ is a Lagrange multiplier controlling the balance. If $\beta$ is small, the model compresses everything into a trivial representation; if $\beta$ is large, it memorizes the data to maximize prediction accuracy.

Researchers have long observed that as $\beta$ increases, a model’s accuracy and mutual information do not always scale smoothly. Instead, they "jump" at specific values of $\beta$. This paper seeks to answer why these jumps happen, whether they can be predicted, and what they reveal about the underlying dataset.

Phase transitions across the information plane, information metrics, and accuracy.

Methods

The authors define an IB phase transition as a qualitative change in the loss landscape—specifically, a point where the local curvature of the objective function changes from a minimum to a saddle point.

  1. Second-Order Calculus of Variations: The authors use second-order variations to derive a threshold function, $G[p(z|x)]$. They prove that a phase transition occurs exactly when $G = \beta$.
  2. Fisher Information Connection: For modern parameterized neural networks, they show that this condition can be calculated using the Fisher Information Matrix, providing a crucial bridge between theoretical information theory and practical deep learning.
  3. Nonlinear CCA Perspective: The paper reveals that each phase transition is mathematically equivalent to finding a new "component of maximum correlation" between $X$ and $Y$ that is orthogonal to the current representation $Z$. This frames the IB as a nonlinear generalization of Canonical Correlation Analysis (CCA).
  4. Discovery Algorithm: The authors present an iterative algorithm that uses the Blahut-Arimoto method and Singular Value Decomposition (SVD) to "hop" from one transition point to the next, efficiently mapping out the "information bifurcation" tree.

Key Insights and Ideas

  • Learning is Quantized: Representation learning isn't a continuous slide; it is a series of "aha moments." Each phase transition represents the model "deciding" that the predictive gain of a new feature finally outweighs the "cost" of the extra bits required to store it.
  • The Orthogonality Principle: When a model hits a transition, it isn't just refining what it already knows. It is specifically isolating the strongest signal in the data that it hasn't learned yet.
  • Class Difficulty as a Metric: The value of $\beta$ at which a class is learned serves as a natural measure of difficulty. "Easy" classes (those with high correlation to the input) are learned at low $\beta$ values, while "hard" or noisy classes require a much higher $\beta$ to emerge from the bottleneck.
  • Predicting the Number of Transitions: For classification tasks, the authors conjecture and empirically support that there are at most $|Y| - 1$ phase transitions, where $|Y|$ is the total number of classes.

Experimental Results

The authors validated their theory across three types of data: small synthetic categorical datasets, MNIST (modified with controlled label noise), and CIFAR-10 (with 20% label noise).

  • Theoretical Match: In categorical datasets, the predicted transition points ($G = \beta$) matched the observed jumps in mutual information perfectly.
  • MNIST Class Onset: The algorithm accurately predicted the order in which digits were learned. In a noisy 4-class MNIST setup, it correctly identified that classes with less label noise are learned at lower $\beta$ values.
  • CIFAR-10 Transitions: Even in complex ResNet models, the algorithm successfully identified prominent phase transitions. It specifically found 9 transitions for the 10-class dataset, perfectly matching their theoretical conjecture of $|Y| - 1$.
  • Efficiency: Their "stairstep" discovery algorithm was able to skip over large, uninteresting regions of $\beta$ and precisely land on transition points, proving much faster than a brute-force sweep.

Limitations and Future Work

While the theory is exact for the "true" IB objective, practical deep learning relies on the Variational Information Bottleneck (VIB). The authors note a small gap between their predictions and empirical results in continuous settings due to this variational approximation. Furthermore, while the algorithm is efficient for small-to-medium models, calculating the Fisher Information Matrix for massive modern architectures remains computationally challenging. Future work may extend this framework to other trade-off-based objectives, such as $\beta$-VAEs or InfoDropout, to better understand how they disentangle representations.

Summary updated

BibTeX
@inproceedings{wu2020phase,
  author = {Tailin Wu and Ian S. Fischer},
  title = {{Phase Transitions for the Information Bottleneck in Representation Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=Oya6AoPiLd},
}
Figure