Phase Transitions for the Information Bottleneck in Representation Learning

Tailin Wu, Ian S. Fischer

Introduction

In representation learning, the Information Bottleneck (IB) framework provides a principled way to extract relevant features from data. It seeks a representation $Z$ that compresses the input $X$ while preserving information about a target $Y$. This trade-off is governed by the objective:

$$\min_{p(z|x)} I(X;Z) - \beta I(Y;Z)$$

where $\beta$ is a Lagrange multiplier controlling the balance between compression and prediction. While it was long assumed that increasing $\beta$ leads to a smooth improvement in performance, empirical observations often reveal sudden "jumps" in accuracy. This paper provides a theoretical explanation for these phenomena, characterizing them as discrete phase transitions in the learning process.

Theoretical Framework: Learning as a Series of Bifurcations

The authors demonstrate that representation learning via IB does not progress continuously. Instead, as $\beta$ increases, the model undergoes qualitative shifts in its loss landscape.

The $G$ Function and Curvature Analysis

Using second-order calculus of variations, the researchers analyzed the stability of the IB objective. They found that a phase transition occurs when the curvature of the loss function along a specific direction in probability space flips from positive to negative. This instability forces the model to "bifurcate," effectively unlocking a new feature or class to achieve a lower minimum. They derived a threshold function, $G[p(z|x)]$, such that a transition is triggered precisely when $\beta$ reaches the value of $G$.

Connection to Fisher Information

For neural network implementations, these transitions are linked to the Fisher Information Matrix of the encoder. This connection suggests that the "geometry" of the model's parameters determines when and how it learns new information. The paper also shows that these transitions represent a nonlinear generalization of Canonical Correlation Analysis (CCA), where each jump corresponds to identifying a new orthogonal component of correlation between $X$ and $Y$.

Empirical Evidence and Results

The theory was validated across synthetic categorical datasets, MNIST, and CIFAR-10. The results consistently showed that "easier" features (those with higher signal-to-noise ratios) are learned at lower $\beta$ values, while "harder" features require higher $\beta$ to trigger a transition.

Discrete phase transitions observed in the information plane (a), mutual information (b), and classification accuracy (c) as a function of the trade-off parameter $\beta$. Red and orange arrows indicate the predicted critical points.

Key findings include:

  • Predictable Jumps: In classification tasks, the number of phase transitions is bounded by $|Y| - 1$, where $|Y|$ is the number of classes. This was empirically confirmed on CIFAR-10, where 9 distinct transitions were identified.
  • Discontinuous Slopes: At critical $\beta$ values, the derivative of predictive power ($dI(Y;Z)/d\beta$) is discontinuous, creating visible "kinks" in the Information Plane plots.
  • Algorithmic Efficiency: The authors introduced an SVD-based "stairstep" algorithm (Algorithm 1) that can efficiently identify these critical points without requiring a dense, computationally expensive sweep of $\beta$ values.

Discussion and Limitations

While the theoretical framework is robust, its application to deep learning (Variational IB) faces some challenges. The "variational gap" between the true IB objective and its practical approximation can lead to missed or shifted transitions in high-dimensional spaces. Furthermore, the current practical algorithm relies on discretizing the latent space $Z$. Future work aims to extend these insights to fully continuous generative models and regression tasks, potentially providing a deeper understanding of how complex neural networks organize information during training.

Summary updated

BibTeX
@inproceedings{wu2020phase,
  author = {Tailin Wu and Ian S. Fischer},
  title = {{Phase Transitions for the Information Bottleneck in Representation Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=Oya6AoPiLd},
}

Loading related content...

Figure