Introduction
In representation learning, the Information Bottleneck (IB) framework provides a principled way to extract relevant features from data. It seeks a representation $Z$ that compresses the input $X$ while preserving information about a target $Y$. This trade-off is governed by the objective:
$$\min_{p(z|x)} I(X;Z) - \beta I(Y;Z)$$
where $\beta$ is a Lagrange multiplier controlling the balance between compression and prediction. While it was long assumed that increasing $\beta$ leads to a smooth improvement in performance, empirical observations often reveal sudden "jumps" in accuracy. This paper provides a theoretical explanation for these phenomena, characterizing them as discrete phase transitions in the learning process.
Theoretical Framework: Learning as a Series of Bifurcations
The authors demonstrate that representation learning via IB does not progress continuously. Instead, as $\beta$ increases, the model undergoes qualitative shifts in its loss landscape.
The $G$ Function and Curvature Analysis
Using second-order calculus of variations, the researchers analyzed the stability of the IB objective. They found that a phase transition occurs when the curvature of the loss function along a specific direction in probability space flips from positive to negative. This instability forces the model to "bifurcate," effectively unlocking a new feature or class to achieve a lower minimum. They derived a threshold function, $G[p(z|x)]$, such that a transition is triggered precisely when $\beta$ reaches the value of $G$.
Connection to Fisher Information
For neural network implementations, these transitions are linked to the Fisher Information Matrix of the encoder. This connection suggests that the "geometry" of the model's parameters determines when and how it learns new information. The paper also shows that these transitions represent a nonlinear generalization of Canonical Correlation Analysis (CCA), where each jump corresponds to identifying a new orthogonal component of correlation between $X$ and $Y$.
Empirical Evidence and Results
The theory was validated across synthetic categorical datasets, MNIST, and CIFAR-10. The results consistently showed that "easier" features (those with higher signal-to-noise ratios) are learned at lower $\beta$ values, while "harder" features require higher $\beta$ to trigger a transition.
Key findings include:
- Predictable Jumps: In classification tasks, the number of phase transitions is bounded by $|Y| - 1$, where $|Y|$ is the number of classes. This was empirically confirmed on CIFAR-10, where 9 distinct transitions were identified.
- Discontinuous Slopes: At critical $\beta$ values, the derivative of predictive power ($dI(Y;Z)/d\beta$) is discontinuous, creating visible "kinks" in the Information Plane plots.
- Algorithmic Efficiency: The authors introduced an SVD-based "stairstep" algorithm (Algorithm 1) that can efficiently identify these critical points without requiring a dense, computationally expensive sweep of $\beta$ values.
Discussion and Limitations
While the theoretical framework is robust, its application to deep learning (Variational IB) faces some challenges. The "variational gap" between the true IB objective and its practical approximation can lead to missed or shifted transitions in high-dimensional spaces. Furthermore, the current practical algorithm relies on discretizing the latent space $Z$. Future work aims to extend these insights to fully continuous generative models and regression tasks, potentially providing a deeper understanding of how complex neural networks organize information during training.
Summary updated