Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning

Andreas Kirsch, Clare Lyle, Yarin Gal

Abstract

The Information Bottleneck principle offers both a mechanism to explain how deep neural networks train and generalize, as well as a regularized objective with which to train models. However, multiple competing objectives are proposed in the literature, and the information-theoretic quantities used in these objectives are difficult to compute for large deep neural networks, which in turn limits their use as a training objective. In this work, we review these quantities and compare and unify previously proposed objectives, which allows us to develop surrogate objectives more friendly to optimization without relying on cumbersome tools such as density estimation. We find that these surrogate objectives allow us to apply the information bottleneck to modern neural network architectures. We demonstrate our insights on MNIST, CIFAR-10 and Imagenette with modern DNN architectures (ResNets).

Introduction: The Practical Challenge of Information Bottlenecks

In deep learning, the goal is to develop models that are not only accurate but also generalize robustly to unseen data. The Information Bottleneck (IB) principle provides a theoretical framework for this, suggesting an optimal trade-off: a model should extract all information from the input $X$ that is relevant to the label $Y$ (maximizing accuracy) while simultaneously discarding "nuisance" information from $X$ that is irrelevant to the task (maximizing compression).

Despite its theoretical elegance, the IB principle has been notoriously difficult to implement in practice. Calculating Mutual Information in high-dimensional spaces, such as natural images, typically requires computationally expensive and often unstable tools like density estimation or generative models. Consequently, IB has largely remained a tool for analyzing small-scale "toy" models. This work bridges the gap between theory and practice by introducing tractable "surrogate" objectives that behave like IB but are as easy to optimize as standard weight decay.

Methods: Tractable Surrogates and the Role of Noise

The researchers unified several variations of the IB objective by decomposing them into two core components: Decoder Uncertainty ($H[Y|Z]$), which measures label information loss in the latent representation $Z$, and a Regularization Term, which acts as the bottleneck forcing compression.

Key Innovations in Optimization

  • Dropout as an IB Approximation: The authors demonstrate that the standard cross-entropy loss used with Dropout effectively approximates the "Decoder Uncertainty" term, suggesting that modern architectures already incorporate elements of IB.
  • The Necessity of Noise: A critical finding is that noise is a mathematical requirement, not just a stochastic trick. Without it, the latent space can use "infinite precision" to bypass the bottleneck. Adding Gaussian noise "floors" the entropy, ensuring stable training.
  • Simple Surrogate Regularizers: Instead of complex estimators, the paper proposes three simple proxies for the bottleneck: penalizing latent code variance (log-variance), penalizing variance relative to the class label (conditional log-variance), and penalizing the squared norm of noisy feature vectors (Activation $L_2$ regularizer).
Figure 5: The impact of noise on training error (left) and a comparison of how different surrogate regularizers achieve the information bottleneck (right).

Key Insights: Visualizing Information Flow

The paper introduces a conceptual "I-diagram" (often called the "Mickey Mouse" diagram due to its shape) to visualize the overlap between relevant, redundant, and residual information. This provides a mental model for how a bottleneck filters information. Furthermore, the authors successfully generated "Information Plane" plots for modern architectures like ResNets on complex datasets like CIFAR-10 and Imagenette, showing the distinct expansion and compression phases of learning that were previously difficult to observe in large-scale models.

Figure 3: The Information Plane (left) showing learning trajectories across epochs and the 'I-diagram' (right) visualizing the decomposition of information-theoretic quantities.

Results and Robustness

The validation across MNIST, CIFAR-10, and Imagenette demonstrated that these techniques scale effectively. Key findings include:

  • Scaling to Modern Architectures: The surrogate objectives allowed for the first clear visualization of the Information Plane in ResNet18 models.
  • Adversarial Robustness: Models trained with the $L_2$ surrogate objective exhibited significantly higher resistance to adversarial attacks (e.g., FGSM, PGD) compared to those trained with standard weight decay. By forcing the model to ignore "nuisance" pixels, it naturally becomes less susceptible to small, irrelevant perturbations.

Conclusion and Future Directions

While the surrogate objectives are highly effective, the mapping between the practical balancing parameter ($\gamma$) and the theoretical Lagrange multiplier ($\beta$) is not always direct. Future work may explore the constraints imposed by network capacity and the deep connections between IB and Bayesian Neural Networks to create even more robust, uncertainty-aware models.

Summary updated

Abstract

The Information Bottleneck principle offers both a mechanism to explain how deep neural networks train and generalize, as well as a regularized objective with which to train models. However, multiple competing objectives are proposed in the literature, and the information-theoretic quantities used in these objectives are difficult to compute for large deep neural networks, which in turn limits their use as a training objective. In this work, we review these quantities and compare and unify previously proposed objectives, which allows us to develop surrogate objectives more friendly to optimization without relying on cumbersome tools such as density estimation. We find that these surrogate objectives allow us to apply the information bottleneck to modern neural network architectures. We demonstrate our insights on MNIST, CIFAR-10 and Imagenette with modern DNN architectures (ResNets).
BibTeX
@misc{kirsch2020unpacking,
  author = {Andreas Kirsch and Clare Lyle and Yarin Gal},
  title = {{Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning}},
  year = {2020},
  eprint = {2003.12537},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.12537},
}
Figure