In the quest to understand why deep neural networks generalize, researchers have increasingly turned to the language of Information Theory. The core intuition is simple: a model that "memorizes" its training data too precisely will fail to generalize to new examples. Conversely, a model that "distills" the input—stripping away irrelevant details while retaining only what is necessary to predict the target—is likely to be more robust.
This perspective treats a neural network as a communication channel, where the goal is to optimize the flow of information through the architecture.
The Information Bottleneck
The foundational framework for this direction is the Information Bottleneck (IB) principle, pioneered by Naftali Tishby and Noga Zaslavsky in Deep learning and the information bottleneck principle.
They proposed that deep learning consists of two distinct phases:
- Fitting: The network learns to represent the input $X$ to predict the label $Y$.
- Compression: The network systematically "forgets" information about $X$ that is irrelevant to $Y$.
Mathematically, this is a trade-off between two mutual information terms: $I(X; Z)$ (the information the internal representation $Z$ keeps about the input) and $I(Z; Y)$ (the information $Z$ provides about the label). A "good" model minimizes $I(X; Z)$ while maximizing $I(Z; Y)$. This "bottleneck" forces the network to find the most efficient, compressed representation possible.
Generalization as Information Minimization
Why does compression lead to better performance on unseen data? Aolin Xu and Maxim Raginsky provided a rigorous answer in Information-theoretic analysis of generalization capability of learning algorithms. They proved that the generalization error—the gap between training and test performance—is upper-bounded by the mutual information between the training dataset and the learned model weights.
In this view, overfitting is literally "too much information." If the weights of a network contain a high degree of information about the specific training samples, they have effectively memorized the noise or "accidents" of that dataset. By restricting this information flow, we force the model to learn general patterns.
Recent work has refined these bounds to be even tighter. For example, Gholamali Aminian and colleagues in An Exact Characterization of the Generalization Error for the Gibbs Algorithm moved beyond upper bounds to provide exact formulas for specific learning algorithms using "symmetrized KL information."
From Representations to Weights
While early work focused on the information in a network's representations (the activations), newer research suggests the information stored in the weights is a more universal measure of complexity.
Xi Chen and Zifeng Wang introduced the PAC-Bayes Information Bottleneck (PIB) in PAC-Bayes Information Bottleneck. By quantifying the information stored in the weights, they created a complexity measure that works across different architectures and activation functions, often outperforming traditional regularization like weight decay or Dropout.
The Challenge of High-Dimensional Estimation
If information theory is so powerful, why isn't it used in every training loop? The primary obstacle is computational intractability.
Calculating mutual information in high-dimensional, continuous spaces—like a layer with thousands of neurons—is notoriously difficult. As noted by Hao Cheng in Evaluating Capability of Deep Neural Networks for Image Classification via Information Plane, standard estimators often fail or become prohibitively expensive when applied to modern convolutional layers.
To solve this, researchers are developing "tractable proxies":
- Variational Bounds: Andriy Serdega and Dae-Shik Kim use variational methods in VMI-VAE to maximize mutual information between data and latent codes without direct density estimation.
- Coding Rate: Junzhou Huang and Long-Kai Huang developed TransRate (Frustratingly Easy Transferability Estimation), a simple metric that uses the "coding rate" from information theory to predict how well a model will transfer to a new task in just a few lines of code.
- Maximum Entropy: Long Zhao and others use the Maximum Entropy Principle to generate "harder" adversarial examples that improve robustness (Maximum-Entropy Adversarial Data Augmentation).
Practical Applications: Pruning and Robustness
This theoretical framework is yielding practical tools for network design. In An Information Theory-inspired Strategy for Automatic Network Pruning, Xiawu Zheng and colleagues introduced ITPruner. Instead of expensive trial-and-error, they use the Hilbert-Schmidt Independence Criterion (HSIC) to identify and remove redundant layers in seconds, treating pruning as a principled information-reduction problem.
Similarly, the information-theoretic lens helps explain adversarial vulnerability. Jiebao Zhang found that robust models are essentially more "selective" in the information they extract, focusing on stable shape-based features rather than brittle image textures (Exploring Adversarial Examples and Adversarial Robustness of Convolutional Neural Networks by Mutual Information).
By viewing deep learning as a process of information distillation, researchers are moving toward a "grand unified theory" that explains not just that neural networks work, but exactly how much they need to forget to succeed.