Information Theoretic Generalization and the Bottleneck Principle

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2516 papers565 conceptsfrozen 2026-10-04

In the quest to understand why deep neural networks generalize, researchers have increasingly turned to the language of Information Theory. The core intuition is simple: a model that "memorizes" its training data too precisely will fail to generalize to new examples. Conversely, a model that "distills" the input—stripping away irrelevant details while retaining only what is necessary to predict the target—is likely to be more robust.

This perspective treats a neural network as a communication channel, where the goal is to optimize the flow of information through the architecture.

The Information Bottleneck

The foundational framework for this direction is the Information Bottleneck (IB) principle, pioneered by Naftali Tishby and Noga Zaslavsky in Deep learning and the information bottleneck principle.

They proposed that deep learning consists of two distinct phases:

  1. Fitting: The network learns to represent the input $X$ to predict the label $Y$.
  2. Compression: The network systematically "forgets" information about $X$ that is irrelevant to $Y$.

Mathematically, this is a trade-off between two mutual information terms: $I(X; Z)$ (the information the internal representation $Z$ keeps about the input) and $I(Z; Y)$ (the information $Z$ provides about the label). A "good" model minimizes $I(X; Z)$ while maximizing $I(Z; Y)$. This "bottleneck" forces the network to find the most efficient, compressed representation possible.

Generalization as Information Minimization

Why does compression lead to better performance on unseen data? Aolin Xu and Maxim Raginsky provided a rigorous answer in Information-theoretic analysis of generalization capability of learning algorithms. They proved that the generalization error—the gap between training and test performance—is upper-bounded by the mutual information between the training dataset and the learned model weights.

In this view, overfitting is literally "too much information." If the weights of a network contain a high degree of information about the specific training samples, they have effectively memorized the noise or "accidents" of that dataset. By restricting this information flow, we force the model to learn general patterns.

Recent work has refined these bounds to be even tighter. For example, Gholamali Aminian and colleagues in An Exact Characterization of the Generalization Error for the Gibbs Algorithm moved beyond upper bounds to provide exact formulas for specific learning algorithms using "symmetrized KL information."

From Representations to Weights

While early work focused on the information in a network's representations (the activations), newer research suggests the information stored in the weights is a more universal measure of complexity.

Xi Chen and Zifeng Wang introduced the PAC-Bayes Information Bottleneck (PIB) in PAC-Bayes Information Bottleneck. By quantifying the information stored in the weights, they created a complexity measure that works across different architectures and activation functions, often outperforming traditional regularization like weight decay or Dropout.

The Challenge of High-Dimensional Estimation

If information theory is so powerful, why isn't it used in every training loop? The primary obstacle is computational intractability.

Calculating mutual information in high-dimensional, continuous spaces—like a layer with thousands of neurons—is notoriously difficult. As noted by Hao Cheng in Evaluating Capability of Deep Neural Networks for Image Classification via Information Plane, standard estimators often fail or become prohibitively expensive when applied to modern convolutional layers.

To solve this, researchers are developing "tractable proxies":

Practical Applications: Pruning and Robustness

This theoretical framework is yielding practical tools for network design. In An Information Theory-inspired Strategy for Automatic Network Pruning, Xiawu Zheng and colleagues introduced ITPruner. Instead of expensive trial-and-error, they use the Hilbert-Schmidt Independence Criterion (HSIC) to identify and remove redundant layers in seconds, treating pruning as a principled information-reduction problem.

Similarly, the information-theoretic lens helps explain adversarial vulnerability. Jiebao Zhang found that robust models are essentially more "selective" in the information they extract, focusing on stable shape-based features rather than brittle image textures (Exploring Adversarial Examples and Adversarial Robustness of Convolutional Neural Networks by Mutual Information).

By viewing deep learning as a process of information distillation, researchers are moving toward a "grand unified theory" that explains not just that neural networks work, but exactly how much they need to forget to succeed.

Papers in this version

Showing 60 of 516 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 312 papers
  2. Version 1 · 2026-10-04 · 516 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_high_dimensional_mutual_information_esti_ba39ea83,
  title = {high-dimensional mutual information estimation (high-dimensional information estimation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-dimensional-mutual-information-estimation-high-dimensional-information/rdr_ba39ea83b4494c648958aad7598a054d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_information_theoretic_generalization_and_ba39ea83_v1,
  title = {Information Theoretic Generalization and the Bottleneck Principle},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/information-theoretic-generalization-and-the-bottleneck-principle/rdr_ba39ea83b4494c648958aad7598a054d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}