Information Compression and Generalization in Deep Networks

25 concepts13 papers

Deep neural networks are often described as "black boxes," but the Information Bottleneck (IB) theory offers a compelling narrative for what happens inside them. The theory suggests that learning consists of two distinct phases: a "fitting" phase where the network learns to predict the labels, and a "compression" phase where it systematically discards irrelevant input information. By shedding these noisy details, the network supposedly gains the ability to generalize to new, unseen data.

However, as researchers have peered deeper into this bottleneck, the relationship between compression and generalization has proven to be far more complex than a simple two-phase story.

The Measurement Challenge

To study the Information Bottleneck, researchers must measure Mutual Information (MI)—a quantity that captures how much information one variable (like a hidden layer) contains about another (like the input image). In deterministic neural networks, this is notoriously difficult because MI can technically be infinite.

Goldfeld et al. argue that many previous observations of "compression" were actually artifacts of how researchers estimated these values. They propose that what looks like information loss is often a geometric phenomenon: the clustering of internal representations. Similarly, Adilova and Geiger show that by adding continuous noise (like Gaussian dropout), we can make these information measures finite and reliable, providing a clearer view of how networks actually handle data.

Is Compression Universal?

A central debate in the field is whether compression is a fundamental requirement for learning or just a byproduct of specific settings. Early IB studies focused on networks with "saturating" activation functions like tanh or sigmoid, which naturally squash data.

Saxe et al. challenged the universality of the IB theory by showing that networks using ReLU (Rectified Linear Unit) activations—the standard in modern AI—often do not exhibit a global compression phase. In these models, the hidden layers can retain high mutual information with the input throughout the entire training process while still achieving excellent generalization.

However, the story isn't that simple. Houghton et al. found that even ReLU networks can discard irrelevant information, suggesting that compression isn't just a quirk of saturating functions but a general potential of the learning process. Lorenzen et al. used Quantization Aware Training to calculate exact MI, confirming that while compression is real, it is highly sensitive to the specific architecture and activation functions used.

The Geometry of Compression

If compression isn't always about "forgetting bits," what is it doing? Recent work suggests it might be about the spatial organization of data.

Rangamani et al. describe a "hunchback" dimensionality profile in deep networks. The data first expands into high dimensions in the early layers to learn complex features, then progressively "collapses" into structured clusters in the deeper layers. This "Intermediate Neural Collapse" suggests that compression is the process of squeezing different examples of the same class into the same point in space.

This transition between expansion and compression is not fixed. Tan et al. found that hyperparameters like learning rate and batch size act as a dial: large learning rates drive the network to explore more complex, expanded representations, while smaller rates lead to simpler, more compressed ones.

The most provocative finding across these studies is the decoupling of compression from generalization. While the Information Bottleneck theory posits that compression causes generalization, empirical evidence remains elusive.

Manoel et al. and Gabrié et al. used tools from statistical physics to show that while compression frequently occurs, its functional link to a model's performance on new data is not direct. A network can reach peak accuracy long before any compression begins, and some models generalize perfectly without ever undergoing a "simplifying phase" Lee and Jo.

As Polyanskiy and Goldfeld summarize, the Information Bottleneck remains a powerful framework for understanding the trade-offs in learning, but it may not be the universal law of generalization it was once thought to be. Whether compression is a necessary driver of intelligence or merely a common byproduct of how we optimize neural networks remains one of the most important open questions in deep learning theory.

Key Foundational Work

Entropy and mutual information in models of deep neural networks (2018) — contributed 3 concept elements to this direction.
Estimating Information Flow in Deep Neural Networks (2019) — contributed 3 concept elements to this direction.
Entropy and mutual information in models of deep neural networks (2018) — contributed 3 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_compression_generalization_decoupling_eba168dd,
  title = {compression-generalization decoupling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compression-generalization-decoupling/rdr_eba168dd6512497fa9c5ce89e6dbb68c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_compression_generalization_decoupling_eba168dd_v1,
  title = {compression-generalization decoupling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compression-generalization-decoupling/rdr_eba168dd6512497fa9c5ce89e6dbb68c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}

See something wrong? Report a mistake