Mutual Information Singularities in Deterministic Deep Learning

25 concepts23 papers1 proposal

Information theory offers a natural language for deep learning: we want models to "compress" irrelevant noise while "retaining" essential information about the task. However, a fundamental mathematical trap exists at the heart of this analogy. In standard neural networks, the mappings between layers and the optimization process itself are often entirely deterministic. This determinism causes classical information-theoretic measures to collapse into "singularities"—becoming infinite, undefined, or vacuous.

The Determinism Trap

The core of the problem lies in how Shannon Mutual Information (MI) behaves when randomness disappears. MI measures the reduction in uncertainty about one variable given another. If a hidden layer $Z$ is a deterministic function of the input $X$, then $Z$ contains no "new" randomness. In continuous spaces, the probability density of $Z$ becomes a singular "delta mass" supported on a low-dimensional manifold.

As Achille et al. and Goldfeld et al. observe, this makes the mutual information $I(X; Z)$ technically infinite. When researchers attempt to estimate this information by binning or discretizing the activations, they often aren't measuring information flow at all. Instead, Goldfeld et al. argue that these estimates act as unintended proxies for the geometric clustering of representations—how tightly the data points are packed in the latent space—rather than true Shannon information.

Why Vacuous Bounds Matter

This mathematical degeneracy isn't just a theoretical nuisance; it breaks the tools we use to understand generalization. A common goal in learning theory is to bound the "generalization gap" (the difference between training and test error) using the information the model weights $W$ carry about the training data $S$.

Standard information-theoretic bounds typically take the form $\text{Gap} \le \sqrt{I(W; S) / n}$. As Hellström et al. and Clerico et al. point out, if the learning algorithm is deterministic (like SGD with a fixed seed), the mutual information $I(W; S)$ becomes infinite. The resulting bound tells us only that the error is less than infinity—a "vacuous" statement that provides no insight into why the model generalizes. This "determinism trap" extends to generative models as well; Chen et al. show that when encoders or generators in VAEs and Diffusion Models become perfectly deterministic, the mutual information terms used to analyze them encounter similar singularities.

Strategies for Finite Information

To bypass these singularities, researchers have developed several distinct strategies to restore the utility of information theory in deterministic settings.

1. Injecting Virtual or Real Noise

The most direct solution is to re-introduce stochasticity. Pensia et al. show that generalization bounds require continuous noise to remain finite and informative. Gergely Neu extends this by analyzing a "virtual noise" framework for SGD, where the path taken by the optimizer is treated as a stochastic process, allowing for finite bounds based on gradient variance and loss flatness.

2. Shifting the Source of Randomness

Instead of assuming the model itself is noisy, one can look at the randomness inherent in the data distribution. Balestriero et al. and the authors apply this to Self-Supervised Learning (SSL). By treating the input augmentations as the source of entropy, they show that methods like VICReg can be rigorously understood as maximizing representation entropy even when the network architecture is entirely deterministic.

3. Alternative Dependence Measures

If Shannon MI is too fragile, we can replace it with more robust metrics.

  • Jensen-Shannon Information: Aminian et al. propose using JS-information, which is guaranteed to be finite even for deterministic mappings.
  • Wasserstein Distance: Rodríguez Gálvez et al. move toward optimal transport, using Wasserstein distance to bound generalization in a way that handles deterministic point-mass distributions.
  • Kernelized Rényi Entropy: Gong et al. utilize kernel-based entropy measures that are dimension-independent and remain well-defined for standard SGD.

4. Functional and Compression Perspectives

Rather than looking at the information in the internal weights, we can look at the information in the model's outputs. Harutyunyan et al. introduce "Functional Conditional Mutual Information" ($f$-CMI), which measures how much information the model's predictions carry about the training labels. This functional approach remains finite for deterministic algorithms and tracks the actual performance gap of deep networks more closely than weight-based measures.

Alternatively, Sefidgaran et al. suggest moving away from Shannon MI entirely in favor of Minimum Description Length (MDL). By measuring the "compressibility" of the learning process rather than the raw information flow, they derive non-vacuous bounds that apply to deterministic encoders and continuous parameter spaces.

Key Foundational Work

Generalization Error Bounds for Noisy, Iterative Algorithms (2018) — contributed 2 concept elements to this direction.
Where is the Information in a Deep Neural Network? (2019) — contributed 1 concept elements to this direction.
Estimating Information Flow in Deep Neural Networks (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_mutual_information_singularity_786ca8b3,
  title = {mutual information singularity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mutual-information-singularity/rdr_786ca8b37e01447b9366285d20b111f6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_mutual_information_singularity_786ca8b3_v1,
  title = {mutual information singularity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mutual-information-singularity/rdr_786ca8b37e01447b9366285d20b111f6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake