Scaling Mutual Information Estimation to High Dimensions

399 concepts308 papers1 proposal

Information theory offers a beautiful, unified language for deep learning. It allows us to formalize the Information Bottleneck principle (Yury Polyanskiy et al., The Information Bottleneck Problem and its Applications in Machine Learning), prove generalization bounds (Xuetong Wu et al., Fast Rate Generalization Error Bounds: Variations on a Theme), and enforce feature independence (Hyunjik Kim et al., Disentangling by Factorising). Yet, when we try to compute these metrics on modern neural networks, we hit a fundamental wall: high-dimensional intractability.

The Core Obstacle: Why High Dimensions Break Information Estimation

Mutual information $I(X;Y) = H(X) - H(X|Y)$ measures how much knowing one variable reduces uncertainty about another. To compute it exactly, we must know the underlying probability density functions of our data or network states.

In high dimensions—such as the weight space of a network with millions of parameters, or the activation maps of deep layers—traditional non-parametric density estimation suffers from the curse of dimensionality. The number of samples required to accurately map the space scales exponentially with the number of dimensions.

To bypass this, researchers often turn to neural-network-based variational estimators. However, as demonstrated in Understanding the Limitations of Variational Mutual Information Estimators, these estimators frequently fail to satisfy basic mathematical properties like additivity and tend to saturate as the true mutual information grows. This leaves us unable to reliably measure information flow in large-scale architectures.

To make information theory practical for modern deep learning, researchers have converged on two primary strategies: shifting the evaluation space and learned low-dimensional projections.

Strategy 1: Shifting from Weights to Losses and Predictions

If calculating the mutual information between a high-dimensional weight space $\mathbf{W}$ and the training data is impossible, can we measure something simpler that captures the same dynamics?

Instead of looking at the coordinates of the weights themselves, we can look at what those weights produce. This insight has transformed the study of generalization bounds.

[High-Dimensional Weight Space (Millions of Dimensions)] 
                       │
                       ▼ (Map via Learning Algorithm)
[One-Dimensional Loss / Prediction Space (Tractable)]
                       │
                       ▼ 
         (Compute Mutual Information)

Hrayr Harutyunyan et al. introduced this paradigm shift with Functional Conditional Mutual Information ($f$-CMI) in Information-theoretic generalization bounds for black-box learning algorithms. Rather than tracking the information retained in the massive weight space, $f$-CMI measures the information shared between the training data and the model's low-dimensional predictions.

Building on this, Fredrik Hellström et al. proposed evaluated CMI (e-CMI) in A New Family of Generalization Bounds Using Samplewise Evaluated CMI. They proved that by tracking the information encoded in one-dimensional scalar losses rather than the parameters themselves, you can derive tight, non-vacuous generalization bounds for deep networks.

This approach was simplified even further by Yuxin Dong et al. in Rethinking Information-theoretic Generalization: Loss Entropy Induced PAC Bounds, who replaced high-dimensional mutual information entirely with the entropy of the loss distribution. Because the loss is a simple one-dimensional variable, its entropy can be estimated stably and efficiently, even when the model completely interpolates the training data.

Strategy 2: Compression and Learned Low-Dimensional Representations

When the goal is to regularize or understand the hidden features of a network directly, we cannot simply look at the final loss. We must estimate information metrics on the high-dimensional layers themselves. The solution here is to compress the data into a lower-dimensional subspace where information estimation becomes tractable, while ensuring that no statistical dependence is lost in the process.

Gokul Gowri et al. developed Latent Mutual Information (LMI) in Approximating mutual information of high-dimensional variables using learned representations. LMI uses auxiliary neural networks to map thousands of dimensions down to a compact latent space. Crucially, the mapping is optimized to be "mutually informative," meaning it explicitly preserves the statistical dependencies of the original high-dimensional variables while shedding irrelevant noise.

Alternatively, we can use geometric and mathematical corrections on simpler projections. Kristjan Greenewald et al. demonstrated in High-Dimensional Smoothed Entropy Estimation via Dimensionality Reduction that projecting data onto a low-dimensional subspace via Principal Component Analysis (PCA), combined with a specific analytical correction factor, allows for the accurate estimation of "smoothed" entropy. This technique bypasses the curse of dimensionality entirely, providing a scalable way to validate theories like the Information Bottleneck hypothesis across deep layers.

Unlocking Practical Applications

By making high-dimensional information estimation computationally feasible, these frameworks allow researchers to deploy information-theoretic objectives directly into training loops and system designs:

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 312 papers
  2. Version 1 · 2026-10-04 · 516 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_high_dimensional_mutual_information_esti_ba39ea83,
  title = {high-dimensional mutual information estimation (high-dimensional information estimation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-dimensional-mutual-information-estimation-high-dimensional-information/rdr_ba39ea83b4494c648958aad7598a054d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_high_dimensional_mutual_information_esti_ba39ea83_v2,
  title = {high-dimensional mutual information estimation (high-dimensional information estimation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-dimensional-mutual-information-estimation-high-dimensional-information/rdr_ba39ea83b4494c648958aad7598a054d/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake