Mutual Information (MI) is a foundational metric for quantifying non-linear dependencies between random variables. Unlike simple linear correlation, MI captures the total information shared between variables, making it highly valuable for feature selection, representation learning, and multi-modal data fusion.
However, calculating Shannon's mutual information requires knowing or estimating the underlying joint and marginal probability density functions (PDFs). In low dimensions, we can easily discretize the space into histograms or use kernel density estimation. But as the number of dimensions grows, the number of required bins scales exponentially—a classic manifestation of the curse of dimensionality. As noted by Yang et al., evaluating joint information across multiple features quickly becomes statistically and computationally intractable due to this exponential cell growth. In high-dimensional spaces, finite samples become sparse, causing traditional estimators to suffer from extreme bias and high variance, or fail entirely due to the "density-chasm problem" where distributions have negligible overlap (Srivastava et al.).
To make information theory usable under modern computational and sample constraints (Xu et al.), a rich body of research has emerged to bypass explicit high-dimensional density estimation. These approaches can be broadly categorized into variational optimization, geometric slicing, semiparametric copulas, and matrix-based functionals.
Variational Optimization and Neural Estimation
Instead of estimating a probability density and then computing its integral, why not frame mutual information estimation as an optimization problem?
This is the core insight behind Mutual Information Neural Estimation (MINE), introduced by Baratin et al.. By leveraging the Donsker-Varadhan dual representation of the Kullback-Leibler (KL) divergence, the authors show that MI can be computed as the supremum over a class of functions (the "witness functions"):
$$I(X; Y) \ge \mathbb{E}_{P_{XY}}[T(x, y)] - \log(\mathbb{E}_{P_X \otimes P_Y}[e^{T(x, y)}])$$
By parameterizing the witness function $T$ as a deep neural network, we can maximize this lower bound using gradient descent. The neural network acts as a flexible discriminator, learning to distinguish between samples drawn from the joint distribution $P_{XY}$ and the product of marginals $P_X \otimes P_Y$.
This optimization paradigm scales remarkably well to continuous, high-dimensional spaces because it shifts the burden from local density counting to global function approximation. For instance, Deep InfoMax (Hjelm et al.) utilizes this mechanism to maximize the mutual information between local patches of an image and its global representation, enabling powerful unsupervised representation learning without explicit density modeling. Similarly, Garg et al. apply neural dual estimation to transform combinatorial clustering into a continuous divergence maximization problem.
Geometric Slicing and Subspace Projections
Another powerful intuition for beating the curse of dimensionality is geometric: if a problem is too hard in a high-dimensional space, project it down to a lower-dimensional subspace where estimation is easy.
Goldfeld et al. formalized this via $k$-Sliced Mutual Information ($k$-SMI). Instead of computing MI directly on the ambient high-dimensional variables, $k$-SMI takes the average mutual information over all possible $k$-dimensional linear projections of the data. The authors rigorously demonstrate that this approach completely circumvents the curse of dimensionality, achieving a parametric sample complexity rate that does not degrade as the ambient dimension increases.
While $k$-SMI averages over projections, a parallel approach focuses on finding the most informative projection. Tsur et al. introduced Max-Sliced Mutual Information (mSMI), which searches for the specific low-dimensional linear subspace that maximizes the mutual information. This acts as a non-linear generalization of Canonical Correlation Analysis (CCA), capturing complex dependencies while maintaining robust sample efficiency.
Moving beyond linear projections, Gowri et al. proposed Latent Mutual Information (LMI), which uses neural networks to compress high-dimensional data into mutually informative, low-dimensional latent representations before performing estimation. By squeezing the data through a low-dimensional bottleneck, these geometric and projection-based methods ensure that the downstream estimator never encounters the sparse, high-dimensional regimes where traditional methods fail.
Semiparametric Copulas and Matrix-Based Functionals
A third paradigm avoids density estimation by changing the mathematical representation of information itself, either by separating the dependency structure from the marginal distributions or by using kernel-based matrix functionals.
The Copula Trick
The difficulty in high-dimensional estimation often stems from messy, non-Gaussian marginal distributions. However, Sklar's theorem states that any multivariate distribution can be expressed in terms of its marginals and a copula, which captures the pure dependency structure.
Singh et al. introduced Nonparanormal Information Estimation, a semiparametric approach that applies monotonic transformations to make the marginals Gaussian, while using a Gaussian copula to model the joint dependence. This allows them to achieve a highly efficient parametric convergence rate of $O(D^2/n)$, combining the flexibility of nonparametric methods with the dimensional scaling of parametric models. Similarly, Rey et al. leverage copula theory in the Meta-Gaussian Information Bottleneck to derive analytical compression solutions that depend solely on the underlying dependency structure, bypassing the need for stable analytical density functions.
Matrix-Based Entropy
Can we compute information metrics directly from data matrices without ever defining a probability space? Yu et al. and Zhang et al. utilize a matrix-based functional to compute Rényi’s $\alpha$-order entropy. Instead of estimating a PDF, this framework computes entropy directly from the eigenvalues of a normalized kernel matrix constructed from the data samples.
Because the kernel matrix implicitly evaluates similarities in a Reproducing Kernel Hilbert Space (RKHS), it captures non-linear dependencies directly from data samples. This eliminates both data discretization and variational approximations, providing a stable, deterministic signal for feature selection and multi-view learning even when data is scarce. This builds on earlier foundational work like Constrained Covariance (COCO) by Smola et al., which similarly used universal kernels in an RKHS to detect complex dependencies without density estimation.
Sample-Efficient Ensembles and Supervised Meta-Estimation
When structural assumptions (like copulas) or geometric projections are not ideal, researchers have pushed the statistical limits of nonparametric estimation using ensemble methods and supervised meta-learning.
Moon et al. developed ensemble estimators that combine multiple base plug-in estimators (such as $k$-nearest neighbor estimators with different values of $k$) to cancel out the leading bias terms caused by high dimensionality. In subsequent work, Moon et al. introduced GENIE, proving that an ensemble approach can achieve the optimal parametric convergence rate ($1/N$) even when dealing with high-dimensional, mixed discrete-continuous datasets common in genomics.
Finally, a recent paradigm shifts the problem from unsupervised estimation to supervised meta-learning. Gritsai et al. introduced MIST (Mutual Information Estimation via Supervised Training). Instead of deriving an analytical estimator for a specific dataset, they train a neural network on a massive synthetic dataset of diverse distributions where the ground-truth MI is mathematically known. The network learns the mapping from finite data samples to the true MI value, outperforming traditional estimators in low-sample, high-dimensional regimes by effectively "amortizing" the estimation challenge across a wide space of distributions.
Key Foundational Work
Kernel Constrained Covariance for Dependence Measurement (2005) — contributed 1 concept elements to this direction.
Search for Information Bearing Components in Speech (1999) — contributed 1 concept elements to this direction.