The Mixtures and the Neural Critics: On the Pointwise Mutual Information Profiles of Fine Distributions

Pawel Czyz, Frederic Grabowski, Julia E. Vogt, Niko Beerenwinkel, Alexander Marx

Abstract

The pointwise mutual information profile, or simply profile, is the distribution of pointwise mutual information for a given pair of random variables. One of its important properties is that its expected value is precisely the mutual information between these random variables. In this paper, we analytically describe the profiles of multivariate normal distributions and introduce a novel family of distributions, Bend and Mix Models, for which the profile can be accurately estimated using Monte Carlo methods. We then show how Bend and Mix Models can be used to study the limitations of existing mutual information estimators, investigate the behavior of neural critics used in variational estimators, and understand the effect of experimental outliers on mutual information estimation. Finally, we show how Bend and Mix Models can be used to obtain model-based Bayesian estimates of mutual information, suitable for problems with available domain expertise in which uncertainty quantification is necessary.

Introduction: Beyond a Single Number

Mutual Information (MI) is a fundamental metric in information theory, quantifying the total dependence between two variables. However, reducing complex dependencies to a single scalar value often obscures the underlying structure of the relationship. This paper advocates for a more granular perspective: the Pointwise Mutual Information (PMI) profile. By examining the full distribution of information values rather than just their average, researchers can gain deeper insights into the behavior of MI estimators and the nature of the data they analyze.

The PMI Profile and Bend and Mix Models (BMMs)

The PMI profile is defined as the distribution of the random variable $T = \log \frac{p(X,Y)}{p(X)p(Y)}$. While the expectation of $T$ yields the MI, the profile itself captures the "shape" of the information distribution. A critical property of the PMI profile is its invariance under smooth, invertible transformations (diffeomorphisms). This means that "bending" a distribution does not change its informational characteristics, even if it significantly alters its spatial representation.

To leverage this, the authors introduce Bend and Mix Models (BMMs). These models are constructed by applying diffeomorphisms to base distributions and combining them into mixtures. Because BMMs have tractable densities, ground truth MI and PMI profiles can be calculated to high precision using Monte Carlo methods, providing a rigorous "boss-level" testing ground for estimators.

The PMI profile remains invariant under smooth transformations (blue), while changing the underlying mixture structure (green) creates a distinct profile.

Benchmarking Mutual Information Estimators

The authors established a comprehensive benchmark featuring 26 continuous distributions, including complex cases like "Waves" (a mixture of twelve multivariate normals) and "Galaxy" (a spiral-transformed mixture). They evaluated five prominent estimators: histogram-based, KSG (neighborhood-based), CCA (canonical correlation), and two neural variational estimators (InfoNCE and MINE).

The results highlight significant gaps in current technology. While the neighborhood-based KSG estimator remains a strong baseline for low-dimensional tasks, it—and all other tested estimators—struggled significantly with the "Waves" distribution. This failure underscores the need for more robust estimation techniques for highly structured data.

Complex BMM-generated distributions like 'Waves' and 'Galaxy' reveal the performance gaps in standard MI estimators.

Key Insights: Neural Critics and Noise Sensitivity

A fascinating finding concerns the "neural critics" used in variational estimators like MINE. While these networks often fail to learn the exact PMI function across the entire domain, they are remarkably effective at capturing the overall PMI profile. This explains their ability to estimate total MI accurately even when their internal representations are imperfect.

Furthermore, the study investigated the impact of noise. By simulating "failing channels" with BMMs, the authors demonstrated that "inlier noise"—noise that overlaps with the signal distribution—is far more detrimental to MI estimation than "outlier noise" (extreme values).

Bayesian MI Estimation

One of the most promising applications of BMMs is Bayesian MI estimation. By fitting a BMM to data using Markov Chain Monte Carlo (MCMC) methods, such as the NUTS sampler, researchers can obtain a posterior distribution of MI. This allows for actual uncertainty quantification, providing credible intervals rather than just point estimates. This approach is particularly valuable in fields where domain expertise can guide the model structure and where understanding the reliability of an estimate is crucial.

Limitations and Future Work

BMMs are not "black-box" solutions; they require domain knowledge to specify an appropriate generative model. Misspecification can lead to significant bias. Additionally, the computational cost of MCMC limits scalability to very high-dimensional data. Future research may explore integrating Normalizing Flows into the BMM framework to handle more complex dependencies more efficiently.

Summary updated

Abstract

The pointwise mutual information profile, or simply profile, is the distribution of pointwise mutual information for a given pair of random variables. One of its important properties is that its expected value is precisely the mutual information between these random variables. In this paper, we analytically describe the profiles of multivariate normal distributions and introduce a novel family of distributions, Bend and Mix Models, for which the profile can be accurately estimated using Monte Carlo methods. We then show how Bend and Mix Models can be used to study the limitations of existing mutual information estimators, investigate the behavior of neural critics used in variational estimators, and understand the effect of experimental outliers on mutual information estimation. Finally, we show how Bend and Mix Models can be used to obtain model-based Bayesian estimates of mutual information, suitable for problems with available domain expertise in which uncertainty quantification is necessary.
BibTeX
@inproceedings{czyz2024mixtures,
  author = {Pawel Czyz and Frederic Grabowski and Julia E. Vogt and Niko Beerenwinkel and Alexander Marx},
  title = {{The Mixtures and the Neural Critics: On the Pointwise Mutual Information Profiles of Fine Distributions}},
  booktitle = {AABI},
  year = {2024},
  url = {https://openreview.net/forum?id=cRtZ4sx3cU},
}
Figure