Renormalized Mutual Information for Extraction of Continuous Features

Leopoldo Sarra, Andrea Aiello, Florian Marquardt

Abstract

We derive a well-defined renormalized version of mutual information that allows to estimate the dependence between continuous random variables in the important case when one is deterministically dependent on the other. This is the situation relevant for feature extraction, where the goal is to produce a low-dimensional effective description of a high-dimensional system. Our approach enables the discovery of collective variables in physical systems, thus adding to the toolbox of artificial scientific discovery, while also aiding the analysis of information flow in artificial neural networks.

Introduction: The Challenge of Continuous Feature Extraction

In the study of complex systems—ranging from fluid dynamics to deep neural networks—scientists seek "collective variables." These are low-dimensional features, such as the center of mass or an order parameter, that distill the essential behavior of a high-dimensional system.

Conceptual mapping of high-dimensional observations to low-dimensional features.

Information Theory offers a natural metric for this task: Mutual Information (MI). Ideally, a feature $y = f(x)$ should maximize the information it retains about the input $x$. However, a significant technical hurdle exists: for continuous variables where $y$ is a deterministic function of $x$, the conditional entropy $H(y|x)$ diverges to negative infinity, causing the standard MI to become infinite. Previous attempts to resolve this often involved adding arbitrary noise or ignoring divergent terms, which typically breaks reparametrization invariance—the requirement that information content remains constant regardless of coordinate systems or units.

Methods: Renormalization and Invariance

The authors propose a renormalization procedure inspired by physical regularization techniques. By introducing an infinitesimal measurement uncertainty $\epsilon$ to the input $x$ and then isolating the finite part of the MI as $\epsilon \to 0$, they derive the Renormalized Mutual Information ($\tilde{I}$):

$$\tilde{I}(x, y) = H(y) - \int dx P_x(x) \ln \sqrt{\det \nabla f(x) \cdot \nabla f(x)}$$

The second term, involving the Jacobian of the feature function $f(x)$, accounts for the geometric stretching or compression of the input space. This ensures the metric remains invariant under non-linear but bijective transformations of the feature. The authors implement $f(x)$ using neural networks, optimizing them via gradient ascent to maximize $\tilde{I}$ in an unsupervised manner.

Empirical Validation and Key Results

The approach was validated across three distinct scenarios: a 2D spiral-shaped distribution, a 1D wave packet on a lattice, and a 60-particle "liquid drop" system.

Visual comparison of feature extraction methods across Spiral, Wave Packet, and Liquid Drop datasets.

Key Insights:

  • Geometric Awareness: The Jacobian term allows the network to recognize that information is not just about the spread of the output, but how sensitively the output reflects changes in the input.
  • Unsupervised Discovery: Without labels, the system naturally discovered fundamental physical concepts. In the liquid drop experiment, the network identified features nearly identical to handcrafted physical descriptors like orientation and deformation, significantly outperforming Principal Component Analysis (PCA).
  • Downstream Utility: Features extracted by maximizing $\tilde{I}$ proved superior for supervised tasks. For instance, predicting the center of a wave packet was far more accurate using $\tilde{I}$-optimized features than those from PCA or standard autoencoders.

Conclusion and Future Directions

Renormalized Mutual Information provides a mathematically rigorous "compass" for navigating high-dimensional data. While currently focused on 1D and 2D features, future work may address the computational cost of Jacobian determinants for higher-dimensional spaces. By ensuring that extracted features are truly informative and independent of coordinate artifacts, this method strengthens the toolkit for both artificial scientific discovery and the interpretability of neural architectures.

Summary updated

Abstract

We derive a well-defined renormalized version of mutual information that allows to estimate the dependence between continuous random variables in the important case when one is deterministically dependent on the other. This is the situation relevant for feature extraction, where the goal is to produce a low-dimensional effective description of a high-dimensional system. Our approach enables the discovery of collective variables in physical systems, thus adding to the toolbox of artificial scientific discovery, while also aiding the analysis of information flow in artificial neural networks.
BibTeX
@article{sarra2021renormalized,
  author = {Leopoldo Sarra and Andrea Aiello and Florian Marquardt},
  title = {{Renormalized Mutual Information for Extraction of Continuous Features}},
  journal = {Physical Review Letters},
  year = {2021},
  url = {https://openreview.net/forum?id=6HbACZSG9V},
}
Figure