In the quest to understand complex systems—from the swirling patterns of a fluid to the inner workings of a deep neural network—scientists often look for "collective variables." These are low-dimensional "features" (like the center of mass or an order parameter) that capture the essence of a high-dimensional mess.
But how do you mathematically define the "best" feature? A new paper by Sarra, Aiello, and Marquardt introduces Renormalized Mutual Information ($\tilde{I}$), a robust tool that allows us to measure information flow in continuous, deterministic systems without the math blowing up.
TL;DR
Standard Mutual Information fails when one variable is a deterministic function of another, resulting in infinite values. This paper introduces a "renormalized" version that stays finite, remains invariant under coordinate changes, and enables neural networks to discover optimal physical features in an unsupervised way.
What Problem Does This Paper Address?
Information Theory provides a natural candidate for feature extraction: Mutual Information (MI). If we have a high-dimensional input $x$ and a feature $y = f(x)$, MI tells us how much knowing $y$ reduces our uncertainty about $x$.
However, there is a massive technical hurdle. For continuous variables where $y$ is a deterministic function of $x$, the conditional entropy $H(y|x)$ diverges to negative infinity. Consequently, the Mutual Information $I(x, y)$ becomes infinite.
Previous "fixes" usually involved adding random noise to the output $y$ or simply ignoring the problematic term. But these hacks break a fundamental requirement: reparametrization invariance. If you change your units from meters to feet, or apply a non-linear but bijective transformation to your feature, the "amount of information" shouldn't change. Standard fixes fail this test, making them unreliable for optimizing features.
Methods
The authors propose a "renormalization" procedure inspired by physics.
- Regularization: They imagine the input $x$ has a tiny amount of measurement uncertainty (noise) $\epsilon$.
- Expansion: They calculate the MI for this noisy system. As $\epsilon$ gets smaller, the MI diverges logarithmically.
- Subtraction: By subtracting the divergent part that depends only on the noise, they isolate a finite, well-defined quantity: the Renormalized Mutual Information.
The resulting formula is elegant: $$\tilde{I}(x, y) = H(y) - \int dx P_x(x) \ln \sqrt{\det \nabla f(x) \cdot \nabla f(x)}$$
The second term involves the Jacobian of the feature function, which accounts for how the function "stretches" or "compresses" the input space. This term is the "secret sauce" that ensures the metric remains invariant even if you warp the feature's coordinates.
The authors then use Neural Networks to represent the function $f(x)$ and train them to maximize $\tilde{I}$ using gradient ascent.
Datasets
The authors validated their approach on three representative scenarios:
- Spiral-shaped Distribution: A 2D toy dataset used to visualize how the algorithm draws "contours" of information across a non-Gaussian probability cloud.
- Wave Packet: A 1D field on a lattice containing a localized "pulse" at a random position. This simulates signal processing or particle detection.
- Liquid Drop: A many-particle system (60 atoms) forming an elliptical drop. The drop's shape (deformation) and orientation fluctuate, providing a complex test for discovering physical collective variables.
Key Insights and Ideas
- Noise Location Matters: Adding noise to the input (microscopic) variables preserves the geometric structure of the mapping, whereas adding noise to the output (features) destroys the very invariance scientists need.
- The Geometry of Information: The Jacobian term effectively introduces a metric on the input space. It recognizes that "information" isn't just about the spread of the output, but how sensitively that output responds to changes in the input.
- Unsupervised Discovery: The system doesn't need labels. By simply asking "what 1D or 2D summary preserves the most information about the whole system?", the neural network naturally discovers physical concepts like "orientation" or "position."
Key Results
- Superior Feature Quality: In the "Liquid Drop" experiment, the neural network discovered features that were nearly identical to the best handcrafted physical descriptors (deformation and orientation), outperforming standard methods like Principal Component Analysis (PCA).
- Downstream Performance: Features extracted via $\tilde{I}$ were significantly better for supervised tasks. For example, a secondary network could predict the center of a wave packet much more accurately using the $\tilde{I}$-optimized feature than it could using PCA or autoencoder bottlenecks.
- Robustness: Unlike standard autoencoders, which can fail when the dimensionality reduction is extreme, the $\tilde{I}$ approach remains stable because it focuses on information content rather than reconstruction accuracy.
Limitations and Future Work
- High-Dimensional Features: The examples in the paper focus on 1D and 2D features. Calculating the determinant of the Jacobian for very high-dimensional feature spaces ($y \in \mathbb{R}^K$ where $K$ is large) is computationally expensive.
- Metric Assumptions: The renormalization assumes a specific noise structure (usually isotropic) on the input $x$. In systems with mixed units (e.g., positions and momenta), choosing the right "noise metric" requires physical intuition.
- Scaling Up: The authors suggest that for more complex datasets (like images), adversarial training techniques could be used to estimate the entropy term $H(y)$ more efficiently.
This work provides a mathematically sound "compass" for researchers trying to navigate the high-dimensional landscapes of modern science, ensuring that the features we extract are truly informative, not just artifacts of our coordinate systems.
Summary updated