When we try to reverse-engineer a complex system—whether it is a gene regulatory network or a decaying quantum state—we usually start by collecting data and fitting a model. The standard approach is to maximize the likelihood of the observed data. However, this requires an accurate "noise model": you must know exactly how your measurements are distorted by the environment. If your noise model is wrong, your parameter estimates will be biased.
Research by Gurinder S. Atwal and Justin B. Kinney suggests a more robust alternative: maximizing Mutual Information (MI). In the limit of large datasets, maximizing MI allows you to reconstruct a system's internal filters without needing to know the specific noise characteristics of your instruments. But this robustness comes with a fundamental trade-off: certain aspects of the system become mathematically "invisible."
The Invariance of Information
The core strength of information-theoretic inference is its invariance. Mutual information is preserved under any invertible transformation. If you have a set of predictions and you pass them through a monotonic function (like scaling them by two or taking their logarithm), the mutual information between those predictions and the ground truth remains exactly the same.
As Atwal and Kinney demonstrate in Parametric inference in the large data limit using maximally informative models, this property makes MI-based inference immune to "nuisance" noise. You don't need to know if your measurement device is linear or logarithmic; the MI objective will find the same optimal internal parameters regardless.
Diffeomorphic Modes: The Blind Spots
However, this same invariance creates "blind spots" in the parameter space. Atwal and Kinney term these directions diffeomorphic modes.
A diffeomorphic mode is a direction in a model's parameter space where changes to the parameters result only in invertible transformations of the internal representation. Because these transformations preserve the rank order of predictions, they do not change the mutual information.
In Learning Quantitative Sequence–Function Relationships from Massively Parallel Experiments, the authors explain that these modes are analogous to Goldstone modes in physics—symmetries that prevent the unique identification of certain physical scales or additive shifts. If you can slide along a parameter direction without changing the information content of your model, that parameter is "unconstrained." You might be able to tell that one input is twice as "strong" as another, but you cannot determine the absolute physical magnitude of either without an explicit noise model or a specific nonlinearity to break the symmetry.
Breaking Symmetries to Recover Physics
This problem of identifiability isn't limited to biology. In quantum mechanics, researchers face similar hurdles when trying to learn the dynamics of dissipation. WU Ya explores this in Learning quantum dissipation by the neural ordinary differential equation.
When the initial state of a system and the measurement probes share the same physical symmetries as the system's dynamics, the resulting data is often too "simple" to resolve the underlying parameters. Multiple different mathematical models will fit the data perfectly, leading to degenerate solutions.
To solve this, WU Ya suggests a strategy of symmetry breaking:
- Diverse Initial States: Using randomized or asymmetric initial states to "poke" the system from different angles.
- Diverse Measurement Bases: Ensuring the probes do not align perfectly with the system's inherent symmetries.
By intentionally breaking these symmetries, the researcher ensures that the data contains enough information to uniquely resolve the full physics of the system, effectively "pinning down" the diffeomorphic modes that would otherwise remain unconstrained.
Summary
Information-theoretic inference offers a powerful, noise-agnostic way to learn how systems process information. However, the very thing that makes it robust—its invariance to invertible transformations—creates a geometry in parameter space where certain directions (diffeomorphic modes) are unobservable. Understanding these symmetries is essential: they tell us which parts of a model are truly determined by the data and which parts are merely arbitrary choices of scale or representation.