Information geometry for multiparameter models: New perspectives on the origin of simplicity

Katherine N. Quinn, Michael C. Abbott, Mark K. Transtrum, Benjamin B. Machta, James P. Sethna

Abstract

Complex models in physics, biology, economics, and engineering are often "sloppy," meaning that the model parameters are not well determined by the model predictions for collective behavior. Many parameter combinations can vary over decades without significant changes in the predictions. This review uses information geometry to explore sloppiness and its deep relation to emergent theories. We introduce the model manifold of predictions, whose coordinates are the model parameters. Its hyperribbon structure explains why only a few parameter combinations matter for the behavior. We review recent rigorous results that connect the hierarchy of hyperribbon widths to approximation theory, and to the smoothness of model predictions under changes of the control variables. We discuss recent geodesic methods to find simpler models on nearby boundaries of the model manifold—emergent theories with fewer parameters that explain the behavior equally well. We discuss a Bayesian prior which optimizes the mutual information between model parameters and experimental data, naturally favoring points on the emergent boundary theories and thus simpler models. Finally, we discuss the way the renormalization group coarse-graining in statistical mechanics introduces a flow of the model manifold, connecting stiff and sloppy directions with relevant and irrelevant eigendirections.

The Problem of Sloppiness

In fields ranging from systems biology to climate science, researchers often employ models with dozens, hundreds, or even millions of parameters. Conventional statistical wisdom suggests these models should overfit and fail to generalize. However, in practice, they often work remarkably well. This phenomenon is known as "sloppiness": the observation that most parameter combinations in these models have almost no effect on the final predictions. This creates a gap between our microscopic understanding (the complex model) and the emergent macroscopic behavior (the simple results).

The Geometry of Predictions: The Model Manifold

To understand why simplicity emerges from complexity, we utilize Information Geometry, treating the space of all possible model predictions as a mathematical surface called a Model Manifold. Parameters $\theta$ map to a point in the prediction space $Y(\theta)$.

Mapping from parameter space to the model manifold in prediction space. The manifold's "hyperribbon" structure explains why only a few parameter combinations dominate the model's behavior.

As shown in the figure above, the model manifold is typically a "hyperribbon"—vastly longer than it is wide, and wider than it is thick. The widths typically decay exponentially (e.g., $1, 10^{-1}, 10^{-2} \dots$), meaning only the first few dimensions (the "stiff" directions) truly matter for the model's behavior. The remaining "sloppy" directions represent parameter combinations that can be changed significantly without altering the predictions.

Methods for Model Reduction

Manifold Boundary Approximation Method (MBAM)

The MBAM is a novel algorithm that simplifies models by following "geodesics" (the shortest paths on the manifold) toward its edges. These edges represent simpler, "effective" theories with fewer parameters. Effective theories, such as Newtonian mechanics, are not just approximations; they are literally the boundaries of more complex manifolds (like General Relativity). By identifying these boundaries, researchers can reduce complex power grid models or biological signaling pathways into identifiable, "human-readable" models that retain predictive accuracy.

Information-Theoretic Priors

The paper introduces the "Projected Maximum Likelihood" prior. Unlike the standard Jeffreys prior—which can be wildly distorting in high dimensions by favoring "thick" but unmeasurable regions—these new priors naturally favor the "edges" of the manifold. This effectively implements Occam’s Razor by favoring simpler models on the boundary that are consistent with the data.

Connections to Physics and Visualization

The authors formally link Information Geometry to the Renormalization Group (RG) in physics. They demonstrate that "coarse-graining" a system is equivalent to a geometric flow that compresses the model manifold along its sloppy (irrelevant) dimensions.

Furthermore, new visualization techniques called Intensive Embeddings (InPCA and isKL) use Minkowski-like geometries to map high-dimensional probabilistic models into 2D or 3D plots. These methods allow researchers to visualize complex phenomena, such as the phase transitions of the Ising model or the fluctuations of the early universe, in a way that preserves both local and global geometric features.

Conclusion and Future Directions

The geometric perspective on sloppiness provides a rigorous foundation for understanding why complex systems often exhibit simple emergent behavior. While computational challenges remain in scaling these methods to million-parameter neural networks, they offer a promising path toward "Explainable AI" and the systematic derivation of effective theories across the sciences. Future work will explore the global topology of these manifolds and the potential for identifying reduced models within the manifold's interior.

Summary updated

Abstract

Complex models in physics, biology, economics, and engineering are often sloppy, meaning that the model parameters are not well determined by the model predictions for collective behavior. Many parameter combinations can vary over decades without significant changes in the predictions. This review uses information geometry to explore sloppiness and its deep relation to emergent theories. We introduce the model manifold of predictions, whose coordinates are the model parameters. Its hyperribbon structure explains why only a few parameter combinations matter for the behavior. We review recent rigorous results that connect the hierarchy of hyperribbon widths to approximation theory, and to the smoothness of model predictions under changes of the control variables. We discuss recent geodesic methods to find simpler models on nearby boundaries of the model manifold -- emergent theories with fewer parameters that explain the behavior equally well. We discuss a Bayesian prior which optimizes the mutual information between model parameters and experimental data, naturally favoring points on the emergent boundary theories and thus simpler models. We introduce a `projected maximum likelihood' prior that efficiently approximates this optimal prior, and contrast both to the poor behavior of the traditional Jeffreys prior. We discuss the way the renormalization group coarse-graining in statistical mechanics introduces a flow of the model manifold, and connect stiff and sloppy directions along the model manifold with relevant and irrelevant eigendirections of the renormalization group. Finally, we discuss recently developed `intensive' embedding methods, allowing one to visualize the predictions of arbitrary probabilistic models as low-dimensional projections of an isometric embedding, and illustrate our method by generating the model manifold of the Ising model.
BibTeX
@inproceedings{quinn2022information,
  author = {Katherine N. Quinn and Michael C. Abbott and Mark K. Transtrum and Benjamin B. Machta and James P. Sethna},
  title = {{Information geometry for multiparameter models: New perspectives on the origin of simplicity}},
  booktitle = {Reports},
  year = {2022},
  doi = {10.1088/1361-6633/aca6f8},
}
Figure