Chebyshev approximation and the global geometry of sloppy models

Katherine N. Quinn, Heather Wilber, Alex Townsend, James P. SethnaarXiv 2018 · arXiv

Abstract

Sloppiness is a pervasive empirical phenomenon in high-dimensional nonlinear modeling where model predictions are sensitive to only a few parameter combinations. This paper establishes a formal mathematical foundation for sloppiness by connecting model geometry to Chebyshev approximation theory. We demonstrate that the "hyperribbon" structure of model manifolds—where the manifold is significantly thinner than it is long—is a necessary consequence of the mathematical smoothness (analyticity) of the underlying model functions.

Introduction

In diverse scientific fields, researchers frequently encounter "sloppy models." These models exhibit a characteristic hierarchy of parameter sensitivities: a few "stiff" directions dominate the output behavior, while most other directions have negligible impact. Geometrically, the set of all possible model predictions forms a "model manifold" that resembles a hyperribbon—an object much longer than it is wide, and much wider than it is thick. While widely observed, a universal explanation for this structure has been lacking. This work proves that this geometry arises directly from the analyticity of the model functions.

Methodology: Linking Geometry to Approximation

The authors unify geometric descriptions of sloppiness with classical polynomial approximation theory. The approach treats model predictions as points in an $N$-dimensional prediction space. By approximating model functions using Chebyshev polynomials—which are "near-best" global approximators—the authors show that the expansion coefficients for smooth functions decay rapidly.

Visualization of model predictions, their corresponding manifolds, and the decay of manifold lengths compared to Chebyshev bounds.

By representing the mapping from parameters to predictions as a matrix operation and analyzing its singular values, the authors characterize the distortion of parameter space into prediction space. They derive explicit theorems showing that the model manifold is contained within a hyperellipsoid whose widths decay exponentially (or sub-geometrically) based on the model's smoothness.

Key Results and Verification

The theoretical bounds were evaluated against three representative sloppy models: Exponential Decay, Enzyme Kinetics (Michaelis-Menten), and the SIR Epidemiology Model. Monte Carlo sampling was used to visualize these manifolds and compare them against the derived bounds.

In all cases, the actual model manifolds were found to reside within the predicted hyperellipsoid bounds. A key insight is that sloppiness is a fundamental feature of any model that depends smoothly on its inputs. If a function is analytic, its high-order polynomial components must be small, naturally "squashing" the manifold into a thin ribbon.

Extension of the manifold length analysis to models with two independent variables (t and s), showing the persistence of the hyperribbon structure.

Furthermore, the theory successfully extends to models with multiple experimental conditions (e.g., time and temperature). As shown in Figure 4, the hyperribbon structure persists even when the input space is higher-dimensional, with manifold lengths following a predictable decay rate determined by the "Bernstein ellipse" of the function's analyticity.

Discussion and Limitations

While providing a robust framework, the current bounds assume a bounded parameter space. If parameters are unconstrained, the manifold might not be as strictly contained, although physical constraints usually prevent this in practice. For "non-analytic" models (only $k$-times differentiable), the decay is algebraic rather than exponential, resulting in "fatter" hyperribbons. Future work may address the "curse of dimensionality" for models with many independent variables, although the qualitative phenomenon of sloppiness is expected to hold.

Summary updated

Abstract

Sloppy models are complex nonlinear models with outcomes that are significantly affected by only a small subset of parameter combinations. Despite forming an important universality class and arising frequently in practice, formal and systematic explanations of sloppiness are lacking. By unifying geometric interpretations of sloppiness with Chebyshev approximation theory, we offer such an explanation, and show how sloppiness can be described explicitly in terms of model smoothness. Our approach results in universal bounds on model predictions for classes of smooth models, and our bounds capture global geometric features that are intrinsic to their model manifolds. We illustrate these ideas using three disparate models: exponential decay, reaction rates from an enzyme-catalysed chemical reaction, and an epidemiology model of an infected population.
BibTeX
@misc{quinn2018chebyshev,
  author = {Katherine N. Quinn and Heather Wilber and Alex Townsend and James P. Sethna},
  title = {{Chebyshev approximation and the global geometry of sloppy models}},
  year = {2018},
  eprint = {1809.08280},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1809.08280},
}
Figure