The Limitations of Model-Based Experimental Design and Parameter Estimation in Sloppy Systems

Andrew White, Malachi Tolman, Howard D. Thames, Hubert Rodney Withers, Kathy A. Mason, Mark K. TranstrumPLOS Computational Biology 2016 · DOI

Abstract

Mathematical models of biological systems are frequently "sloppy," characterized by parameters that are poorly constrained by experimental data. While Optimal Experimental Design (OED) is typically employed to resolve this uncertainty, this paper identifies a significant risk: OED can inadvertently expose the systematic errors inherent in model approximations. By forcing a model into regimes where it is most sensitive, researchers may inadvertently destroy its predictive power. This study utilizes information geometry and case studies in EGFR signaling and DNA repair to quantify this trade-off between parameter identifiability and model validity.

Introduction

In systems biology, models of pathways like cell signaling are notoriously difficult to parameterize. These systems are "sloppy," meaning their outputs are sensitive to only a few "stiff" combinations of parameters, while most other "sloppy" directions can vary by orders of magnitude without affecting the model's fit to data. The conventional wisdom suggests using Optimal Experimental Design (OED) to select new experiments that specifically target these unconstrained parameters.

However, OED relies on the assumption that the model is a perfect representation of reality. In practice, all models are approximations that omit microscopic details deemed irrelevant. This research investigates a critical danger: if the experiments chosen to "fix" sloppiness make those omitted details relevant, the resulting data may cause the model to fail entirely.

Methodology

The authors employ several quantitative frameworks to analyze model failure:

  • Fisher Information Matrix (FIM): Used to measure parameter sensitivity. Sloppiness is identified by a wide, hierarchical spread of FIM eigenvalues.
  • Information Geometry: Models are visualized as "manifolds" in data space. Sloppy models possess manifolds with a hierarchy of widths; "thin" directions correspond to parameters that are difficult to identify.
  • Systematic Error Hyper-model: To quantify the discrepancy between a model and reality, the authors introduce a hyper-parameter ($f$) representing systematic error. A value of $f > 1$ indicates that the model's underlying approximations have broken down.
  • Comparative Modeling: Approximate models (e.g., Michaelis-Menten kinetics) are tested against complex "mechanistic" surrogates (mass-action kinetics) to observe how approximations fail under OED-driven data acquisition.

Case Studies

The study evaluates two primary biological systems:

  1. EGFR Signaling: Utilizing the 48-parameter Brown et al. model, the authors created a 70-parameter mechanistic "surrogate reality" to generate synthetic data.
  2. DNA Repair: The study modeled mouse jejunal crypt cell survival post-radiation, comparing the Linear-Quadratic (LQ), Lethal-Potentially Lethal (LPL), and Repair-Saturation (RS) models against experimental split-dose radiation data.

Results and Discussion

The OED Trap in EGFR Signaling

In the EGFR case study, OED successfully reduced parameter uncertainty from factors of a million down to approximately 10%. However, this precision came at a high cost: the systematic error exploded. The best-fit error (sum of squares) jumped from an expected baseline of ~7,000 to over 96,000. The model's predictive power was effectively lost because the OED-selected experiments targeted regimes where the model's structural simplifications were no longer valid.

Breakdown in DNA Repair Models

For the DNA repair models, fitting to the original experimental data yielded a healthy hyper-parameter $f = 0.76$. However, when the models were fit to "optimal" expanded data, $f$ rose to 2.0. This indicates that the models could no longer account for the underlying biology under the specific experimental conditions dictated by OED.

Sloppiness as a Shield

A key insight of this work is that sloppiness may be a beneficial feature rather than a bug. By being insensitive to many parameters, a model remains robust to the microscopic details it omits. Attempting to make every parameter identifiable can lead to a model that fails to fit the data because its structural simplifications are exposed. The authors propose that "sloppiness" is a property of the underlying physical system; a "sloppy system" is one where adding detail merely creates a new hierarchy of sloppiness.

Conclusion

There is a fundamental trade-off between parameter identifiability and model predictivity. Instead of pursuing the "one true model" with perfectly estimated parameters, researchers should embrace a hierarchy of models with varying complexities. Future efforts should focus on developing uncertainty quantification methods that help modelers understand the limits of their approximations before designing expensive experiments that might ultimately invalidate their models.

Summary updated

BibTeX
@article{white2016limitations,
  author = {Andrew White and Malachi Tolman and Howard D. Thames and Hubert Rodney Withers and Kathy A. Mason and Mark K. Transtrum},
  title = {{The Limitations of Model-Based Experimental Design and Parameter Estimation in Sloppy Systems}},
  journal = {PLOS Computational Biology},
  year = {2016},
  doi = {10.1371/journal.pcbi.1005227},
}
Figure