TL;DR
This paper explores how the inherent approximations in mathematical models, particularly "sloppy" models common in systems biology, can limit the effectiveness of optimal experimental design. The authors introduce a framework to quantify model discrepancy and show that while optimal experiments can constrain parameters, they can paradoxically reduce a model's predictive power by revealing previously hidden systematic errors.
What Problem Does This Paper Address?
Building accurate mathematical models of complex biological systems is challenging because our understanding is incomplete. Models are always approximations, and it's often unclear which details are truly important for explaining observed behavior. This leads to "sloppy" models, which have many parameters that are difficult to uniquely determine from data. While optimal experimental design aims to improve parameter estimation by selecting informative experiments, this paper investigates a counter-intuitive problem: what happens when these "optimal" experiments, designed to make parameters identifiable, reveal that the model itself is fundamentally flawed? Specifically, the authors address the question of whether improving parameter identifiability through experimental design can inadvertently worsen a model's predictive power if the model's underlying approximations are too significant.
Methods
The authors introduce a "hyper-model" to quantify "model discrepancy" or "model error." This approach extends the standard least-squares regression framework by adding a term that accounts for systematic errors arising from model approximations. They derive a criterion for acceptable model error based on the estimated size of this systematic error relative to experimental noise. The Fisher Information Matrix (FIM) is used throughout to analyze parameter identifiability and the "sloppiness" of models, characterized by a wide range of FIM eigenvalues. They also employ information geometry to conceptualize model manifolds and their widths, relating them to parameter relevance.
Datasets
The paper uses two primary case studies:
- EGFR Signaling: Data from existing experiments on EGFR signaling (Brown et al.) and simulated data generated from a more mechanistic model (a "surrogate for reality") based on the Apgar et al. proposed optimal experiments.
- DNA Repair: Experimental data from a study on jejunal crypt clonogen survival after radiotherapy, using models that combine linear-quadratic (LQ), lethal-potentially lethal (LPL), and repair-saturation (RS) models.
Key Insights and Ideas
- Sloppy Systems: The concept extends "sloppy models" to encompass a physical system and its experimental protocols, where a hierarchy of models of increasing complexity become sloppy in the limit of microscopic accuracy.
- Optimal Design Can Reveal Model Flaws: Experiments designed to improve parameter identifiability can inadvertently highlight significant systematic errors in the model if the model omits crucial mechanisms.
- Model Discrepancy vs. Parameter Uncertainty: When a model has large systematic errors (high model discrepancy), improving parameter estimates (reducing parameter uncertainty) may not improve predictive power and can even decrease it.
- Predictivity Can Be Lost in Identifiability: The paper demonstrates that in some cases, models are more predictive with unconstrained parameters when fit to a few experiments than after being fit to optimally selected experiments that reveal model shortcomings.
- Marginal Parameters: Sloppy systems often have "marginal parameters" that are not tightly constrained but are necessary for explaining behavior. Attempting to constrain these can reduce model accuracy.
Key Results
- EGFR Signaling: Applying optimal experimental design to an approximate EGFR model, even with simulated data from a more realistic model, revealed that the approximate model's predictive power was significantly reduced. The model error (quantified by 'f') became large, indicating a poor fit.
- DNA Repair: Similarly, fitting the DNA repair model to expanded experimental data (35 experiments) resulted in a large model error (f=2.0), indicating that the model lacked relevant mechanisms and was no longer predictive of observed asymmetries in dose response.
- Hyper-model Effectiveness: The proposed hyper-model provided a quantitative measure of model error, showing that when this error is large, the model's predictive capability is compromised.
Limitations and Future Work
- Hyper-model Simplifications: The authors acknowledge that their hyper-model makes simplifying assumptions, such as uncorrelated model errors. More sophisticated hyper-models could be explored.
- Identifying Missing Mechanisms: While the paper shows that model failure can guide the identification of missing mechanisms, it doesn't provide a direct method for discovering these mechanisms.
- Model Reduction: The authors suggest that removing sloppy parameters and using model reduction techniques could lead to simpler, more predictive models.
- Hierarchy of Models: They advocate for considering a hierarchy of models of varying detail rather than focusing solely on parameter estimation in a single model. This approach can provide richer insights into system behavior.
Summary updated