Goodness of fit in nonlinear dynamics: Misspecified rates or misspecified states?

Giles Hooker, Stephen P. EllnerThe Annals of Applied Statistics 2015 · DOI · arXiv

When a mathematical model fails to match experimental data, researchers often find themselves at a frustrating crossroads. Is the data simply noisy? Is the functional form of the growth rate slightly off? Or is there an entirely different, "hidden" variable at play that was never measured?

In their research, Giles Hooker and Stephen P. Ellner provide a statistical "diagnostic kit" to answer these questions. Rather than merely indicating that a model is incorrect, they offer a methodology to determine how it is incorrect, distinguishing between exogenous noise, incorrect rate equations, and missing state variables.

The Hierarchy of Model Failure

Ordinary Differential Equations (ODEs) are the workhorse of modeling in fields like biology and engineering. However, when a proposed model $\dot{x} = f(x, \theta)$ fails to fit the data, standard goodness-of-fit tests often offer no direction for improvement. The authors categorize model failure into a three-level hierarchy:

  1. Exogenous Noise: The model is structurally correct, but the system is buffeted by random, external disturbances.
  2. Misspecified Rates: The variables are correct, but the functional form of the interaction ($f$) is wrong.
  3. Misspecified States: The model is missing a critical variable (e.g., measuring a predator and prey but omitting a second prey species that alters the dynamics).

Methodology: The Forcing Function

The core innovation lies in introducing an empirical "forcing function," $g(t)$, which represents the lack of fit. The model is modified to $\dot{x} = f(x, \theta) + g(t)$. By treating $g(t)$ as a surrogate for what is missing, the authors can analyze its behavior to pinpoint the source of error.

Figure 2: A comprehensive diagnostic overview showing raw data, ODE solutions, and the relationship between the forcing function p(t) and state variables for ecological chemostat data.

To analyze this forcing function, the authors employ several advanced statistical techniques:

  • Gradient Matching: A two-stage estimation technique that allows for the estimation of parameters and the forcing function without the massive computational overhead of solving the ODE repeatedly.
  • The "Case 2" Test: This checks if $g(t)$ depends on the current state $x(t)$. If a strong correlation exists, the rate function $f$ is likely misspecified.
  • The "Case 3" Test: Inspired by Takens’ Embedding Theorem, this checks if $g(t)$ depends on its own past values. If the past helps predict the present lack-of-fit better than the current state does, it indicates a missing state variable.

Diagnostic Insights

The most elegant aspect of this approach is treating the lack-of-fit function as a "ghost" of the missing dynamics. If $g(t)$ has its own internal dynamics, it behaves like the missing variable itself. This is often visible in the geometry of the failure; for instance, in a 2D system, if trajectories appear to "cross" each other, it is a mathematical certainty that a third dimension (a missing state) is required.

Figure 1: A diagnostic plot for rate misspecification (Case 2), comparing estimated derivatives (dx) against the model's predicted rate function f(x).

Key Results and Applications

The authors validated their approach using both synthetic and real-world data, including the Rössler system and ecological chemostat data involving algae and rotifers.

  • High Power in Simulations: The tests were nearly 100% accurate in identifying misspecified rates in the van der Pol system and showed high power (up to 91.5%) in detecting missing variables in chaotic systems.
  • Chemostat Mystery: On real ecological data, the tests found strong evidence that the rate equations were wrong ($p \approx 0.05$). However, they had lower power to detect missing algal subpopulations, likely because the system's dynamics were too constrained to reveal the hidden states clearly.
  • Heart Dynamics: Analysis of ECG data successfully identified that standard second-order models were missing both the correct functional form and additional state variables, confirming the complex "bumps" observed in high-frequency heart rhythm data.

Conclusion

While the tests are somewhat conservative to avoid false alarms, they provide a rigorous framework for model improvement. The authors suggest that future work should focus on experimental designs—such as perturbing the system—to make these hidden variables more "observable" to statistical diagnostics. By moving beyond simple "pass/fail" goodness-of-fit tests, this research allows scientists to debug their dynamics and build more accurate representations of complex systems.

Summary updated

Abstract

This paper introduces diagnostic tests for the nature of lack of fit in ordinary differential equation models (ODEs) proposed for data. We present a hierarchy of three possible sources of lack of fit: unaccounted-for stochastic variation, misspecification of functional forms in rate equations, and omission of dynamic variables in the description of the system. We represent lack of fit by allowing a parameter vector to vary over time, and propose generic testing procedures that do not rely on specific alternative models. Instead, different sources for lack of fit are characterized in terms of nonparametric relationships among latent variables. The tests are carried out through a combination of residual bootstrap and permutation methods. We demonstrate the effectiveness of these tests on simulated data and on real data from laboratory ecological experiments and electro-cardiogram data.
BibTeX
@article{hooker2015goodness,
  author = {Giles Hooker and Stephen P. Ellner},
  title = {{Goodness of fit in nonlinear dynamics: Misspecified rates or misspecified states?}},
  journal = {The Annals of Applied Statistics},
  year = {2015},
  doi = {10.1214/15-aoas828},
}
Figure