How consistent is my model with the data? Information-Theoretic Model Check

Andreas Svensson, Dave Zachariah, Thomas B. SchönarXiv 2017 · arXiv

Abstract

Determining whether a chosen model class is consistent with observed data is a fundamental challenge in statistical learning. Traditional residual-based whiteness tests often fail for complex nonlinear systems or when evaluating entire model classes rather than single point estimates. We present the Information-Theoretic Model Check (ITMC), a framework that utilizes "surprisal" (self-information) as a universal metric for model consistency. ITMC provides a robust, two-sided p-value test applicable to nonlinear and state-space models, effectively detecting misspecifications that traditional methods overlook.

Introduction: The Limitations of Residual Whiteness

In system identification and statistical modeling, the standard procedure for model validation involves analyzing residuals—the difference between observed data and model predictions. If these residuals behave like white noise, the model is deemed "consistent." However, this approach suffers from two critical drawbacks. First, it typically relies on the Maximum Likelihood Estimate (MLE), ignoring the broader uncertainty within the model class. Second, extending these tests to nonlinear dynamical systems or models with latent variables is mathematically arduous and often impractical.

The core question should not just be "are the errors random?" but rather: If this model class were true, how likely is it to generate data that resembles our observations?

The ITMC Framework

The Information-Theoretic Model Check (ITMC) shifts the focus to surprisal (self-information), defined as $D(y, \theta) = -\ln p(y|\theta)$. Surprisal quantifies how "unexpected" a data sequence is given a specific model parameter $\theta$. Under the "typicality" principle, if a model class is well-specified, the observed data should exhibit a surprisal value within the "typical set" of values generated by that model.

Conceptual mapping from model space to data space typicality.

The ITMC Workflow

  1. Metric Definition: Use surprisal as the primary test statistic.
  2. Two-Sided P-Value: ITMC evaluates whether the observed surprisal falls within the typical range. A low p-value indicates that the observed data is either significantly more or significantly less surprising than what the model class typically produces.
  3. Parameter Averaging: Rather than relying on a single MLE, ITMC averages the p-value over a neighborhood of high-performing models using Fisherian or Bayesian weighting.
  4. Monte Carlo Implementation: The p-values are estimated via Monte Carlo sampling, making the framework computationally tractable for complex likelihood functions.

Experimental Evaluation

The authors validated ITMC using synthetic Auto-Regressive (AR) models and the real-world Cascaded Water Tank benchmark—a classic nonlinear state-space problem.

Key Results

A primary advantage of ITMC is its sensitivity to noise variance. While traditional whiteness tests like the Ljung-Box method often ignore the magnitude of variance, ITMC flags incorrect variance specifications because they lead to atypical surprisal levels.

ITMC application to the cascaded water tank benchmark.

When applied to the water tank dataset, ITMC revealed that even sophisticated physical models based on Bernoulli’s principle were inconsistent with real-world data. As shown in the figure above, the ITMC metric drops significantly over time, suggesting that there are unmodeled dynamics—such as overflow or complex friction—that the model class fails to capture.

Insights and Future Directions

ITMC's reliance on the likelihood function $p(y|\theta)$ makes it naturally extensible to nonlinear state-space models, provided the likelihood can be estimated (e.g., via particle filters). However, this dependency also marks a limitation: the method is currently not directly applicable to "likelihood-free" or black-box models without further adaptation.

Future work will focus on reducing the computational cost of the Monte Carlo simulations and deepening the theoretical understanding of ITMC’s behavior across diverse model architectures. Despite these challenges, ITMC represents a significant step toward a more universal and robust framework for model validation in complex systems.

Summary updated

Abstract

The choice of model class is fundamental in statistical learning and system identification, no matter whether the class is derived from physical principles or is a generic black-box. We develop a method to evaluate the specified model class by assessing its capability of reproducing data that is similar to the observed data record. This model check is based on the information-theoretic properties of models viewed as data generators and is applicable to e.g. sequential data and nonlinear dynamical models. The method can be understood as a specific two-sided posterior predictive test. We apply the information-theoretic model check to both synthetic and real data and compare it with a classical whiteness test.
BibTeX
@misc{svensson2017how,
  author = {Andreas Svensson and Dave Zachariah and Thomas B. Schön},
  title = {{How consistent is my model with the data? Information-Theoretic Model Check}},
  year = {2017},
  eprint = {1712.02675},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1712.02675},
}
Figure