Data Consistency Approach to Model Validation

Andreas Lindholm, Dave Zachariah, Petre Stoica, Thomas B. Schön

Introduction

In modern science and engineering, statistical models serve as the primary lens through which we interpret complex datasets. However, a fundamental challenge persists: while tools like the Akaike Information Criterion (AIC) excel at model comparison—identifying which of several candidates is the "best"—they often fail to address model validation. A model may be the best among its peers yet still fundamentally inconsistent with the observed data. Traditional validation tests, such as the Kolmogorov-Smirnov test, are frequently limited to specific distributions and do not scale well to high-dimensional or non-linear time-series data.

To bridge this gap, researchers have introduced the Data Consistency Check (DCC). This "plug-and-play" framework provides an absolute measure of whether a model’s internal logic aligns with reality, regardless of the data's complexity.

The Data Consistency Check (DCC) Framework

The core philosophy of DCC is that a valid model should generate synthetic data that is statistically indistinguishable from the observed data. Rather than comparing raw data points, which is difficult in high dimensions, DCC operates in the likelihood space.

Methodology

  1. Likelihood as a Universal Metric: DCC calculates the log-likelihood of the observed data under the model and compares it to the log-likelihoods of data generated by the model itself. This transformation allows the method to handle univariate, multivariate, and time-series data within a single mathematical framework.
  2. Dispersion Statistic ($T$): The method defines a statistic to measure the variance or dispersion of these log-likelihoods. If the observed data's likelihoods are significantly more or less spread out than the model's self-generated benchmarks, the model is flagged as inconsistent.
  3. Automatic Benchmarking: Because the test uses the model’s own probability density function to create the comparison baseline, it eliminates the need for manual setup or distribution-specific assumptions.

Visualizing Model Consistency

The power of DCC lies in its ability to detect when a model fails to capture the underlying structure of the data. This is often visualized by comparing the distribution of observed data points against the model's expected contours.

Figure 1: An example of an inconsistent model. The observed data (red triangles) is tightly clustered and fails to follow the broader distribution of the model-generated data (green circles).
Figure 2: An example of a consistent model. Here, the observed data (red triangles) is distributed across the space in a manner that aligns with the model's generated samples (green circles).

Empirical Validation and Results

The authors demonstrated the versatility of DCC across several domains:

  • Seismology: Testing the USGS Earthquake Catalog, DCC confirmed that while a Poisson process accurately models massive earthquakes (magnitude $\ge 7$), it fails for smaller ones due to clustering effects (aftershocks). A Negative Binomial distribution was found to be more consistent across all scales.
  • Ecology: The method was applied to a complex bi-variate time-series of kangaroo populations in Australia. DCC successfully validated a latent-variable model involving stochastic differential equations, a task typically too complex for standard closed-form validation tests.
  • Synthetic Benchmarks: In controlled tests, DCC matched or exceeded the performance of specialized classical tests (like Jarque-Bera for normality) while remaining a general-purpose tool.

Conclusion and Future Directions

The Data Consistency Check offers a robust, automated alternative to traditional validation methods. By focusing on likelihood dispersion, it identifies both under-dispersion (models that are too simple) and over-dispersion (models that overfit noise). While computationally intensive due to its reliance on Monte Carlo simulations, DCC provides a critical safeguard against drawing conclusions from fundamentally flawed models. Future work aims to integrate DCC into standard statistical software as a routine diagnostic for researchers across disciplines.

Summary updated

Abstract

In scientific inference problems, the underlying statistical modeling assumptions have a crucial impact on the end results. There exist, however, only a few automatic means for validating these fundamental modeling assumptions. The contribution in this paper is a general criterion to evaluate the consistency of a set of statistical models with respect to observed data. This is achieved by automatically gauging the models' ability to generate data that is similar to the observed data. Importantly, the criterion follows from the model class itself and is therefore directly applicable to a broad range of inference problems with varying data types, ranging from independent univariate data to high-dimensional time-series. The proposed data consistency criterion is illustrated, evaluated, and compared with several well-established methods using three synthetic and two real data sets.
BibTeX
@article{lindholm2019data,
  author = {Andreas Lindholm and Dave Zachariah and Petre Stoica and Thomas B. Schön},
  title = {{Data Consistency Approach to Model Validation}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2915109},
}
Figure