Abstract
In scientific inference problems, the underlying statistical modeling assumptions have a crucial impact on the end results. There exist, however, only a few automatic means for validating these fundamental modelling assumptions. The contribution in this paper is a general criterion to evaluate the consistency of a set of statistical models with respect to observed data. This is achieved by automatically gauging the models' ability to generate data that is similar to the observed data. Importantly, the criterion follows from the model class itself and is therefore directly applicable to a broad range of inference problems with varying data types, ranging from independent univariate data to high-dimensional time-series. The proposed data consistency criterion is illustrated, evaluated and compared to several well-established methods using three synthetic and two real data sets.
Introduction
In the fields of statistics and machine learning, researchers typically operate under the assumption of a specific "model class"—for instance, assuming data follows a Gaussian distribution or a linear regression pattern. Standard practice often focuses on model selection, utilizing metrics like the Akaike Information Criterion (AIC) or Bayesian Information Criterion (BIC) to identify the "best" model within a predefined set.
However, a significant gap exists: these tools are designed to compare models against each other, but they do not inherently determine if the entire set of models is fundamentally flawed. Even the "best" selected model might be entirely inconsistent with the actual observed data. While specialized tests like Kolmogorov-Smirnov or Ljung-Box exist, they are often restricted to specific data types. The Data Consistency Criterion (DCC) introduced here provides an automatic, general-purpose framework for model validation that transcends these limitations.
The Data Consistency Criterion (DCC)
The fundamental intuition behind DCC is straightforward: if a statistical model is an accurate representation of the real world, then the observed data should not appear "atypical" when compared to data simulated from that model.
Methodology
The DCC framework relies on a few key components:
- The Test Statistic ($T$): Instead of manually defining a discrepancy measure, DCC uses the log-likelihood of the data points. It calculates how much the observed log-likelihood deviates from the expected log-likelihood under the model.
- Handling Dependency: For complex data like time-series, the method utilizes incremental likelihoods, $p(y_i | y_1, \dots, y_{i-1}, \theta)$, to account for temporal correlations.
- False Alarm Probability (PFA): The core metric is the PFA, which represents the probability that the model would generate a statistic $T$ more extreme than the one observed. A PFA near 0 indicates under-dispersion (data is "too perfect"), while a PFA near 1 indicates over-dispersion (data contains unexpected outliers or variance).
To handle unknown parameters $\theta$, DCC averages the PFA across the model class using weights derived from the likelihood, effectively integrating out parameter uncertainty.
Experimental Evaluation
The authors validated DCC across various scenarios, demonstrating its robustness and sensitivity.
Synthetic Benchmarks
In experiments comparing Gaussian and Uniform distributions, DCC proved highly sensitive to the tails of the distribution. As sample sizes increased, it correctly rejected Gaussian models when the underlying data was uniform, a task where traditional selection methods might struggle if the "best fit" Gaussian still looks superficially plausible.
Real-World Applications
- Seismology: Analyzing global earthquake data, DCC revealed that simple Poisson models are consistent for large earthquakes ($\ge 7$ magnitude) but fail for smaller ones due to clustering effects. A Negative Binomial model was found to be more consistent.
- Ecology: The criterion was successfully applied to complex latent-variable models for kangaroo populations, showing it can work even when likelihoods must be approximated via particle filters.
Discussion and Conclusion
The primary advantage of DCC is its "automatic" nature—the validation criterion is derived directly from the model class itself, removing the need for subjective test statistics. It serves as a "misspecification test" that can reject all models in a set if none are consistent with the data.
While the Monte Carlo implementation can be computationally intensive for high-dimensional data, DCC offers a rigorous path toward ensuring that scientific conclusions are grounded in statistically sound models. It bridges the gap between frequentist sampling properties and Bayesian parameter weighting, providing a versatile tool for modern data science.
Summary updated