Automatic cross-validation in structured models: Is it time to leave out leave-one-out?

Aritz Adín, Elias Teixeira Krainski, Amanda Lenzi, Zhiyuan Liu, Joaquín Martínez‐Minaya, Håvard RueEdinburgh 2023 · DOI · arXiv

Abstract

Standard techniques such as leave-one-out cross-validation (LOOCV) might not be suitable for evaluating the predictive performance of models incorporating structured random effects. In such cases, the correlation between the training and test sets could have a notable impact on the model's prediction error. To overcome this issue, an automatic group construction procedure for leave-group-out cross validation (LGOCV) has recently emerged as a valuable tool for enhancing predictive performance measurement in structured models. This paper compares LOOCV and LGOCV within structured models, emphasizing model selection and predictive performance, and provides real data applications in spatial statistics using complex structured models fitted with INLA.

Introduction

In many statistical models, we assume data points are independent. In these cases, LOOCV works effectively: one point is hidden, the model is trained on the remainder, and the prediction error is calculated. However, in structured models—such as those used in spatial mapping or time-series forecasting—data points are inherently correlated.

If a single data point is removed from a spatial map, its immediate neighbors still retain significant information about it. The model can use this neighboring information to "cheat," resulting in a low prediction error that does not reflect how the model would perform in a truly unobserved region. This "information leakage" makes standard metrics like LOOCV, DIC (Deviance Information Criterion), and WAIC (Watanabe-Akaike Information Criterion) over-optimistic, often leading researchers to choose models that excel at interpolation but fail at extrapolation.

Methods: Automatic LGOCV within INLA

The authors leverage a recently proposed automatic group construction procedure for LGOCV within the Integrated Nested Laplace Approximation (INLA) framework. Key aspects of this method include:

  • Automatic Grouping: Instead of manually defining "neighborhoods," the algorithm uses the correlation matrix of the latent Gaussian field to identify the most informative data points for a specific target. It then excludes this entire "group" from the training set.
  • Level Sets ($m$): A parameter $m$ defines the "depth" of the group to be excluded. A higher $m$ implies a more challenging extrapolation task.
  • Efficiency: By working within the latent Gaussian model (LGM) framework, these computations can be performed in seconds using fast approximations, avoiding the high costs usually associated with group cross-validation.

Simulation Study: Temporal Forecasting

To demonstrate the effectiveness of LGOCV, the paper compares an AR1 (autoregressive) model against a simple linear model for temperature forecasting.

The AR1 model (red) captures non-linearities better than the linear model (blue) for interpolation tasks.

As shown in the figure above, the AR1 model appears superior when fitting existing data (interpolation) because it can track local fluctuations more closely. Standard LOOCV and DIC scores typically favor the AR1 model for this reason.

Model Selection and the Extrapolation Challenge

The true utility of LGOCV is revealed when shifting from interpolation to extrapolation. While the AR1 model is excellent at "filling in" missing points, its predictive power decays rapidly when forecasting into the future.

Forecasting comparison: the linear model (left) maintains the trend while the AR1 model (right) reverts to the mean, demonstrating the importance of LGOCV for extrapolation.

In this forecasting task, the AR1 effect quickly decays to the mean, leading to poor long-term predictions. LGOCV correctly identifies that the linear model with a covariate is superior for extrapolation, whereas LOOCV would erroneously suggest the AR1 model is better. This highlights the "Aha" moment in model selection: choosing a model based on its ability to handle unobserved regions or future time points rather than its ability to smooth over existing data.

Real-World Applications

The paper validates the LGOCV method across several complex datasets:

  • Spatial Disease Mapping: In a study of dowry deaths in Uttar Pradesh, India, LGOCV revealed that P-spline models performed better at predicting risks in unobserved districts than the CAR (Conditional Autoregressive) models favored by DIC.
  • Spatio-temporal Health Data: For pancreatic cancer mortality in England, LGOCV identified a shared-component multivariate model as the most robust choice for predicting mortality based on incidence data.
  • Geospatial Wind Speed: LGOCV confirmed that a non-separable space-time model provided the best predictive performance for UK wind data, despite the dataset's high complexity.

Conclusion

For structured models, standard leave-one-out techniques provide a false sense of security regarding predictive accuracy. By accounting for the dependencies inherent in spatial and temporal data, automatic LGOCV provides a more realistic measure of performance. The paper concludes that it is indeed time to move beyond LOOCV in favor of group-based techniques that better reflect the challenges of real-world extrapolation.

Summary updated

Abstract

Standard techniques such as leave-one-out cross-validation (LOOCV) might not be suitable for evaluating the predictive performance of models incorporating structured random effects. In such cases, the correlation between the training and test sets could have a notable impact on the model's prediction error. To overcome this issue, an automatic group construction procedure for leave-group-out cross validation (LGOCV) has recently emerged as a valuable tool for enhancing predictive performance measurement in structured models. The purpose of this paper is (i) to compare LOOCV and LGOCV within structured models, emphasizing model selection and predictive performance, and (ii) to provide real data applications in spatial statistics using complex structured models fitted with INLA, showcasing the utility of the automatic LGOCV method. First, we briefly review the key aspects of the recently proposed LGOCV method for automatic group construction in latent Gaussian models. We also demonstrate the effectiveness of this method for selecting the model with the highest predictive performance by simulating extrapolation tasks in both temporal and spatial data analyses. Finally, we provide insights into the effectiveness of the LGOCV method in modelling complex structured data, encompassing spatio-temporal multivariate count data, spatial compositional data, and spatio-temporal geospatial data.
BibTeX
@inproceedings{adin2023automatic,
  author = {Aritz Adín and Elias Teixeira Krainski and Amanda Lenzi and Zhiyuan Liu and Joaquín Martínez‐Minaya and Håvard Rue},
  title = {{Automatic cross-validation in structured models: Is it time to leave out leave-one-out?}},
  booktitle = {Edinburgh},
  year = {2023},
  doi = {10.48550/arxiv.2311.17100},
}
Figure