Abstract
This article is an invited discussion of the article by Gronau and Wagenmakers (2018) that can be found at https://dx.doi.org/10.1007/s42113-018-0011-7.
Introduction
In recent years, Bayesian leave-one-out cross-validation (LOO) has become a standard tool for assessing the predictive performance of Bayesian models. However, its utility in model selection has been challenged. In a direct response to Gronau and Wagenmakers (2018), Vehtari and colleagues defend LOO, arguing that criticisms of the method often stem from a misunderstanding of its purpose. Rather than a "black-box" decision rule designed to identify a "true" model, LOO is a robust framework for model criticism and predictive assessment, particularly when the "true" model is not among the candidates being tested.
The M-Open Perspective vs. The M-Closed Assumption
A central point of contention in this debate is the philosophical framework in which model selection occurs. Gronau and Wagenmakers (2018) critique LOO from an "M-closed" perspective—a scenario where the researcher assumes that the true data-generating process is exactly represented by one of the models in the candidate set. From this viewpoint, a selection method is often judged by its "consistency," or its ability to select the true model with a probability of one as the sample size approaches infinity.
Vehtari et al. argue that this perspective is rarely applicable to real-world scientific problems. Instead, they advocate for the "M-open" perspective, which acknowledges that all models are approximations. In an M-open reality, the goal is not to find the "true" model, but to identify which model generalizes best to future data. LOO is specifically designed for this task, measuring the expected log predictive density (ELPD) to evaluate how well a model handles unseen observations.
Moving Beyond Single-Number Selection
The authors argue against using LOO as a simple device that spits out a "yes/no" decision. They propose several best practices to improve the depth of model comparison:
- Stacking of Predictive Distributions: Instead of picking a single "winner," the authors recommend stacking (Yao et al., 2018). This method assigns weights to multiple models to maximize the total expected predictive density, often resulting in better performance than any single model could provide alone.
- Pseudo-BMA+: This is an improvement over standard Bayesian Model Averaging that accounts for the empirical variance of LOO scores, providing a more stable weighting scheme.
- Model Criticism via LOO: Rather than looking only at the total score, researchers should plot individual log-predictive densities for every data point. This allows for the identification of specific observations where a model performs poorly, offering insights into how the model might be improved.
- Diagnostic Tools: The authors highlight the use of $\hat{k}$ diagnostics (from the
loopackage) to identify high-leverage observations that might make the cross-validation estimate unreliable.
Re-evaluating the Beta-Bernoulli Examples
To demonstrate their points, Vehtari et al. re-analyze the Beta-Bernoulli "toy" examples used by Gronau and Wagenmakers. These examples involve simple coin-flip models designed to show that LOO can fail to select a simple null hypothesis even with large amounts of data.
The authors show that this "failure" is actually a misunderstanding of predictive equivalence. When a simple model is nested within a more complex one, and the simple model is "true," the complex model will eventually make predictions identical to the simple one as data increases. LOO correctly identifies that the models have become indistinguishable in their predictive power. In such cases, the choice between models should be based on parsimony or computational efficiency, not just a predictive score. Furthermore, they demonstrate that when stacking is applied to these same examples, it correctly assigns a weight of 1 to the simpler model, resolving the issues raised by the critics.
Key Insights
- Marginal Likelihood vs. LOO: Marginal likelihoods (used in Bayes Factors) can confidently select the "best" model even if that model is a poor fit for the data. LOO is more "honest" in that it reflects the actual predictive quality; if all models are bad, LOO scores will reflect that.
- Variance Matters: LOO scores have inherent randomness. If two models have similar scores within the bounds of their standard error, they should be treated as predictively similar rather than forcing a selection.
- The Role of Parsimony: While LOO focuses on predictive performance, it does not automatically penalize complexity if that complexity does not hurt predictions. For nested models, the authors suggest using "projective prediction" to find the simplest model that retains the predictive power of the full model.
Limitations and Considerations
While defending LOO, the authors acknowledge it is not a silver bullet. Its validity depends on conditional exchangeability—the assumption that the data points are independent and identically distributed. If the data has temporal, spatial, or multilevel structures, standard LOO can be misleading unless the cross-validation strategy is modified (e.g., leave-one-cluster-out). Additionally, LOO can be highly variable in the presence of rare events that are not well-represented in the training data.
Conclusion
Vehtari et al. conclude that the perceived limitations of LOO are largely artifacts of applying M-closed logic to an M-open tool. By incorporating uncertainty, using stacking, and employing diagnostic checks, LOO remains one of the most powerful and transparent tools available for Bayesian model assessment and improvement.
Summary updated