Estimating Functionals of the Out-of-Sample Error Distribution in High-Dimensional Ridge Regression

Pratik Patil, Alessandro Rinaldo, Ryan J. TibshiraniAISTATS 2022 · OpenReview

Introduction

In machine learning and statistics, the Mean Squared Error (MSE) is the standard metric for evaluating model performance. While MSE provides a useful average, it fails to capture the full picture of predictive uncertainty. For high-stakes applications—such as medical diagnosis or financial risk management—understanding the entire distribution of out-of-sample errors is critical. This paper addresses the challenge of estimating this full distribution in the context of high-dimensional ridge regression.

The High-Dimensional Challenge

Traditional statistical methods often falter in "high-dimensional" regimes where the number of features ($p$) is large relative to the number of observations ($n$). In such cases, models tend to overfit the training data. Training errors become overly optimistic, shrinking toward zero as the model interpolates the noise. While cross-validation is a well-established tool for correcting this bias in MSE estimation, its applicability to the entire error distribution in high dimensions was previously unproven.

Distributional Extensions of GCV and LOOCV

The researchers propose extending the "shortcut" formulas for Generalized Cross-Validation (GCV) and Leave-One-Out Cross-Validation (LOOCV). These shortcuts traditionally allow for the calculation of cross-validated MSE without the computational burden of re-training the model $n$ times.

The key insight is that the individual reweighted residuals used in these shortcuts contain enough information to reconstruct the entire out-of-sample error distribution. By treating these adjusted residuals as an empirical distribution, the authors demonstrate that they converge weakly to the true test error distribution. This holds even in the "ridgeless" limit ($p > n$ with zero regularization), where the model perfectly fits the training data.

Figure 4: Comparison of training, test, and GCV-estimated error distributions across low (p/n=0.04), moderate (p/n=0.8), and high (p/n=2) dimensions.

As shown in the figure above, while training errors (red) become increasingly deceptive as the dimensionality ratio $p/n$ grows, the GCV-based estimator (yellow) remains a remarkably accurate proxy for the true out-of-sample distribution (blue).

Estimating Error Functionals

Beyond the raw error distribution, the proposed framework allows for the consistent estimation of various functionals of the error, both linear and nonlinear. This includes absolute error, squared error, and specific quantiles.

Figure 6: Estimation of different error functionals: (a) raw error, (b) absolute error, and (c) squared error. The GCV-estimated distributions and means closely track the test data.

The ability to estimate quantiles is particularly powerful, as it enables the construction of prediction intervals with asymptotically exact coverage. For example, a practitioner can now determine a 95% prediction interval that is mathematically guaranteed to contain the true value 95% of the time, even in overparameterized settings.

Conclusion

This work bridges a significant gap in high-dimensional statistics by moving beyond point estimates of error. By leveraging the distributional properties of GCV and LOOCV shortcuts, it provides a robust toolkit for uncertainty quantification in ridge regression. Future research may extend these distributional shortcuts to other regularized models like Lasso or more complex Kernel Ridge Regression.

Summary updated

Abstract

We study the problem of estimating the distribution of the out-of-sample prediction error associated with ridge regression. In contrast, the traditional object of study is the uncentered second moment of this distribution (the mean squared prediction error), which can be estimated using cross-validation methods. We show that both generalized and leave-one-out cross-validation (GCV and LOOCV) for ridge regression can be suitably extended to estimate the full error distribution. This is still possible in a high-dimensional setting where the ridge regularization parameter is zero. In an asymptotic framework in which the feature dimension and sample size grow proportionally, we prove that almost surely, with respect to the training data, our estimators (extensions of GCV and LOOCV) converge weakly to the true out-of-sample error distribution. This result requires mild assumptions on the response and feature distributions. We also establish a more general result that allows us to estimate certain functionals of the error distribution, both linear and nonlinear. This yields various applications, including consistent estimation of the quantiles of the out-of-sample error distribution, which gives rise to prediction intervals with asymptotically exact coverage conditional on the training data.
BibTeX
@inproceedings{patil2022estimating,
  author = {Pratik Patil and Alessandro Rinaldo and Ryan J. Tibshirani},
  title = {{Estimating Functionals of the Out-of-Sample Error Distribution in High-Dimensional Ridge Regression}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2022},
  url = {https://openreview.net/forum?id=xpFBuU6xnMK},
}
Figure