Optimal Ridge Regularization for Out-of-Distribution Prediction

Pratik Patil, Jin-Hong Du, Ryan J. Tibshirani

TL;DR

This paper reveals that for ridge regression in high-dimensional settings, the "optimal" penalty parameter ($\lambda$) can often be zero or even negative, especially when the test data distribution differs from the training data (Out-of-Distribution). Crucially, the authors prove that if you tune this regularization perfectly—even allowing for negative values—the prediction risk becomes a smooth, monotonic curve, effectively "curing" the infamous double descent phenomenon.

What Problem Does This Paper Address?

In classical statistics, we are taught that regularization (like Ridge or Lasso) should always be positive to prevent overfitting. However, modern machine learning often operates in the "overparameterized" regime, where the number of features ($p$) exceeds the number of observations ($n$). In these settings, researchers have observed strange behaviors like "double descent," where risk spikes and then drops as models get larger.

The authors tackle two fundamental questions:

  1. (Q1) The Sign of $\lambda$: Under what conditions is the optimal ridge penalty positive, zero, or negative?
  2. (Q2) Risk Monotonicity: Does the prediction risk always improve with more data if we tune the model optimally?

They specifically look at these questions through the lens of Out-of-Distribution (OOD) prediction, where the test data might have different feature correlations (covariate shift) or a different relationship between features and labels (regression shift) compared to the training data.

Methods

The researchers use high-dimensional asymptotics, a framework where both the number of samples ($n$) and features ($p$) grow to infinity, but their ratio ($p/n = \phi$) remains constant. This allows them to derive "deterministic equivalents"—mathematical formulas that precisely describe how the model will behave on average.

Key technical contributions include:

  • General Alignment Conditions: They derive conditions based on the alignment between the signal (the true $\beta$), the noise, and the feature covariance ($\Sigma$) that determine the sign of the optimal $\lambda$.
  • Negative Regularization Analysis: Unlike most studies that restrict $\lambda \ge 0$, this paper explores the full range of mathematically valid negative values (down to a specific lower bound $\lambda_{min}$ where the ridge solution is still well-defined).
  • Subsampling Equivalences: They link ridge regularization to "ensembling" (averaging models trained on subsets of data), showing that ensembling provides a form of implicit positive regularization.

Datasets

The paper validates its theoretical findings using:

  • Synthetic Data: Simulated datasets with varying signal-to-noise ratios (SNR), feature correlations, and distribution shifts.
  • MNIST: The classic handwritten digit dataset. To simulate OOD shifts, they "exclude" certain digit classes from the test set (e.g., training on all digits but testing only on a subset) to create varying degrees of covariate and regression shifts.

Key Insights and Ideas

  • Negative $\lambda$ is a Feature, Not a Bug: Negative regularization can be optimal when the "implicit bias" of the model is too high. Essentially, if the data structure already provides too much "self-regularization," a negative external $\lambda$ can "subtract" some of that bias to improve performance.
  • OOD Shifts Drive Negative Tuning: Distribution shifts make negative $\lambda$ much more common. For example, under a "regression shift," the optimal $\lambda$ can be negative even in the "underparameterized" regime ($n > p$), which is highly counter-intuitive to classical theory.
  • Tuning "Cures" Double Descent: The "double descent" spike typically happens at the interpolation boundary ($p \approx n$) when using a fixed or suboptimal $\lambda$. The authors prove that the optimally tuned risk is always monotonic. If you have more data, your best possible risk will be lower, provided you tune $\lambda$ (potentially into the negative range) correctly.
  • Alignment Matters: The "sign" of the optimal penalty depends on how the signal vector aligns with the eigenvectors of the covariance matrix. If the signal is concentrated in directions where the features have low variance, you might need negative $\lambda$ to "boost" those directions.

Key Results

  • Theorem 4 & 5: These provide the exact mathematical conditions for when $\lambda^*$ flips from positive to negative under covariate and regression shifts.
  • Theorem 7 (The Monotonicity Result): This is a major theoretical result showing that the OOD risk of an optimally tuned ridge predictor is strictly increasing in the aspect ratio $\phi = p/n$ (meaning it's strictly decreasing in the number of samples $n$).
  • MNIST Evidence: In their MNIST experiments (Table 2), as the distribution shift became more severe, the optimal $\lambda$ moved from $1.03$ (positive) to $-0.48$ (negative).

Limitations and Future Work

  • Model Scope: The analysis is strictly for Ridge Regression. While the authors mention that similar patterns likely exist for the Lasso or Kernel Ridge Regression, those require separate, complex proofs.
  • Linearity: The results assume a linear projection framework. While this covers a wide range of "feature map" models (like Random Fourier Features), it doesn't directly model deep neural networks with learned features.
  • Computational Tuning: The paper focuses on the existence of an optimal $\lambda$ (an "oracle" view). In practice, finding a negative $\lambda$ via cross-validation is tricky because the risk surface can become non-convex or unstable near the $\lambda_{min}$ boundary.

The authors suggest that extending these monotonicity results to other "penalized M-estimators" (like the Lasso) is the next logical step in understanding why overparameterized models generalize so well.

Summary updated

BibTeX
@inproceedings{patil2024optimal,
  author = {Pratik Patil and Jin-Hong Du and Ryan J. Tibshirani},
  title = {{Optimal Ridge Regularization for Out-of-Distribution Prediction}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=bvPYroQgc3},
}
Figure