Robust priors for regularized regression

matt jones, Sebastian Bobadilla-Suarez, Matt Jones 0002, Bradley C. LovearXiv 2020 · arXiv

Introduction

In the field of statistical learning, the bias-variance tradeoff represents a fundamental challenge. To prevent models from overfitting to noise—essentially "hallucinating" patterns in limited data—researchers typically employ regularization. The most ubiquitous method, Ridge regression, penalizes the magnitude of model weights, effectively shrinking them toward zero. However, this approach relies on a "zero-association" prior: the assumption that, in the absence of overwhelming evidence, a predictor has no effect.

This paper argues that a prior of zero is often an inefficient starting point. In many real-world domains, humans utilize "fast and frugal" heuristics—simple decision rules that are remarkably robust in low-information environments. By mathematically bridging these simple heuristics with high-performance regression models, the authors propose a framework that maintains accuracy when data is scarce and remains interpretable when data is abundant.

The Heuristic-Prior Framework

The core innovation of this research is a two-step model-fitting procedure that modifies the standard $L_2$ (Ridge) cost function. Instead of penalizing the distance from the origin, the model penalizes the distance from a weight vector derived from a human-inspired heuristic.

1. Defining the Heuristic Prior

First, a weight vector ($w_{prior}$) is calculated using a simple rule of thumb. Common examples include:

  • Tallying: Assigning equal weight to all predictors.
  • Take-the-Best: Relying exclusively on the single most valid predictor.

2. Regularized Regression

This prior is then integrated into a modified Ridge equation: $$\hat{w}_{ridge} = \arg \min_{w} \{ \|y - Xw\|^2_2 + \theta \|w - w_{prior}\|^2_2 \}$$

In this formulation, the parameter $\theta$ serves as a tuning dial. At low values of $\theta$, the model behaves like standard Ordinary Least Squares (OLS), relying heavily on the training data. At high values, the model converges toward the simple heuristic. This allows for a smooth transition from a "novice" state (relying on simple rules) to an "expert" state (incorporating complex data patterns).

Applications Across Domains

The researchers validated this approach across three distinct areas:

  1. Decision Making: Using 20 classical psychological datasets (e.g., predicting city populations or salaries), the authors tested how well heuristic-prior models performed compared to standard Ridge.
  2. Medical Classification: Using the Breast Cancer Wisconsin (Diagnostic) Data Set, the model predicted malignancy based on cell nuclei characteristics.
  3. Neuroimaging: The authors applied the framework to functional Magnetic Resonance Imaging (fMRI) data. They used "Least Squares Separate" (LSS)—a technique robust to the high collinearity of brain imaging—as a prior for the more general "Least Squares All" (LSA) model.

Key Insights and Results

Graceful Failure vs. Catastrophic Failure

A primary advantage of heuristic-based priors is their behavior under high regularization. Standard Ridge regression fails "catastrophically" as $\theta$ increases because it eventually predicts that no variables are relevant (all weights become zero). In contrast, heuristic-prior models "fail gracefully," reverting to a sensible, predictive rule of thumb even when the penalty is extreme or data is minimal.

Robustness in "Small $n$, Large $p$" Scenarios

Heuristics often ignore the covariance between predictors. While this is usually seen as a limitation, the authors demonstrate it is a strength in high-noise or small-sample scenarios. Because calculating correlations is unreliable when data is scarce, ignoring them (via a heuristic prior) provides a stabilizing effect that prevents the model from being misled by spurious correlations.

Interpretability through Deviation

By shrinking toward a known heuristic, the resulting model weights become more interpretable. Instead of viewing coefficients in isolation, they can be understood as deviations from a simple rule. This provides a clearer narrative of what the data has added to our prior knowledge.

Conclusion and Future Directions

The use of robust, human-inspired priors offers a powerful alternative to standard zero-centered regularization. By "interpolating expertise," these models provide high performance across a wider range of data availability.

However, several challenges remain. The "strategy selection" problem—determining which heuristic is most appropriate for a given task—remains an open question for meta-learning. Additionally, the current two-step process uses the same data twice, which may introduce bias. Future work aims to expand this framework into a full Bayesian context, allowing for a more normative treatment of uncertainty and a deeper understanding of why these simple priors are so effective.

Summary updated

BibTeX
@misc{jones2020robust,
  author = {matt jones and Sebastian Bobadilla-Suarez and Matt Jones and Bradley C. Love},
  title = {{Robust priors for regularized regression}},
  year = {2020},
  eprint = {2010.02610},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.02610},
}
Figure