In classical statistics, we are taught that models require explicit regularization—like a weight decay penalty—to avoid overfitting. Without this "leash," a model with enough capacity will chase every bit of noise in the training data, causing its test error to explode.
However, recent research into "ridgeless" regression (where the regularization parameter $\lambda$ is set to zero) challenges this necessity. In the underparameterized regime—where the number of features or the rank of the model is smaller than the number of training samples—the model generalizes surprisingly well even without an explicit penalty. This happens because the architecture of the model itself provides an implicit regularization.
Feature Count as a Natural Penalty
When we train a model with fewer features than data points, the model is physically unable to "memorize" the noise in the dataset. In Ridgeless Regression with Random Features, Jian Li and colleagues demonstrate that the number of random features (RF) acts as a tuning knob for regularization.
Think of it this way: if you have 1,000 data points but only 100 features, the model is forced to find the most significant patterns that explain the data. It doesn't have enough "degrees of freedom" to fit the random fluctuations. Jian Li shows that reducing the number of features increases this implicit regularization effect, mirroring the behavior of explicit weight decay. They even propose an algorithm, RFTK, that optimizes the spectral density of these features to maximize this effect, essentially "shaping" the kernel to better suit the data during training.
Stability at the Zero Limit
A major concern in ridgeless regression is the "limit" behavior. In many theoretical models, as the regularization parameter $\lambda$ approaches zero, the variance of the estimator is expected to diverge, leading to infinite error.
However, Tin Sum Cheng, Aurélien Lucchi, and Ivan Dokmanić prove in A Theoretical Analysis of the Test Error of Finite-Rank Kernel Ridge Regression that this explosion doesn't happen in the underparameterized regime. By analyzing finite-rank kernels—kernels that can be represented by a fixed, limited number of features—they show that the test error remains stable and well-behaved even at $\lambda = 0$.
Their work provides "sharp" bounds, meaning they've identified the exact mathematical ceiling and floor for how much error the model will incur. This stability is crucial for modern machine learning workflows like transfer learning or fine-tuning, where we often work with low-rank approximations of massive models. It tells us that we don't need to obsess over finding the perfect non-zero $\lambda$ to keep the model from breaking; the finite rank of the system keeps the error in check.
Why the Underparameterized Regime Matters
While much of the recent "double descent" literature focuses on overparameterized models (where features outnumber samples), understanding the underparameterized side is equally vital.
- Computational Efficiency: Finite-rank and random-feature models are much cheaper to compute than full-rank kernels.
- Implicit vs. Explicit: By understanding how feature count and kernel rank act as implicit regularizers, researchers can design models that are "self-regularizing" by construction.
- Theoretical Bridging: This research bridges the gap between classical "simple" models and modern "complex" interpolating models. It shows that the "interpolation phenomenon"—the ability to generalize well with zero training error—is not a quirk of massive neural networks, but a fundamental property of how models interact with the dimensionality of their feature spaces.
In short, when the model's capacity is constrained by its rank or feature count, the "ridgeless" limit is not a danger zone of overfitting, but a stable and predictable regime for generalization.