In supervised learning, we often treat regularization as a static hyperparameter tuned via cross-validation. However, as the volume of available data grows, the role of the regularization parameter $\lambda$ must fundamentally shift. If we keep $\lambda$ constant while the number of samples $n$ increases, the model remains permanently biased toward its prior, failing to take full advantage of the new information. Conversely, if we drop regularization too quickly, the model becomes hypersensitive to noise.
The core of this research direction is the discovery of "scaling laws" for regularization—mathematical rules that dictate exactly how fast the penalty strength should vanish as the dataset grows to ensure the model converges to the true underlying solution.
Learning as an Inverse Problem
To understand why scaling matters, it helps to view learning through the lens of "ill-posed inverse problems." In Learning, Regularization and Ill-Posed Inverse Problems, Lorenzo Rosasco and colleagues argue that supervised learning is essentially the task of inverting a data-generation process. Because we only have a finite, noisy sample of data, this inversion is unstable.
Regularization (like Tikhonov regularization) stabilizes this process by adding a penalty term. Rosasco proves that "consistency"—the guarantee that our algorithm will eventually find the best possible model—is only possible if the regularization parameter is scaled as a specific power of the sample size. As $n \to \infty$, the penalty must vanish so the data can eventually dominate the solution, but it must vanish at a rate that keeps the "noise" of the finite sample under control.
The Goldilocks Zone of Decay Rates
The central challenge is finding the "Goldilocks" rate of decay for $\lambda$. In Stability Analysis for Regularized Least Squares Regression, Cynthia Rudin demonstrates that if the regularization strength vanishes too quickly, the variance introduced by noisy data overwhelms the model. The estimator "blows up" because it tries to fit every fluctuation in a massive, noisy dataset.
Different models require different scaling strategies:
- Linear Ridge Regression: Petri Koistinen shows in Asymptotic Theory for Regularization: One-Dimensional Linear Case that the optimal parameter scales inversely with the sample size ($1/n$).
- Smoothing Splines: In Pointwise Convergence in Probability of General Smoothing Splines, Adam Michael Johansen and Matthew Thorpe identify a sharp threshold: if $\lambda$ scales as $n^{-p}$ where $p > 1/2$, the estimator fails to filter out noise and loses consistency.
- Kernel CCA: For complex correlation analysis, Kenji Fukumizu and Francis Bach found in Statistical Convergence of Kernel CCA that the product of the cube root of the sample size and the regularization coefficient must approach infinity to maintain stability.
Achieving Optimal Convergence Rates
The goal of these scaling laws isn't just to ensure the model eventually works, but to make it work as efficiently as possible. Ingo Steinwart and Clint Scovel show in Fast Rates to Bayes for Kernel Machines that by precisely scheduling the decay of $\lambda$, Support Vector Machines can achieve nearly linear convergence rates. This means the error drops much faster than the standard $1/\sqrt{n}$ rate typically expected in statistical learning.
While these rates were historically derived through complex "a priori" mathematical proofs, recent work by Jonathan Chirinos Rodriguez and Lorenzo Rosasco in On Learning the Optimal Regularization Parameter in Inverse Problems suggests we can learn these optimal scaling rules directly from data. By using Empirical Risk Minimization on a training set of known solutions, a meta-model can discover the optimal tuning rule without needing the researcher to know the underlying smoothness or noise level of the problem.
Why This Works
The intuition behind these results is rooted in the balance between bias and variance.
- Early on (small $n$): We have little data, so the variance of our estimate is high. We use strong regularization (high $\lambda$) to bias the model toward a simpler solution, "pinning" it down so it doesn't chase noise.
- As data grows ($n \to \infty$): The sheer volume of data naturally reduces variance. We can afford to relax the "pinning" (lower $\lambda$).
- The Limit: As noted by Tianshi Chen and Lennart Ljung in their Tutorial on Asymptotic Properties, the influence of the regularization must eventually vanish relative to the data likelihood. This ensures the estimator becomes asymptotically unbiased, allowing the model to perfectly reflect the reality hidden within the data.