Abstract
Kernel Ridge Regression (KRR) is a fundamental model in machine learning, often analyzed through the lens of the "spectral bias" hypothesis, which posits that the model learns components associated with larger kernel eigenvalues first. However, this theoretical framework, typically derived using the replica method, often fails to predict generalization performance in low-dimensional settings. This paper investigates the root of this failure, identifying a breakdown in the "self-averaging" assumption when data density vanishes near decision boundaries. We introduce a characteristic crossover ridge value, $\lambda^*$, which separates the regime where standard theory holds from a regime of hypersensitivity to boundary data. Our results, supported by analytical physics-based modeling and empirical tests on MNIST and CIFAR-10, provide a more robust understanding of KRR generalization.
Introduction
Kernel Ridge Regression (KRR) serves as a vital theoretical proxy for understanding more complex architectures, including deep neural networks. A prevalent theory suggests that KRR exhibits "spectral bias," where the model preferentially fits function components corresponding to the largest eigenvalues of the kernel. This behavior is often predicted using the replica method from statistical mechanics, which relies on the "self-averaging" property—the assumption that the specific realization of training points becomes irrelevant in the large-data limit.
In practice, however, data is frequently low-dimensional, and its density may vanish near decision boundaries (e.g., the separation between classes in image datasets). This paper addresses the discrepancy between theoretical predictions and empirical observations in these low-dimensional, sparse-boundary regimes.
Theoretical Framework and Methods
The authors utilize a multidisciplinary approach combining theoretical physics and numerical simulation:
- Schrödinger-like Mapping: The problem of finding kernel eigenvectors in 1D is mapped to solving a Schrödinger equation. This allows the use of the Wentzel–Kramers–Brillouin (WKB) approximation to determine the asymptotic behavior of eigenvalues and eigenvectors.
- Scaling Analysis: The 1D results are generalized to higher dimensions ($d$) through scaling laws, predicting the behavior of generalization error ($\epsilon_t$) as a function of the number of training points ($P$) and the regularization parameter ($\lambda$).
- Variance and Self-Averaging: The study quantifies the breakdown of self-averaging by analyzing the variance of predictors across different training samples.
The Crossover Phenomenon
A central finding is the identification of a crossover ridge value, $\lambda^* \sim P^{-1/(d+\chi)}$, where $\chi$ represents the exponent of the data density vanishing at the boundary.
When $\lambda \gg \lambda^*$, the replica method's predictions are accurate. However, when $\lambda \ll \lambda^*$, the predictor becomes hypersensitive to training points near the decision boundary, leading to a breakdown of the spectral bias prediction. In this regime, the generalization error scales as $\epsilon_t \sim P^{-(1+\chi)/(d+\chi)}$, a result that deviates from standard theory unless the dimensionality $d$ is very large.
Empirical Results and Scaling Laws
The proposed scaling laws were validated using both synthetic datasets and real-world benchmarks. By plotting the error and variance against the scaled ridge $\lambda/\lambda^*$, the authors demonstrate a clear scaling collapse.
For real-world datasets like CIFAR-10 and MNIST, the authors modeled the data geometry using an intrinsic dimension $d$ and a boundary density exponent $\chi$. For CIFAR-10, parameters $d=35$ and $\chi=1.5$ yielded a near-perfect fit to empirical generalization curves. This suggests that the "vanishing density" model effectively captures the essential geometric features of high-dimensional image data.
Discussion and Conclusion
This work clarifies why spectral bias predictions, while often "good enough" in high dimensions, fail in low-dimensional contexts. The discovery of the crossover ridge $\lambda^*$ provides a precise boundary for the validity of traditional KRR theories. Future research may explore more complex boundary geometries beyond the hyperplane and cylindrical models used here, potentially accounting for the fractal nature of real-world data manifolds.
Summary updated