Abstract: We compare the risk of ridge regression to a simple variant of ordinary least squares, in which one simply projects the data onto a finite dimensional subspace (as specified by a Principal Component Analysis) and then performs an ordinary (un-regularized) least squares regression in this subspace. This note shows that the risk of this ordinary least squares method is within a constant factor (namely 4) of the risk of ridge regression.
Introduction
In high-dimensional linear regression, where the number of features $p$ may approach or exceed the number of observations $n$, standard Ordinary Least Squares (OLS) often fails due to overfitting. Practitioners typically turn to regularization techniques to stabilize estimates. Ridge Regression (RR), or Tikhonov regularization, is the most common approach, employing a smooth $\ell_2$ penalty to shrink coefficients.
An alternative, simpler strategy is Principal Component Regression (PCA-OLS). Rather than smooth shrinkage, PCA-OLS applies a "hard cutoff" by projecting data onto the top $k$ principal components and performing un-regularized OLS in that reduced space. This paper quantifies the theoretical performance gap between these two paradigms, asking whether the mathematical elegance of Ridge Regression translates to a significant risk advantage over the simpler PCA-based projection.
Methodology
The study utilizes a fixed design setting, where the feature matrix $X$ is constant and randomness arises from label noise. Both methods can be viewed through the lens of spectral scaling of principal components:
- Ridge Regression (RR): Scales each component by $\frac{\lambda_j}{\lambda_j + \lambda}$, where $\lambda_j$ is the eigenvalue and $\lambda$ is the regularization parameter.
- PCA-OLS: Employs a binary threshold. Components with $\lambda_j \ge \lambda$ are kept (scale factor 1), while those with $\lambda_j < \lambda$ are discarded (scale factor 0).
By performing a bias-variance decomposition of the expected excess loss (risk), the authors compare these scaling factors term-by-term to bound the risk ratio.
Theoretical Results: The Factor of 4 Bound
The central theoretical contribution is Theorem 2, which establishes a universal upper bound on the risk ratio:
$$0 \le \frac{\text{Risk(PCA-OLS)}}{\text{Risk(RR)}} \le 4$$
This result guarantees that PCA-OLS is never more than four times worse than Ridge Regression for any $\lambda \ge 0$. Crucially, the reverse is not true: Ridge Regression can be arbitrarily worse than PCA-OLS. This occurs when signal is concentrated in high-variance components, yet Ridge continues to accumulate variance from low-variance components that PCA-OLS would simply truncate.
Empirical Evidence
The theoretical bounds were tested using both synthetic simulations and real-world datasets (USPS handwritten digits and BCI brain-computer interface data).
In synthetic trials with $p=100$ and varying $n$, the risk ratio consistently remained well below the theoretical limit of 4. Interestingly, as the regularization parameter $\lambda$ increases, PCA-OLS often begins to outperform Ridge Regression.
Experiments on the USPS and BCI datasets further support this. In many practical regimes, the risk ratio drops below 1, indicating that the "hard" truncation of PCA-OLS is more efficient than the "soft" shrinkage of Ridge. This suggests that the simplicity of subspace projection does not necessarily come at the cost of accuracy.
Conclusion and Limitations
This research provides a rigorous safety net for using PCA-based dimensionality reduction, showing it is a robust approximation of Ridge Regression. However, several caveats remain. The findings rely on a fixed design assumption and assume the same $\lambda$ is used for both methods. Future work is needed to explore how these relationships hold under random designs and when $\lambda$ is independently tuned for each method via cross-validation.
Summary updated