Kernel Partial Least Squares is Universally Consistent

Gilles Blanchard, Nicole KrämerAISTATS 2010 · OpenReview

Abstract

We prove the statistical consistency of kernel Partial Least Squares Regression applied to a bounded regression learning problem on a reproducing kernel Hilbert space. Partial Least Squares stands out of well-known classical approaches as e.g. Ridge Regression or Principal Components Regression, as it is not defined as the solution of a global cost minimization procedure over a fixed model nor is it a linear estimator. Instead, approximate solutions are constructed by projections onto a nested set of data-dependent subspaces. To prove consistency, we exploit the known fact that Partial Least Squares is equivalent to the conjugate gradient algorithm in combination with early stopping. The choice of the stopping rule (number of iterations) is a crucial point. We study two empirical stopping rules. The first one monitors the estimation error in each iteration step of Partial Least Squares, and the second one estimates the empirical complexity in terms of a condition number. Both stopping rules lead to universally consistent estimators provided the kernel is universal.

Introduction

Partial Least Squares (PLS) has long been a cornerstone of high-dimensional data analysis, particularly in fields like chemometrics and bioinformatics where the number of features often dwarfs the number of available samples. By seeking directions that maximize the covariance between predictors and the target variable, PLS often outperforms Principal Component Regression (PCR) in predictive tasks. However, the mathematical nature of PLS—specifically its non-linear, iterative construction of data-dependent subspaces—has historically made it resistant to rigorous statistical analysis.

This paper addresses a fundamental question: Is Kernel Partial Least Squares (KPLS) "universally consistent"? That is, as the sample size grows to infinity, is the estimator guaranteed to converge to the true underlying regression function?

The authors' primary theoretical lever is the equivalence between KPLS and the Conjugate Gradient (CG) algorithm. Specifically, KPLS can be viewed as the CG method applied to the "normal equations" within a Reproducing Kernel Hilbert Space (RKHS). Unlike Ridge Regression or Lasso, which minimize a global cost function, KPLS builds its solution greedily through projections onto nested subspaces.

This iterative nature introduces a significant challenge: the estimator is non-linear because the subspaces themselves depend on the labels ($y$). Consequently, standard consistency proofs for linear estimators do not apply. The key to ensuring consistency lies in the "early stopping" rule—deciding exactly when to stop adding latent components to prevent overfitting.

Data-Dependent Stopping Rules

A central contribution of this work is the development of two novel, data-dependent stopping rules. The authors demonstrate that a fixed, data-independent stopping rule (e.g., always stopping at a fixed number of iterations) cannot guarantee consistency in this framework. Instead, the stopping point must adapt to the specific noise and structure of the data.

  1. Iterative Error Monitoring: This rule calculates upper bounds on the estimation error at each iteration. The algorithm terminates when the error bound exceeds a threshold relative to the sample size $n$.
  2. Empirical Complexity Control: This strategy monitors a "pseudo-condition number" of the data matrix, measuring the stability of the current solution. It stops the process when the model's complexity begins to grow too rapidly relative to the available data.

Key Insights and Theoretical Results

The paper provides several critical insights into the behavior of KPLS:

  • Bias Reduction: KPLS is shown to be more efficient than PCR at reducing bias. Because it is response-dependent, it cancels out errors associated with the largest eigenvalues of the kernel operator more effectively than methods that only consider feature variance.
  • Universal Consistency: The main results (Theorems 2 and 3) establish that if the chosen kernel is "universal" (such as the Gaussian RBF kernel), the KPLS estimator converges to the true regression function in the $L_2$ sense as $n \to \infty$.

The proof structure involves showing that the population version of KPLS (the ideal case with infinite data) converges to the truth, and then demonstrating that the discrepancy between the empirical KPLS and this population version remains bounded under the proposed stopping rules.

Limitations and Future Outlook

While this work provides the first formal proof of universal consistency for KPLS, it opens several avenues for future research:

  • Convergence Rates: The paper proves that KPLS converges, but it does not establish how fast (the minimax rates). Determining the optimal convergence rates for KPLS remains an open theoretical challenge.
  • Practical Application: The proposed stopping rules are mathematically rigorous but complex to implement. In applied settings, cross-validation remains the standard method for selecting the number of components.
  • Assumptions: The current proof assumes the target variable $Y$ is bounded, an assumption that may need relaxation for certain real-world applications.

Conclusion

By bridging the gap between iterative optimization and statistical learning theory, this paper provides a solid theoretical foundation for Kernel Partial Least Squares. The proof of universal consistency ensures that KPLS is not just a powerful heuristic, but a statistically sound method for non-linear regression in infinite-dimensional spaces.

Summary updated

Abstract

We prove the statistical consistency of kernel Partial Least Squares Regression applied to a bounded regression learning problem on a reproducing kernel Hilbert space. Partial Least Squares stands out of well-known classical approaches as e.g. Ridge Regression or Principal Components Regression, as it is not defined as the solution of a global cost minimization procedure over a fixed model nor is it a linear estimator. Instead, approximate solutions are constructed by projections onto a nested set of data-dependent subspaces. To prove consistency, we exploit the known fact that Partial Least Squares is equivalent to the conjugate gradient algorithm in combination with early stopping. The choice of the stopping rule (number of iterations) is a crucial point. We study two empirical stopping rules. The first one monitors the estimation error in each iteration step of Partial Least Squares, and the second one estimates the empirical complexity in terms of a condition number. Both stopping rules lead to universally consistent estimators provided the kernel is universal.
BibTeX
@inproceedings{blanchard2010kernel,
  author = {Gilles Blanchard and Nicole Krämer},
  title = {{Kernel Partial Least Squares is Universally Consistent}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2010},
  url = {https://openreview.net/forum?id=qwIP1PMdLz5},
}
Figure