Kernel dimension reduction in regression

Kenji Fukumizu, Francis Bach, Michael I. JordanThe Annals of Statistics 2009 · DOI

Abstract

We present a new methodology for sufficient dimension reduction (SDR). Our methodology derives directly from the formulation of SDR in terms of the conditional independence of the covariate $X$ from the response $Y$, given the projection of $X$ on the central subspace. We show that this conditional independence assertion can be characterized in terms of conditional covariance operators on reproducing kernel Hilbert spaces (RKHS) and we show how this characterization leads to an M-estimator for the central subspace. The resulting estimator is shown to be consistent under weak conditions; in particular, we do not have to impose linearity or ellipticity conditions of the kinds that are generally invoked for SDR methods. We also present empirical results showing that the new methodology is competitive in practice.

Introduction

In high-dimensional regression analysis, the primary challenge is to identify a low-dimensional representation of the input variables ($X$) that retains all necessary information to predict a response variable ($Y$). This objective is formalized as Sufficient Dimension Reduction (SDR). Specifically, researchers seek the "central subspace"—the smallest linear subspace such that $Y$ is independent of $X$ given the projection of $X$ onto that subspace.

Historically, SDR has been addressed through methods such as Sliced Inverse Regression (SIR), Sliced Average Variance Estimation (SAVE), and Principal Hessian Directions (pHd). While effective in certain contexts, these classical approaches carry a significant theoretical burden: they typically require the distribution of $X$ to be elliptically symmetric (e.g., multivariate normal). When data is skewed, clustered, or exhibits complex non-linear dependencies, these parametric assumptions fail, and the resulting dimension reduction may lose critical information.

The Kernel Dimension Reduction (KDR) Framework

Kernel Dimension Reduction (KDR) provides a non-parametric alternative that sidesteps the restrictive distributional assumptions of traditional SDR. By leveraging the mathematical framework of Reproducing Kernel Hilbert Spaces (RKHS), KDR transforms the search for the central subspace into a clean optimization problem.

Characterizing Independence via RKHS

The core innovation of KDR lies in its use of conditional covariance operators. The authors demonstrate that the conditional independence of $Y$ and $X$ (given a projection) can be characterized by minimizing the "residual covariance" within a kernel space. By mapping the data into a high-dimensional feature space using "characteristic kernels" (such as the Gaussian RBF kernel), the method becomes sensitive to virtually any form of statistical dependency, including higher-order moments that linear methods ignore.

The M-Estimator and Consistency

KDR functions as an M-estimator. The authors derive a contrast function based on Gram matrices—matrices representing kernel evaluations between data points. To find the optimal subspace, the algorithm minimizes the trace of the kernel-based conditional covariance.

Crucially, KDR does not require the explicit estimation of the regression function $E[Y|X]$ or the density of the data. This focus on the independence assertion itself allows the authors to provide a rigorous proof of statistical consistency. Under very weak conditions, the KDR estimator is guaranteed to converge to the true central subspace as the sample size increases, providing a theoretical foundation that was previously missing from heuristic kernel-based attempts.

Empirical Performance

The authors evaluated KDR against traditional methods across several benchmarks:

  • Synthetic Simulations: In tests involving non-elliptic distributions and models where the relationship between $X$ and $Y$ was hidden in the variance (Model C), KDR significantly outperformed SIR and pHd. In many cases, the classical methods failed to identify the subspace entirely, while KDR remained robust.
  • Swiss Bank Notes: In this classification task (distinguishing genuine from counterfeit bills), KDR identified a two-dimensional projection that provided clear class separation. In contrast, methods like SAVE produced overlapping clusters that hindered classification.
  • Evaporation Data: Even on small, complex meteorological datasets, KDR successfully identified relevant features relating soil evaporation to environmental conditions without requiring a pre-specified model.

Key Insights and Advantages

  1. Breaking the Ellipticity Constraint: KDR proves that the "linearity condition" is not a prerequisite for SDR. This allows for the analysis of real-world data that is skewed or non-Gaussian.
  2. Universality: The use of characteristic kernels ensures that the method captures non-linear dependencies and complex interactions between variables.
  3. Computational Tractability: While the underlying math involves abstract Hilbert spaces, the final algorithm is a practical optimization of Gram matrices, making it accessible for implementation using standard numerical techniques.

Limitations and Future Directions

Despite its strengths, KDR presents several avenues for further research:

  • Dimensionality Selection: The current framework assumes the dimension ($d$) of the central subspace is known a priori. Developing internal criteria to determine the optimal $d$ remains a priority.
  • Hyperparameter Tuning: Like all kernel methods, KDR is sensitive to the kernel scale (e.g., the width of the Gaussian kernel). While the authors utilize a continuation method to address this, automated scale selection is an ongoing challenge.
  • Scalability: The reliance on $n \times n$ Gram matrices suggests that for very large datasets ($n > 10,000$), the method may require computational optimizations such as low-rank approximations or Nyström methods to maintain efficiency.

Conclusion

Kernel Dimension Reduction represents a significant advancement in the field of regression graphics. By replacing restrictive parametric assumptions with the flexible geometry of RKHS, KDR offers a robust, consistent, and powerful tool for discovering the hidden structures within high-dimensional data.

Summary updated

Abstract

We present a new methodology for sufficient dimension reduction (SDR). Our methodology derives directly from the formulation of SDR in terms of the conditional independence of the covariate X from the response Y, given the projection of X on the central subspace [cf. J. Amer. Statist. Assoc. 86 (1991) 316–342 and Regression Graphics (1998) Wiley]. We show that this conditional independence assertion can be characterized in terms of conditional covariance operators on reproducing kernel Hilbert spaces and we show how this characterization leads to an M-estimator for the central subspace. The resulting estimator is shown to be consistent under weak conditions; in particular, we do not have to impose linearity or ellipticity conditions of the kinds that are generally invoked for SDR methods. We also present empirical results showing that the new methodology is competitive in practice.
BibTeX
@article{fukumizu2009kernel,
  author = {Kenji Fukumizu and Francis Bach and Michael I. Jordan},
  title = {{Kernel dimension reduction in regression}},
  journal = {The Annals of Statistics},
  year = {2009},
  doi = {10.1214/08-aos637},
}
Figure