Abstract
For multiple index models, it has recently been shown that the sliced inverse regression (SIR) is consistent for estimating the sufficient dimension reduction (SDR) space if and only if $\rho=\lim\frac{p}{n}=0$, where $p$ is the dimension and $n$ is the sample size. Thus, when $p$ is of the same or a higher order of $n$, additional assumptions such as sparsity must be imposed in order to ensure consistency for SIR. By constructing artificial response variables made up from top eigenvectors of the estimated conditional covariance matrix, we introduce a simple Lasso regression method to obtain an estimate of the SDR space. The resulting algorithm, Lasso-SIR, is shown to be consistent and achieve the optimal convergence rate under certain sparsity conditions when $p$ is of order $o(n^2\lambda^2)$, where $\lambda$ is the generalized signal-to-noise ratio. We also demonstrate the superior performance of Lasso-SIR compared with existing approaches via extensive numerical studies and several real data examples.
Introduction
In modern high-dimensional data analysis, researchers frequently encounter datasets where the number of features ($p$) significantly exceeds the number of observations ($n$). Sufficient Dimension Reduction (SDR) serves as a critical tool in this context, aiming to identify a low-dimensional subspace of predictors that captures all relevant information about a response variable $y$ without assuming a specific parametric model.
Sliced Inverse Regression (SIR) is one of the most popular SDR techniques. However, classical SIR faces a fundamental limitation: it is only mathematically consistent when $p$ is much smaller than $n$. In high-dimensional settings where $p \ge n$, the covariance matrices involved become singular, rendering the standard SIR estimator unstable or undefined. To address this, the authors propose Lasso-SIR, a framework that incorporates sparsity assumptions to enable consistent dimension reduction in the "large $p$, small $n$" regime.
The Lasso-SIR Framework
The innovation of Lasso-SIR lies in its ability to transform a complex dimension reduction problem into a series of manageable penalized regression tasks. The method follows a three-step logic:
- The Pseudo-Response Trick: Rather than attempting to directly invert a high-dimensional covariance matrix, the algorithm first computes the top eigenvectors of a sliced conditional covariance matrix. These eigenvectors are used to construct "pseudo-response" variables.
- Lasso Integration: By treating these pseudo-responses as targets in a standard linear regression, the authors can apply the $L_1$ penalty (Lasso). This step effectively performs variable selection and identifies the important directions in the predictor space while handling the high dimensionality of $p$.
- Determining Subspace Dimension: The authors introduce a practical k-means clustering approach on adjusted eigenvalues to estimate the number of important directions ($d$), a task that is traditionally difficult in SDR.
This approach is computationally advantageous because it allows researchers to utilize highly optimized Lasso solvers, such as the glmnet package, making it scalable to massive datasets.
Theoretical and Empirical Results
Theoretical Optimality
The paper provides a rigorous proof that Lasso-SIR achieves the minimax optimal convergence rate of $s \log(p) / n\lambda$. This indicates that the algorithm is theoretically as efficient as possible under the given sparsity conditions, provided the generalized signal-to-noise ratio ($\lambda$) is sufficiently high.
Numerical Performance
Through extensive simulations involving non-linear relationships (such as cubic and sine functions), Lasso-SIR demonstrated superior performance over competing methods like Diagonal Thresholding-SIR and Matrix Lasso. It proved particularly robust in scenarios where predictors were highly correlated.
Real-World Applications
The utility of Lasso-SIR was further validated on high-dimensional biological datasets:
- Arcene Data: In a mass-spectrometry dataset with 10,000 features and only 100 samples, Lasso-SIR achieved a higher Area Under the Curve (AUC) for cancer classification than Sparse PCA or standard Lasso.
- HapMap Data: The algorithm successfully processed a gene expression dataset containing over 47,000 features.
- Wine Cultivars: Lasso-SIR provided superior 2D visualizations, showing clearer separation between different cultivars compared to other sparse SIR variants.
Discussion and Limitations
While Lasso-SIR is a powerful advancement, its theoretical consistency relies on the assumption of elliptical symmetry in the predictor distribution (e.g., multivariate normality). For real-world data that violates this, the authors suggest quantile normalization as a preprocessing step. Additionally, the method's efficacy is tied to the signal-to-noise ratio; if the signal is too weak relative to the dimensionality, the pseudo-response construction may lose accuracy.
Future research may extend this "pseudo-response" framework to other regularization techniques, such as SCAD or Group Lasso, and explore its application to other dimension reduction methods beyond SIR.
Summary updated