Abstract
The central subspace of a pair of random variables $(y,x) \in \mathbb{R}^{p+1}$ is the minimal subspace $\mathcal{S}$ such that $y \perp \hspace{-2mm} \perp x\mid P_{\mathcal{S}}x$. This paper investigates the minimax rate of estimating the central space of multiple index models $y=f(\beta_{1}^T x, \dots, \beta_{d}^T x, \epsilon)$ with at most $s$ active predictors, where $x \sim N(0,I_{p})$. We introduce a class of models characterized by the smallest non-zero eigenvalue $\lambda$ of $var(\mathbb{E}[x|y])$. We demonstrate that an aggregated estimator based on Sliced Inverse Regression (SIR) achieves a convergence rate of $d \wedge ((sd + s \log(ep/s))/(n\lambda))$. This rate is shown to be optimal for single-index models and multiple-index models with fixed central dimensions. These results provide fundamental insights into high-dimensional Sufficient Dimension Reduction (SDR).
Introduction
In the era of high-dimensional data, the "curse of dimensionality" often manifests as the $p \gg n$ problem, where the number of predictors ($p$) significantly exceeds the number of observations ($n$). Sufficient Dimension Reduction (SDR) offers a powerful paradigm to mitigate this by identifying a low-dimensional "central subspace" that retains all the predictive information of $x$ regarding a response $y$.
Among SDR techniques, Sliced Inverse Regression (SIR) is widely utilized due to its computational efficiency and lack of requirement for a specific model form for the link function. However, classical SIR is inconsistent in high-dimensional settings. To restore consistency, researchers typically assume sparsity—the notion that only a small subset $s$ of the $p$ predictors are truly influential. While various "Sparse SIR" variants have been proposed, the fundamental statistical limits—the minimax rates—for this estimation problem remained unknown until now.
The Minimax Rate of Estimation
This research establishes the theoretical "speed limit" for estimating the central subspace. The authors identify the critical role of $\lambda$, the smallest non-zero eigenvalue of the conditional covariance matrix $var(\mathbb{E}[x|y])$. This $\lambda$ serves as a proxy for the signal strength; the higher the $\lambda$, the easier the subspace is to estimate.
The core contribution of the paper is the derivation of the minimax rate: $$R = d \wedge \frac{sd + s \log(ep/s)}{n\lambda}$$ This rate elegantly captures the interplay between the subspace dimension ($d$), the sparsity level ($s$), the total number of predictors ($p$), and the signal strength ($\lambda$). For practical purposes in high dimensions, this often simplifies to a rate of $\frac{s \log p}{n\lambda}$.
Methodology: Upper and Lower Bounds
To prove that this rate is indeed the best possible, the authors employ two distinct theoretical approaches:
- Upper Bound (Achievability): The authors construct an "Aggregation Estimator." This theoretical construct works by considering various sparse subsets of predictors and aggregating the results. While computationally intensive, it proves that the rate is mathematically achievable. They also analyze Diagonal Thresholding SIR (DT-SIR), a more practical algorithm that screens variables based on the variance of their conditional means, showing it performs well in practice.
- Lower Bound (Optimality): Using Fano’s Inequality from information theory, the authors prove that no possible algorithm, regardless of its computational complexity, can achieve a better convergence rate than the one derived.
The analysis relies heavily on Sin-Theta Theorems to measure the distance between the estimated projection matrix and the true projection matrix of the central subspace.
Key Insights: SIR vs. PCA
A significant finding of this work is the characterization of SIR's behavior in high dimensions. In many high-dimensional problems, such as sparse Principal Component Analysis (PCA), there exists a "computational-statistical gap" where the best statistical rate cannot be achieved by efficient algorithms.
However, this paper reveals that sparse SIR behaves more like sparse linear regression. For single-index models, the computationally efficient DT-SIR algorithm actually reaches the optimal minimax rate. This suggests that SDR in high dimensions may be more computationally tractable than previously feared.
Furthermore, the study revisits the role of "slicing." Traditional SIR wisdom suggested that the number of slices ($H$) had a minimal impact on performance. This paper demonstrates that for the specific task of estimating eigenvalues (the signal strength), a larger number of slices significantly improves accuracy.
Limitations and Future Directions
While providing a foundational theoretical framework, the study notes several areas for further exploration:
- Distributional Assumptions: The current proofs assume predictors follow a Gaussian distribution ($x \sim N(0, I_p)$). Relaxing this while maintaining the linearity condition is a key next step.
- The Technical Conjecture: The optimality for models where $d$ is bounded but not fixed relies on a technical conjecture regarding d-variate functions. While numerical simulations support this conjecture, a formal proof is pending.
- Predictor Correlation: The current model assumes independent predictors. Future research will need to incorporate more complex regularization (like Lasso-type penalties) to handle highly correlated high-dimensional data.
Conclusion
By establishing the minimax rates for Sliced Inverse Regression, this paper provides a benchmark for evaluating SDR algorithms in high dimensions. The discovery that SIR mirrors the optimality structures of sparse regression rather than sparse PCA offers a promising outlook for the development of efficient, high-performance dimension reduction tools.
Summary updated