RaSE: A Variable Screening Framework via Random Subspace Ensembles

Ye Tian, Yang FengJournal of the American Statistical Association 2021 · OpenReview

Introduction

In modern statistical learning, the "ultra-high dimensional" setting—where the number of predictors $p$ is significantly larger than the sample size $n$—presents a formidable challenge. To handle such data, researchers typically employ variable screening to reduce the feature space to a manageable size before applying more intensive downstream models. The most common approach, Sure Independence Screening (SIS), ranks variables based on their marginal correlation with the response. However, SIS and its variants often fail to detect "hidden" signals: variables that have little to no marginal effect but are highly predictive when considered jointly with others.

To address this limitation, we present RaSE (Random Subspace Ensemble), a flexible framework for variable screening. By evaluating the quality of random subspaces rather than individual features, RaSE can identify complex interactions and joint effects that traditional methods miss.

The RaSE Framework

The core philosophy of RaSE is that a variable's importance is best judged in the context of other variables. The framework operates through a multi-step ensemble process:

  1. Subspace Generation: The algorithm generates a large number of random subspaces, each containing a small subset of the total features.
  2. Local Optimization: These subspaces are organized into groups. Within each group, the "best" subspace is selected based on a user-defined criterion, such as the Bayesian Information Criterion (BIC), extended BIC (eBIC), or cross-validation error.
  3. Importance Scoring: RaSE calculates a "selected proportion" for each variable—the frequency with which a specific variable appears in the winning subspaces across all groups. This proportion serves as a robust measure of a variable's contribution to the model's predictive power.

One of the primary strengths of RaSE is its modularity. It can be paired with various base learners, including linear models, k-Nearest Neighbors (kNN), or Support Vector Machines (SVM), allowing it to adapt to both linear and non-linear data structures.

Iterative Refinement

For particularly challenging datasets where signals are extremely sparse or buried in noise, the authors developed Iterative RaSE. In this version, the screening process is repeated over multiple rounds. After the first round, the sampling probability for each variable is updated based on its performance. Features that showed promise in the initial pass are given a higher probability of being included in the random subspaces of the next round. This "zoom-in" effect allows the algorithm to concentrate its computational resources on the most relevant regions of the feature space.

Theoretical Guarantees

The RaSE framework is backed by rigorous theoretical analysis. The authors demonstrate that RaSE possesses the sure screening property, meaning that with a probability approaching one, the set of selected variables will include all true signals. Furthermore, it achieves rank consistency, ensuring that true signals are ranked higher than noise variables in the final importance scores.

Empirical Performance

The effectiveness of RaSE was validated through extensive simulations and real-world data analysis:

  • Identifying "Invisible" Signals: In simulations where a predictor had zero marginal correlation with the response but a strong joint effect, traditional SIS failed to identify the signal. RaSE-BIC, however, consistently ranked the signal at the top, maintaining a small and accurate model size.
  • Biological Data Application: On a Colon cancer dataset (2,000 genes, 62 patients), RaSE-BIC achieved a classification error of 0.1192. This significantly outperformed the LASSO (0.1792) and SIS (0.1633), demonstrating its utility in high-stakes genomic research.
  • Robustness: In the Rat Eye Expression dataset, RaSE showed superior stability and predictive accuracy compared to iterative SIS and other competitive screening methods.

Conclusion and Future Work

RaSE represents a significant step forward in variable screening by moving from marginal to joint evaluation. While the framework introduces higher computational costs due to the ensemble nature of the algorithm, these costs are easily offset by parallel processing. Future research directions include the development of adaptive methods for selecting the number of iterations and optimizing subspace sizes dynamically to further improve efficiency. By providing a flexible, theoretically grounded approach to feature selection, RaSE offers a powerful tool for navigating the complexities of ultra-high dimensional data.

Summary updated

Abstract

Variable screening methods have been shown to be effective in dimension reduction under the ultra-high dimensional setting. Most existing screening methods are designed to rank the predictors according to their individual contributions to the response. As a result, variables that are marginally independent but jointly dependent with the response could be missed. In this work, we propose a new framework for variable screening, random subspace ensemble (RaSE), which works by evaluating the quality of random subspaces that may cover multiple predictors. This new screening framework can be naturally combined with any subspace evaluation criterion, which leads to an array of screening methods. The framework is capable to identify signals with no marginal effect or with high-order interaction effects. It is shown to enjoy the sure screening property and rank consistency. We also develop an iterative version of RaSE screening with theoretical support. Extensive simulation studies and real-data analysis show the effectiveness of the new screening framework.
BibTeX
@article{tian2021rase,
  author = {Ye Tian and Yang Feng},
  title = {{RaSE: A Variable Screening Framework via Random Subspace Ensembles}},
  journal = {Journal of the American Statistical Association},
  year = {2021},
  url = {https://openreview.net/forum?id=vhoClTYpJh},
}
Figure