Spearman Rank Correlation Screening for Ultrahigh-Dimensional Censored Data

Hongni Wang, Jingxin Yan, Xiaodong YanAAAI 2023 · DOI

Abstract

Feature screening in ultrahigh-dimensional settings is a critical step for identifying relevant predictors when the number of variables significantly exceeds the number of observations. However, traditional screening methods often struggle with censored data, non-linear relationships, and outliers. This paper introduces Spearman Rank Correlation Screening for Censored Data ($SRCS_{cen}$), a model-free approach that leverages rank-based correlation and a local Kaplan-Meier imputation technique. Our results demonstrate that $SRCS_{cen}$ maintains high coverage probability and rank consistency even under extreme censoring rates (up to 65%) and heavy-tailed distributions, significantly outperforming existing model-dependent and model-free alternatives.

1. Introduction

In fields such as genomics and financial risk analysis, researchers frequently encounter "ultrahigh-dimensional" datasets where the number of potential predictors ($p$) is vastly larger than the sample size ($n$). Feature screening serves as a preliminary filter to reduce the dimensionality to a manageable scale. This task is complicated by censoring, which occurs when the exact value of a response variable (e.g., survival time) is only partially observed.

Existing screening methods typically face three primary challenges:

  1. Model Dependency: Many methods assume a specific functional form (e.g., linear or Cox models), leading to failure if the true relationship is non-linear.
  2. Sensitivity to Outliers: Standard correlation measures can be skewed by extreme values or heavy-tailed distributions.
  3. Censoring Fragility: Performance often degrades sharply as the censoring rate (CR) increases.

2. Methodology

The proposed $SRCS_{cen}$ method addresses these limitations through two core innovations:

2.1 Spearman Rank Correlation

By utilizing Spearman rank correlation rather than Pearson correlation, the method becomes "model-free." It is invariant to monotonic transformations of the data, meaning it can identify relationships regardless of whether they are linear or non-linear (e.g., logarithmic). Because it relies on the relative order of data points, it is inherently robust against outliers.

2.2 Imputation for Censoring

To handle censored observations, the authors employ a local Kaplan-Meier estimator to "impute" the conditional expectation of the response variable. This allows the calculation of a robust correlation index ($\omega_k$) for every variable by incorporating information from censored data that other methods might discard. The method establishes "Sure-Screening" properties, ensuring the probability of including all true active variables approaches 1 as the sample size grows.

3. Experimental Results

3.1 Simulation Performance

The authors evaluated $SRCS_{cen}$ against several competitors, including $CQS_{cen}$ and $RCS_{cen}$, across various scenarios. As shown in Figure 1, $SRCS_{cen}$ exhibits superior performance, particularly at high censoring rates. In a linear model simulation with a 65% censoring rate, $SRCS_{cen}$ achieved a Coverage Probability (CP) of 86.5%, while the next best competitor ($CQS_{cen}$) reached only 60%, and others fell as low as 11%.

Comparison of screening and selection methods under different censoring rates.

3.2 Robustness Across Distributions

The robustness of the method was further tested across different error distributions, including Normal, Logistic, and Weibull. As detailed in Figure 2, $SRCS_{cen}$ consistently identified the true active variables (indicated by the $P_{all}$ and $m_{10}$ metrics) more reliably than competing methods across all tested distributions and censoring levels.

Performance of screening methods across various error distributions and censoring rates.

3.3 Real-World Application: Lymphoma Data

When applied to Mantle Cell Lymphoma microarray data (92 patients, 6,312 genes), $SRCS_{cen}$ identified 20 key genes. Notably, it successfully identified the cell division cycle 2 gene (Gene 28990), a well-documented marker in cancer progression, demonstrating its practical utility in biological discovery.

4. Discussion and Conclusion

$SRCS_{cen}$ represents a significant advancement in ultrahigh-dimensional feature screening. Its primary strength lies in its robustness; by relying on ranks rather than absolute values, it remains immune to outliers and heavy-tailed noise. Furthermore, the imputation strategy effectively extracts signal from highly censored datasets where traditional methods fail.

Future work may address the computational complexity of the local Kaplan-Meier estimation for datasets with millions of variables and explore the relaxation of the independent censoring assumption. Nevertheless, $SRCS_{cen}$ provides a powerful, model-free tool for high-dimensional survival analysis and related fields.

Summary updated

BibTeX
@inproceedings{wang2023spearman,
  author = {Hongni Wang and Jingxin Yan and Xiaodong Yan},
  title = {{Spearman Rank Correlation Screening for Ultrahigh-Dimensional Censored Data}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i8.26204},
}
Figure