cdcsis: Conditional Distance Correlation Based Feature Screening and Conditional Independence Inference

Wenhao Hu, Mian Huang, Wenliang Pan, Xueqin Wang, Canhong Wen, Yuan Tian, Heping Zhang, Jin ZhuOpenalex Unknown 2014 · DOI · OpenAlex

Abstract

Conditional distance correlation is a novel conditional dependence measurement of two multivariate random variables given a confounding variable. This package provides conditional distance correlation, performs the conditional distance correlation sure independence screening procedure for ultrahigh dimensional data, and conducts conditional distance covariance test for conditional independence assumption of two multivariate variables.

Introduction

In modern data science, researchers frequently encounter two major hurdles: non-linearity and high dimensionality. Standard tools like Pearson’s correlation often fail to capture complex, non-linear relationships. While "Distance Correlation" was developed to address marginal independence in such cases, it struggles when accounting for a third variable—a confounder.

For instance, determining if a specific gene expression ($X$) affects a disease outcome ($Y$) requires controlling for confounding factors like age or sex ($Z$). If $X$ and $Y$ are multivariate and their relationship is non-linear, traditional partial correlation or linear models fall short. Furthermore, in "ultrahigh-dimensional" settings where predictors vastly outnumber samples, a method is needed to quickly filter out irrelevant features while adjusting for known confounders.

Methodology

The cdcsis R package implements three core functionalities based on recent statistical theory to address these challenges:

  1. Conditional Distance Correlation (CDC): A model-free metric that quantifies the dependence between multivariate random variables $X$ and $Y$ given $Z$. It is robust to various functional forms, including complex manifolds.
  2. CDC-SIS (Sure Independence Screening): A screening procedure designed for "Big $p$, Small $n$" problems. It ranks features based on their conditional distance correlation with the response, reducing thousands of variables to a manageable subset while adjusting for confounding covariates.
  3. Conditional Distance Covariance Test: A nonparametric test for the null hypothesis of conditional independence. It utilizes a local bootstrap procedure to calculate p-values, ensuring validity even when underlying distributions are unknown.

Implementation and Parameters

The package is computationally optimized using Rcpp for core calculations and supports multi-threading. The functions are designed to be flexible, accepting either raw data or pre-computed distance matrices.

Parameters for conditional distance statistics

As shown in the parameter definitions, the "Conditional" aspect of the correlation relies on a Gaussian kernel. The package defaults the bandwidth (width) using standard rules like stats::bw.nrd0, though users can tune this for specific sensitivity requirements.

Statistical Inference and Results

The package provides a unified interface for complex statistical tasks through functions such as cdcor, cdcov, and cdcov.test. For feature screening, cdcsis returns a list of indices (ix) for the most important variables, facilitating immediate integration into machine learning pipelines.

Components of the cdcov.test return object

The cdcov.test function returns a standard R "htest" object. This object includes the test statistic and a bootstrap-derived p-value, allowing researchers to rigorously test for conditional independence without assuming linear relationships.

Conclusion

While distance-based methods are computationally intensive—typically $O(n^2)$—the cdcsis package mitigates this through Rcpp and multi-threading. For researchers dealing with high-dimensional "omics" data, complex physical sensors, or any field where confounding variables are a reality, cdcsis offers a mathematically rigorous yet accessible path to discovery. It successfully identifies dependent variables even when relationships are buried under complex transformations that would baffle traditional linear regressors.

Summary updated

Abstract

Conditional distance correlation &lt;<a href="https://doi.org/10.1080%2F01621459.2014.993081" target="_top">doi:10.1080/01621459.2014.993081</a>&gt; is a novel conditional dependence measurement of two multivariate random variables given a confounding variable. This package provides conditional distance correlation, performs the conditional distance correlation sure independence screening procedure for ultrahigh dimensional data &lt;<a href="https://www3.stat.sinica.edu.tw/statistica/J28N1/J28N114/J28N114.html" target="_top">https://www3.stat.sinica.edu.tw/statistica/J28N1/J28N114/J28N114.html</a>&gt;, and conducts conditional distance covariance test for conditional independence assumption of two multivariate variable.
BibTeX
@inproceedings{hu2014cdcsis,
  author = {Wenhao Hu and Mian Huang and Wenliang Pan and Xueqin Wang and Canhong Wen and Yuan Tian and Heping Zhang and Jin Zhu},
  title = {{cdcsis: Conditional Distance Correlation Based Feature Screening and Conditional Independence Inference}},
  booktitle = {Openalex Unknown},
  year = {2014},
  doi = {10.32614/cran.package.cdcsis},
}
Figure