Abstract
Conditional distance correlation is a novel conditional dependence measurement of two multivariate random variables given a confounding variable. This package provides conditional distance correlation, performs the conditional distance correlation sure independence screening procedure for ultrahigh dimensional data, and conducts conditional distance covariance test for conditional independence assumption of two multivariate variables.
Introduction
In modern data science, researchers frequently encounter two major hurdles: non-linearity and high dimensionality. Standard tools like Pearson’s correlation often fail to capture complex, non-linear relationships. While "Distance Correlation" was developed to address marginal independence in such cases, it struggles when accounting for a third variable—a confounder.
For instance, determining if a specific gene expression ($X$) affects a disease outcome ($Y$) requires controlling for confounding factors like age or sex ($Z$). If $X$ and $Y$ are multivariate and their relationship is non-linear, traditional partial correlation or linear models fall short. Furthermore, in "ultrahigh-dimensional" settings where predictors vastly outnumber samples, a method is needed to quickly filter out irrelevant features while adjusting for known confounders.
Methodology
The cdcsis R package implements three core functionalities based on recent statistical theory to address these challenges:
- Conditional Distance Correlation (CDC): A model-free metric that quantifies the dependence between multivariate random variables $X$ and $Y$ given $Z$. It is robust to various functional forms, including complex manifolds.
- CDC-SIS (Sure Independence Screening): A screening procedure designed for "Big $p$, Small $n$" problems. It ranks features based on their conditional distance correlation with the response, reducing thousands of variables to a manageable subset while adjusting for confounding covariates.
- Conditional Distance Covariance Test: A nonparametric test for the null hypothesis of conditional independence. It utilizes a local bootstrap procedure to calculate p-values, ensuring validity even when underlying distributions are unknown.
Implementation and Parameters
The package is computationally optimized using Rcpp for core calculations and supports multi-threading. The functions are designed to be flexible, accepting either raw data or pre-computed distance matrices.
As shown in the parameter definitions, the "Conditional" aspect of the correlation relies on a Gaussian kernel. The package defaults the bandwidth (width) using standard rules like stats::bw.nrd0, though users can tune this for specific sensitivity requirements.
Statistical Inference and Results
The package provides a unified interface for complex statistical tasks through functions such as cdcor, cdcov, and cdcov.test. For feature screening, cdcsis returns a list of indices (ix) for the most important variables, facilitating immediate integration into machine learning pipelines.
The cdcov.test function returns a standard R "htest" object. This object includes the test statistic and a bootstrap-derived p-value, allowing researchers to rigorously test for conditional independence without assuming linear relationships.
Conclusion
While distance-based methods are computationally intensive—typically $O(n^2)$—the cdcsis package mitigates this through Rcpp and multi-threading. For researchers dealing with high-dimensional "omics" data, complex physical sensors, or any field where confounding variables are a reality, cdcsis offers a mathematically rigorous yet accessible path to discovery. It successfully identifies dependent variables even when relationships are buried under complex transformations that would baffle traditional linear regressors.
Summary updated