High-Dimensional Conditional Independence Testing for Causal Discovery

This direction is split into other directions: residual-based conditional independence testing. Its last version stays readable and citable here.

Version 1 of 1120 papers131 conceptsfrozen 2026-10-04

Conditional independence (CI) testing is the mathematical engine of causal inference. To determine if a specific variable $X$ causes $Y$, we must check if they remain dependent after "controlling for" a set of background variables $Z$. If $X$ and $Y$ become independent given $Z$, we can often conclude that $X$ does not have a direct causal influence on $Y$.

However, modern datasets present a "curse of dimensionality." As the number of variables in the conditioning set $Z$ grows, the data becomes sparse, making it nearly impossible to find enough samples with similar values of $Z$ to perform a reliable test. Researchers are developing new strategies to make CI testing scalable, robust, and computationally efficient in these high-dimensional regimes.

The Challenge of the Conditioning Set

Traditional CI tests, particularly non-parametric kernel-based methods, struggle when $Z$ is high-dimensional. As Biwei Huang and Oliver Schacht note in FastKCI, these methods often face $O(n^3)$ complexity and a rapid loss of statistical power. When $Z$ includes dozens or hundreds of variables, the "volume" of the space increases exponentially, and the available data points drift apart. This makes it difficult to distinguish between a true causal link and a spurious correlation masked by high-dimensional noise.

Transforming Independence into Classification

One of the most intuitive breakthroughs in this field is reframing CI testing as a supervised learning problem. Instead of calculating complex integrals or kernel matrices, researchers like Karthikeyan Shanmugam and Rajat Sen proposed the Mimic and Classify meta-algorithm.

The intuition is simple: if $X$ and $Y$ are independent given $Z$, then the joint distribution $P(X, Y, Z)$ should look exactly like a "shuffled" version where $X$ and $Y$ are decoupled but still respect the distribution of $Z$. By using a generative model (like a GAN) to "mimic" this independent distribution and then training a binary classifier to distinguish the "real" data from the "mimicked" data, we can detect dependencies. If the classifier can't tell them apart, the variables are likely independent. This approach, also seen in the Classifier Conditional Independence Test (CCIT) by Alexandros G. Dimakis, allows researchers to leverage high-performance tools like XGBoost and Deep Neural Networks that are naturally resistant to the curse of dimensionality.

Distillation and Residual Information

Another powerful strategy involves "subtracting" the influence of $Z$ before testing for independence. In Fast and Powerful Conditional Randomization Testing via Distillation, Molei Liu and Aaditya Ramdas introduce dCRT. This method "distills" the relationship between the variables into low-dimensional representations.

Similarly, the Decoupled Independence Test (DIET) by Aahlad Manas Puli and Rajesh Ranganath transforms variables into "information residuals." By using a model to predict $X$ from $Z$ and $Y$ from $Z$, and then testing the independence of the errors (the residuals), the test focuses only on the unique information that $X$ and $Y$ share. This "partialling out" of $Z$ simplifies the high-dimensional problem into a much easier marginal independence test.

Recursive and Greedy Structure Learning

Rather than perfecting a single, massive test, some researchers focus on reducing the number and complexity of tests required to map a causal network. Boaz Lerner developed Recursive Autonomy Identification (RAI), which breaks a large graph into "autonomous sub-structures." By recursively decomposing the problem, RAI avoids high-order CI tests that are statistically unreliable.

A similar logic drives MARVEL, an algorithm by Ehsan Mokhtarian and Negar Kiyavash. MARVEL identifies "removable variables" to simplify the graph step-by-step. These recursive strategies ensure that the most difficult, high-dimensional tests are only performed when absolutely necessary, significantly improving both speed and accuracy in sparse networks.

Latent Spaces and Sufficient Statistics

Recent work has also explored mapping complex data into simpler mathematical spaces. Bao Duong and Thin Nguyen introduced LCIT, which uses Conditional Normalizing Flows to map high-dimensional variables into Gaussian latent spaces. In this simplified space, testing for independence becomes a straightforward correlation check.

For cases where the data distribution is entirely unknown, Xiaotong Lin and colleagues proposed MS-CRT, which uses sufficient statistics to detect weak joint signals in high-dimensional settings. By focusing on the most informative summaries of the data, these methods maintain statistical power even when the number of variables exceeds the number of observations.

Papers in this version

Showing 60 of 120 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_high_dimensional_conditional_independenc_6c145c5f,
  title = {High-Dimensional Conditional Independence Testing for Causal Discovery},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-dimensional-conditional-independence-testing-for-causal-discovery/rdr_6c145c5f9c4f4e239ebab06a07d43530},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_high_dimensional_conditional_independenc_6c145c5f_v1,
  title = {High-Dimensional Conditional Independence Testing for Causal Discovery},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-dimensional-conditional-independence-testing-for-causal-discovery/rdr_6c145c5f9c4f4e239ebab06a07d43530/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}