Determining whether two variables $X$ and $Y$ are independent given a third variable $Z$ is a fundamental challenge in causal discovery and machine learning. In theory, this requires checking if $P(X, Y | Z) = P(X | Z)P(Y | Z)$ for every possible value of $Z$. In practice, especially when $Z$ is high-dimensional or continuous, we rarely see the same value of $Z$ twice, making direct conditioning impossible.
A growing body of research addresses this "curse of dimensionality" by transforming the problem: instead of testing independence within slices of $Z$, we "subtract" the influence of $Z$ from both $X$ and $Y$ and test the independence of what remains. These "leftovers" are known as residuals.
The Core Intuition: Subtracting the Confounder
The central idea is to decouple the modeling of the conditioning variable $Z$ from the testing of the relationship between $X$ and $Y$. If $Z$ is the only reason $X$ and $Y$ appear related, then once we account for $Z$'s influence, the remaining variation in $X$ should have nothing to do with the remaining variation in $Y$.
As Hao Zhang, Kun Zhang, and colleagues demonstrate in Causal Discovery Using Regression-Based Conditional Independence Tests, we can use nonlinear regression to model $X = f(Z) + \epsilon_X$ and $Y = g(Z) + \epsilon_Y$. If $X$ and $Y$ are conditionally independent given $Z$, then the residuals $\epsilon_X$ and $\epsilon_Y$ should be independent of each other. This shifts the difficulty from a complex conditional test to a simpler marginal independence test between two variables.
Beyond Linear Residuals
While simple subtraction works for additive noise, real-world data is often more complex. Researchers have developed more sophisticated ways to "distill" the information in $Z$:
- Kernel Embeddings: Qinyi Zhang, Dino Sejdinovic, and others introduced KRESIT in Feature-to-Feature Regression for a Two-Step Conditional Independence Test. They perform regression in Reproducing Kernel Hilbert Spaces (RKHS), allowing the test to handle complex patterns like heteroscedasticity where the "noise" level changes depending on $Z$.
- Information Residuals: Aahlad Manas Puli, Rajesh Ranganath, and their team proposed DIET in Conditional independence testing with marginal dependence measures of residual information. Instead of standard regression, they use conditional cumulative distribution functions (CDFs) to transform variables into "information residuals" that represent the variation $Z$ cannot explain.
- Normalizing Flows: In Normalizing flows for conditional independence testing, Bao Duong and Thin Nguyen use generative models to map high-dimensional variables into latent Gaussian representations that are statistically stripped of $Z$'s influence.
Computational Efficiency and Distillation
Traditional conditional tests are often slow because they require repeated model fitting or complex permutations. The residual-based approach offers a significant speedup by "distilling" the nuisance variables once.
Molei Liu, Aaditya Ramdas, and colleagues formalize this in Fast and Powerful Conditional Randomization Testing via Distillation. By distilling the effect of $Z$ into low-dimensional residuals, they can perform randomization tests much faster than traditional methods. Similarly, the FRCIT method described in Testing Independence Between Linear Combinations for Causal Discovery achieves massive speedups by focusing on the statistical moments (like kurtosis) of linear residuals rather than expensive kernel computations.
Why It Works: The Power of Decoupling
The success of this direction stems from a simple trade-off: it replaces a hard statistical problem (conditional testing) with a well-understood engineering problem (regression or density estimation).
As noted by Anton Rask Lundborg and Rajen D. Shah in their work on the Generalised Hilbertian Covariance Measure (GHCM), while CI testing is "fundamentally impossible" without assumptions, residual-based methods provide a practical path forward. By leveraging modern machine learning to "clean" the data of confounding influences, we can isolate the specific signals needed to build accurate causal graphs and fair representations. This is exemplified by CIRCE, a method by Roman Pogodin and Arthur Gretton in Efficient Conditionally Invariant Representation Learning, which uses residual covariance to help neural networks ignore distractor variables.