Dynamical causality under invisible confounders

Jinling Yan, Shao‐Wu Zhang, Chihao Zhang, Weitian Huang, Jifan Shi, Luonan Chen

TL;DR

Researchers have developed a new method called Causality under Invisible Confounders (CIC) that accurately identifies causal relationships in complex systems even when key influencing variables are unobserved. By combining Takens’ delay embedding theory with deep learning, the method can distinguish true cause-and-effect from "spurious" correlations and even reconstruct the signals of hidden variables using only two observed time-series.

What Problem Does This Paper Address?

In complex systems—like the human brain, global ecosystems, or gene networks—everything seems connected to everything else. A major headache for scientists is the "invisible confounder" problem. Imagine variables $X$ and $Y$ both fluctuate in sync. Is $X$ causing $Y$, or is a hidden variable $Z$ (the "confounder") driving both of them?

Standard statistical tools often mistake the influence of $Z$ for a direct link between $X$ and $Y$, leading to "spurious causality." While some modern methods can handle visible confounders by "controlling" for them, they fall apart when the confounder is unobserved or when the system is nonlinear and "non-separable" (meaning the variables are so intertwined they can't be easily teased apart). The authors aim to solve this "long-standing problem" in data science: inferring causality from just two observed variables in a high-dimensional, partially hidden system.

Methods

The authors introduce the CIC method, which operates on two main pillars:

  1. Orthogonal Decomposition Theorem: Instead of looking at variables in their original state, the method uses Takens’ delay embedding theory to map time-series data into a higher-dimensional "embedding space." They prove mathematically that this space can be decomposed into two orthogonal (independent) subspaces: a shared subspace (containing information common to both variables, i.e., the confounder) and a private subspace (containing information unique to the cause).
  2. VAE-based Implementation: To actually perform this decomposition on messy, nonlinear real-world data, they use a Variational Autoencoder (VAE). The VAE acts as a "disentangler," learning to separate the shared latent signals from the private ones.

By measuring the "strength" of these subspaces, they calculate a Causal Index. If the private subspace of the "cause" variable effectively reconstructs the "effect" variable, causality is confirmed. If only the shared subspace is active, it’s a sign of a hidden confounder.

Datasets

The researchers validated CIC across a diverse range of synthetic and real-world scenarios:

  • Simulated Benchmarks: 3D and 8D logistic systems with varying noise and coupling strengths.
  • DREAM4 Challenge: Standardized benchmarks for gene regulatory networks (10-node and 100-node versions).
  • Ecological Data: An 8-year study of a plankton food chain in the Baltic Sea.
  • Public Health: Air pollution (SO2, NO2, etc.) and cardiovascular disease hospital admissions in Hong Kong.
  • Biological Systems: Rat circadian rhythm gene expression data and monkey brain neural datasets.

Key Insights and Ideas

  • The "Two-Variable" Miracle: Most causal inference methods require you to measure every possible variable to "rule out" confounders. CIC’s theoretical foundation allows it to detect causality and identify the presence of confounders using only two observed variables.
  • Reconstructing the Invisible: Perhaps the most "aha" moment is that CIC doesn't just ignore the hidden confounder; it reconstructs it. The shared latent space learned by the VAE often shows a high correlation (up to 0.7) with the actual, unobserved ground-truth variable.
  • Solving Non-Separability: In nonlinear dynamics, variables are often "non-separable," meaning they don't satisfy the assumptions of traditional methods like Granger Causality. By moving to the delay embedding space, CIC makes these intertwined dynamics separable and measurable.

Key Results

  • Superior Accuracy: In the DREAM4 gene network challenge, CIC achieved an AUC (Area Under the Curve) between 0.8 and 0.9, significantly outperforming eight other popular methods (like CCM, Granger Causality, and Transfer Entropy), which hovered around 0.7.
  • Robustness to Noise: In 3D logistic systems, CIC correctly distinguished direct causality from "common driver" scenarios even as noise levels increased, a feat the authors claim is "not achievable with other methods."
  • Real-World Success: In the Hong Kong air pollution study, CIC correctly identified that RSP (particulates) and SO2 cause cardiovascular disease, while correctly identifying that the apparent link between NO2 and heart disease was actually a confounder effect—avoiding a false positive that tripped up other algorithms.

Limitations and Future Work

Despite its power, the authors note a few hurdles:

  • Sample Size: The framework relies on temporal information, meaning it needs a decent amount of data points (at least 10, but ideally more) to maintain performance.
  • Computational Cost: Identifying confounders requires training two separate VAE models (one for $X \to Y$ and one for $Y \to X$), which is more resource-intensive than simple statistical tests.
  • Non-Intervention Only: Currently, the study focuses on "passive" observations. The authors highlight that extending this to "intervention" systems (where researchers actively manipulate variables) is a key goal for future research.

Summary updated

Abstract

Causality inference is prone to spurious causal interactions, due to the substantial confounders in a complex system. While many existing methods based on the statistical methods or dynamical methods attempt to address misidentification challenges, there remains a notable lack of effective methods to infer causality, in particular in the presence of invisible/unobservable confounders. As a result, accurately inferring causation with invisible confounders remains a largely unexplored and outstanding issue in data science and AI fields. In this work, we propose a method to overcome such challenges to infer dynamical causality under invisible confounders (CIC method) and further reconstruct the invisible confounders from time-series data by developing an orthogonal decomposition theorem in a delay embedding space. The core of our CIC method lies in its ability to decompose the observed variables not in their original space but in their delay embedding space into the common and private subspaces respectively, thereby quantifying causality between those variables both theoretically and computationally. This theoretical foundation ensures the causal detection for any high-dimensional system even with only two observed variables under many invisible confounders, which is actually a long-standing problem in the field. In addition to the invisible confounder problem, such a decomposition actually makes the intertwined variables separable in the embedding space, thus also solving the non-separability problem of causal inference. Extensive validation of the CIC method is carried out using various real datasets, and the experimental results demonstrates its effectiveness to reconstruct real biological networks even with unobserved confounders.
BibTeX
@misc{yan2024dynamical,
  author = {Jinling Yan and Shao‐Wu Zhang and Chihao Zhang and Weitian Huang and Jifan Shi and Luonan Chen},
  title = {{Dynamical causality under invisible confounders}},
  year = {2024},
  eprint = {2408.05584},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.05584},
}
Figure