Identifiability Guarantees for Causal Disentanglement from Purely Observational Data

Ryan Welch, Jiaqi Zhang, Caroline Uhler

TL;DR

This paper proves that we can recover latent causal factors from purely observational data—no interventions required—provided the underlying causal mechanisms are nonlinear with additive Gaussian noise. The authors demonstrate that while perfect "disentanglement" of every individual variable isn't always possible, we can identify these factors up to their respective "causal layers" using a practical algorithm based on score estimation and quadratic programming.

What Problem Does This Paper Address?

In the quest to make AI more interpretable and robust, researchers have turned to causal disentanglement. The goal is to take high-dimensional, messy data (like pixels in an image) and work backward to find the latent causal variables (like "object position" or "lighting angle") that generated it.

Until now, the prevailing wisdom was that you couldn't achieve this without interventional data—essentially "poking" the system to see what changes. For example, to understand a robot arm, you might need to manually move a joint to see how the pixels change. However, in many fields like medicine or economics, performing such interventions is often impossible, unethical, or prohibitively expensive.

The authors tackle a fundamental question: How much can we learn about the hidden causal structure of a system just by watching it?

Methods

The paper focuses on a specific but flexible model: Nonlinear Additive Gaussian Noise (NLG) models with linear mixing.

  1. The Setup: Latent variables ($Z$) influence each other through nonlinear functions but are observed through a linear "lens" ($H$) to produce the observed data ($X$).
  2. Score Matching: The core technique involves analyzing the "score function"—the gradient of the log-probability of the observed data. Specifically, the authors look at the Jacobian (the matrix of second-order derivatives) of this score.
  3. The "Leaf Node" Principle: A key insight is that the variance of certain elements in this Jacobian matrix hits zero only for "leaf nodes" (variables with no children in the causal graph).
  4. The Algorithm: They translate this theory into a two-step practical approach:
  5. Step 1: Solve a sequence of Quadratic Programs (QCQPs) to find the mixing matrix $H$. This allows them to recover the latent variables up to their "upstream layers."
    • Step 2: Use nonlinear regression to peel back the layers and recover the exogenous noise terms (the unique "source" of randomness for each variable).
Figure 1: The considered data-generating pro- cess. The latent variables Z follow a nonlinear causal model with additive Gaussian noises. We observe them after an unknown linear mixing (gray edges).

Datasets

Because this is a foundational theoretical paper, the authors evaluate their work using synthetic simulations where the ground truth is known.

  • Causal Structures: They tested on a 4-node Line Graph ($Z_1 \to Z_2 \to Z_3 \to Z_4$) and a Y-structure.
  • Score Estimators: They compared "Oracle" scores (perfect knowledge) against practical estimators like the Second-order Stein estimator and Sliced Score Matching with Variance Reduction (SSM-VR).

Key Insights and Ideas

  • Identifiability up to Layers: The most significant "aha" moment is the characterization of what can be learned. You can't always separate two variables that are in the same "causal layer" (e.g., two independent causes of the same effect), but you can separate them from their ancestors and descendants.
  • Asymmetry is a Gift: In linear models with Gaussian noise, causes and effects look the same. But by assuming nonlinearity, the authors exploit an asymmetry in the data distribution that reveals the causal direction.
  • No Graphical Restrictions: Unlike many previous works, this method doesn't require the causal graph to be sparse or follow a specific structure (like a tree).
Figure 2: Layers of the causal DAG. A latent variable is contained in layer(k) if its longest path to a leaf node is k.

Key Results

The paper provides a rigorous comparison against prior work, highlighting that they achieve identifiability without the "crutches" of interventions or structural mixing assumptions (like the "pure child" assumption).

  • Theoretical Guarantee: They prove that latent variables are identifiable up to a "layer-wise transformation."
  • Empirical Validation: In simulations with perfect score information, the algorithm perfectly recovered the root nodes and disentangled the noise variables.
  • Practical Performance: Using real-world score estimators (Stein/SSM-VR), the algorithm still captured meaningful causal representations, though accuracy naturally decreased as the score estimation became noisier. For instance, on a line graph, the root node was consistently identified even with noisy estimates.

Limitations and Future Work

  • Linear Mixing: The current theory relies on the observation process being linear ($X=HZ$). While the authors suggest this might extend to some polynomial cases, general nonlinear mixing remains a "holy grail" for future research.
  • Score Estimation Sensitivity: The algorithm's success is heavily dependent on the quality of the score estimator. Estimating second-order gradients from finite samples is notoriously difficult and computationally expensive.
  • Gaussian Assumption: The results are specific to Gaussian noise. Exploring whether these guarantees hold for more general noise distributions is a clear next step.

By showing that interventions aren't strictly necessary for causal discovery in the latent space, this work opens the door for more interpretable AI in fields where we are forced to be passive observers.

Summary updated

Abstract

Causal disentanglement aims to learn about latent causal factors behind data, holding the promise to augment existing representation learning methods in terms of interpretability and extrapolation. Recent advances establish identifiability results assuming that interventions on (single) latent factors are available; however, it remains debatable whether such assumptions are reasonable due to the inherent nature of intervening on latent variables. Accordingly, we reconsider the fundamentals and ask what can be learned using just observational data. We provide a precise characterization of latent factors that can be identified in nonlinear causal models with additive Gaussian noise and linear mixing, without any interventions or graphical restrictions. In particular, we show that the causal variables can be identified up to a layer-wise transformation and that further disentanglement is not possible. We transform these theoretical results into a practical algorithm consisting of solving a quadratic program over the score estimation of the observed data. We provide simulation results to support our theoretical guarantees and demonstrate that our algorithm can derive meaningful causal representations from purely observational data.
BibTeX
@inproceedings{welch2024identifiability,
  author = {Ryan Welch and Jiaqi Zhang and Caroline Uhler},
  title = {{Identifiability Guarantees for Causal Disentanglement from Purely Observational Data}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=M20p6tq9Hq},
}
Figure