Linear Causal Representation Learning by Topological Ordering, Pruning, and Disentanglement

Hao Chen, Lin Liu, Yu Guang Wang

Abstract

Causal representation learning (CRL) has garnered increasing interest from the causal inference and artificial intelligence community due to its capability of disentangling potentially complex data-generating mechanisms into causally interpretable latent features. In this paper, we further contribute to the CRL literature by focusing on the stylized linear structural causal model over the latent features and assuming a linear mixing function that maps latent features to the observed data. Existing linear CRL methods often rely on stringent assumptions, such as accessibility to single-node interventional data or restrictive distributional constraints. We propose a novel linear CRL algorithm, CREATOR, that operates under weaker assumptions about environment heterogeneity and data-generating distributions while still recovering latent causal features up to an equivalence class. We validate our algorithm via synthetic experiments and an interpretability analysis of large language models (LLMs), demonstrating its superiority over competing methods.

Introduction

In the quest to make AI more interpretable and robust, researchers are turning to Causal Representation Learning (CRL). The goal is to move beyond simple pattern matching and instead identify the underlying "causal" features that generate data—such as identifying that "lighting" and "object shape" are the latent causes of an image's pixels.

However, most existing linear CRL methods require single-node interventions or assume that "background noise" follows the exact same distribution across different environments. In reality, noise is rarely consistent, and interventions are often messy or impossible to obtain. This paper addresses this gap by proposing a method that recovers causal structures even when noise distributions vary and interventions are multi-node or non-existent.

Methodology

The authors propose CREATOR (Causal REpresentation leArning via Topological Ordering, Pruning, and Disentanglement). The algorithm operates in three distinct phases:

1. Topological Ordering & Feature Recovery

Instead of trying to find every latent variable simultaneously, CREATOR uses Independent Component Analysis (ICA) and the Hilbert-Schmidt Independence Criterion (HSIC) to find "root" nodes—variables that aren't caused by anything else. It then "peels" these nodes off one by one, projecting the data onto the remaining space to find the next node in the causal sequence.

Overview of the CREATOR algorithm's iterative process for recovering latent causal features through topological ordering and projection.

2. DAG Pruning

The initial step might suggest too many connections. This phase uses regression and rank-based analysis to identify and remove "spurious" edges, refining the causal graph (Directed Acyclic Graph or DAG).

3. Feature Disentanglement

Finally, the algorithm aligns the recovered features with the identified causal structure to ensure they are interpretable and match the ground truth up to a standard equivalence class.

Experimental Evaluation

The authors evaluated CREATOR using two primary approaches:

  • Synthetic Datasets: Causal models were generated using Erdős-Rényi random graphs with latent dimensions ranging from $d=2$ to $d=9$, tested against various non-Gaussian noise distributions (Laplace, Exponential, Beta) to simulate real-world heterogeneity.
  • LLM Case Study: To test the "Linear Representation Hypothesis," they extracted the last hidden states from several open-source LLMs (Llama 3.1, TinyLlama, Phi-3-Mini, GPT-Neo, and BLOOM) processing stories generated by GPT-4 and DeepSeek.

Key Results and Insights

  • Superior Accuracy: In synthetic tests, CREATOR consistently outperformed the baseline (LiNGCReL) in both LocR² (latent feature recovery) and Structural Hamming Distance (SHD) (graph identification accuracy).
  • LLM Interpretability: CREATOR successfully identified the expected causal flow of a story (Background $\rightarrow$ Condition $\rightarrow$ Ending) across most tested LLMs. For example, it correctly identified all causal edges for Llama Guard and TinyLlama, whereas the baseline method failed on several models.
  • Topological Order as an Anchor: The most powerful idea in the paper is that correctly inferring the order in which variables cause one another allows researchers to bypass the need for complex distributional assumptions. The order itself provides the constraints needed for identification.

Limitations and Future Work

While CREATOR is robust to noise heterogeneity, it currently assumes a linear relationship between latent features and observed data. While this holds for many LLM applications, it may fail for more complex, non-linear data like raw image pixels. Future work will focus on extending CREATOR to non-linear Structural Causal Models (SCMs) and exploring its use in "latent concept" recovery with fewer distinct environments.

Summary updated

Abstract

Causal representation learning (CRL) has garnered increasing interests from the causal inference and artificial intelligence community, due to its capability of disentangling potentially complex data-generating mechanism into causally interpretable latent features, by leveraging the heterogeneity of modern datasets. In this paper, we further contribute to the CRL literature, by focusing on the stylized linear structural causal model over the latent features and assuming a linear mixing function that maps latent features to the observed data or measurements. Existing linear CRL methods often rely on stringent assumptions, such as accessibility to single-node interventional data or restrictive distributional constraints on latent features and exogenous measurement noise. However, these prerequisites can be challenging to satisfy in certain scenarios. In this work, we propose a novel linear CRL algorithm that, unlike most existing linear CRL methods, operates under weaker assumptions about environment heterogeneity and data-generating distributions while still recovering latent causal features up to an equivalence class. We further validate our new algorithm via synthetic experiments and an interpretability analysis of large language models (LLMs), demonstrating both its superiority over competing methods in finite samples and its potential in integrating causality into AI.
BibTeX
@inproceedings{chen2026linear,
  author = {Hao Chen and Lin Liu and Yu Guang Wang},
  title = {{Linear Causal Representation Learning by Topological Ordering, Pruning, and Disentanglement}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=4COS6SMf8r},
}
Figure