On Causally Disentangled Representations

Abbavaram Gowtham Reddy, Benin Godfrey L, Vineeth N. Balasubramanian

Abstract

Representation learners that disentangle factors of variation are essential for fairness and interpretability. While traditional models assume factor independence, real-world data is often confounded. This paper introduces a causal perspective on disentanglement, proposing a framework grounded in Structural Causal Models (SCMs). We define the notion of a disentangled causal process and introduce two novel evaluation metrics: Unconfoundedness (UC) and Counterfactual Generativeness (CG). To support this research, we present CANDLE, a realistic confounded dataset. Our empirical study demonstrates that standard unsupervised models struggle with confounded data, while weak supervision significantly improves causal disentanglement.

Introduction

Disentangled representation learning seeks to identify the underlying "factors of variation"—such as shape, color, or position—that generate observed data. Historically, the field has relied on the assumption that these factors are statistically independent. However, in practical applications, factors are frequently confounded by observed or unobserved variables. For instance, in medical imaging, patient age and disease severity may be correlated due to genetic factors.

Ignoring these causal dependencies leads models to learn spurious correlations rather than true generative mechanisms. This paper argues for a shift toward "causal disentanglement," where representations are evaluated based on their ability to capture the underlying causal structure of the data-generating process.

The Causal Generative Process

We model the data-generating process using a two-level Structural Causal Model. In this regime, a set of confounders $C$ influences the generative factors $G$, which in turn produce the observation $X$. This framework allows for factors to be statistically dependent while maintaining their identity as distinct causal entities.

A causal process where generative factors $G$ are influenced by confounders $C$ to produce observation $X$.

Metrics for Causal Disentanglement

To evaluate models within this causal framework, we propose two metrics that move beyond simple correlation analysis:

  1. Unconfoundedness (UC): This metric assesses the encoder's ability to map distinct generative factors to non-overlapping sets of latent dimensions. By utilizing the Jaccard similarity coefficient, UC penalizes models that conflate multiple factors within the same latent space, even when those factors are correlated in the training data.
  2. Counterfactual Generativeness (CG): This generator-side metric evaluates the model's ability to perform interventions. It checks whether intervening on a specific latent dimension (e.g., changing "object color") affects only the intended attribute in the generated image while leaving other attributes (e.g., "background scene") unchanged.
The UC metric relates factors to latents, while CG evaluates the quality of generated counterfactuals.

The CANDLE Dataset and Weak Supervision

We introduce the CANDLE (Causal ANalysis in DisentangLed rEpresentations) dataset to benchmark these metrics. CANDLE consists of over 12,000 images with realistic HDRI backgrounds and six generative factors (Light, Scene, Object, Size, Color, Angle). Crucially, it incorporates both observed and unobserved confounding, such as constraints where certain large objects are excluded from small indoor scenes.

To address the challenges of complex scenes, we propose SS-FVAE-BB, a semi-supervised variant of Factor-VAE. This model utilizes bounding boxes as weak supervision. By adding a reconstruction loss term focused on the foreground, the model is prevented from "erasing" small but causally significant factors in favor of minimizing global reconstruction error on the background.

Experimental Results and Insights

Our experiments reveal that state-of-the-art unsupervised models, such as $\beta$-TCVAE, perform poorly on the CANDLE benchmark, often achieving UC scores below 0.15. This indicates that without explicit causal considerations, models fail to disentangle confounded factors.

In contrast, the proposed SS-FVAE-BB method improved the UC score on CANDLE to 0.27. Furthermore, our analysis on the dSprites dataset showed that the UC metric is more sensitive than traditional metrics (like DCI), revealing that models often begin to "mix" factors as the capacity of the latent space increases.

Conclusion

Moving beyond the independence assumption is critical for the next generation of representation learners. By grounding disentanglement in causality, we provide a more robust framework for learning representations that are not only statistically efficient but also causally accurate. Future work will explore deeper hierarchical causal structures and the potential for leveraging temporal data to learn these representations without supervision.

Summary updated

Abstract

Representation learners that disentangle factors of variation have already proven to be important in addressing various real world concerns such as fairness and interpretability. Initially consisting of unsupervised models with independence assumptions, more recently, weak supervision and correlated features have been explored, but without a causal view of the generative process. In contrast, we work under the regime of a causal generative process where generative factors are either independent or can be potentially confounded by a set of observed or unobserved confounders. We present an analysis of disentangled representations through the notion of disentangled causal process. We motivate the need for new metrics and datasets to study causal disentanglement and propose two evaluation metrics and a dataset. We show that our metrics capture the desiderata of disentangled causal process. Finally we perform an empirical study on state of the art disentangled representation learners using our metrics and dataset to evaluate them from causal perspective.
BibTeX
@inproceedings{reddy2022causally,
  author = {Abbavaram Gowtham Reddy and Benin Godfrey L and Vineeth N. Balasubramanian},
  title = {{On Causally Disentangled Representations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=wgE1g4qsUw2},
}
Figure