On Causally Disentangled Representations

Abbavaram Gowtham Reddy, Benin Godfrey L, Vineeth N. Balasubramanian

Abstract

Representation learners that disentangle factors of variation have already proven to be important in addressing various real world concerns such as fairness and interpretability. Initially consisting of unsupervised models with independence assumptions, more recently, weak supervision and correlated features have been explored, but without a causal view of the generative process. In contrast, we work under the regime of a causal generative process where generative factors are either independent or can be potentially confounded by a set of observed or unobserved confounders. We present an analysis of disentangled representations through the notion of disentangled causal process. We motivate the need for new metrics and datasets to study causal disentanglement and propose two evaluation metrics and a dataset. We show that our metrics capture the desiderata of disentangled causal process. Finally, we perform an empirical study on state of the art disentangled representation learners using our metrics and dataset to evaluate them from causal perspective.

Introduction

The goal of disentangled representation learning is to identify and separate the underlying factors of variation in data—such as the shape, color, and position of an object in an image. Achieving this separation is critical for building interpretable, fair, and robust AI systems. However, most existing research operates under the simplifying assumption that these factors are statistically independent.

In real-world scenarios, factors are frequently confounded. For example, in medical imaging, a patient's age and the presence of certain symptoms might be correlated due to unobserved physiological factors. Existing benchmarks like dSprites fail to capture these complexities, and standard metrics often focus solely on the encoder's performance, neglecting the generator's ability to produce causally consistent counterfactuals. This paper addresses these gaps by introducing a formal causal framework for disentanglement, new evaluation metrics, and a more realistic dataset.

Methodology: Causal Disentanglement and Metrics

We extend the definition of a "disentangled causal process" to deep generative models, such as Variational Autoencoders (VAEs). To evaluate how well a model captures this process, we introduce two novel metrics:

  1. Unconfoundedness (UC): This metric assesses whether distinct generative factors are mapped to non-overlapping sets of latent dimensions. By utilizing the Jaccard similarity coefficient, UC penalizes models that conflate multiple real-world factors into the same internal latent variables.
  2. Counterfactual Generativeness (CG): This metric evaluates the model's ability to perform "what-if" reasoning. It measures whether modifying a specific latent variable (e.g., "color") results in a change only to the intended attribute in the generated image without affecting others (e.g., "shape").
Figure 2: Overview of the Unconfoundedness (UC) and Counterfactual Generativeness (CG) metrics. UC measures the mapping between factors G and latents Z, while CG evaluates the consistency of counterfactual generation.

Additionally, we propose SS-FVAE-BB (Semi-Supervised Factor-VAE with Bounding Box supervision). This model incorporates a specialized loss term that encourages the latent space to focus on foreground objects, improving disentanglement in complex scenes.

The CANDLE Dataset

To test models in causally complex environments, we introduce CANDLE (Causal ANalysis in DisentangLed rEpresentations). Generated using Blender, CANDLE consists of 12,546 images across six factors: light, scene, object, size, color, and angle.

Unlike previous datasets, CANDLE explicitly incorporates observed and unobserved confounders. For instance, object size is constrained by the scene type (e.g., large objects do not appear in small indoor rooms), and lighting interacts dynamically with the environment to produce realistic shadows and reflections.

Figure 6: Sample images from the CANDLE dataset. The dataset introduces realistic correlations and confounding factors between objects, scenes, and lighting conditions.

Experimental Results

Our empirical study reveals several key insights:

  • Failure of Standard Models: State-of-the-art unsupervised models like $\beta$-VAE and Factor-VAE struggle significantly on the CANDLE dataset. Their low UC and CG scores indicate a failure to disentangle factors when faced with complex backgrounds and confounding variables.
  • Metric Sensitivity: The UC and CG metrics successfully identified failures that traditional metrics (like IRS) missed. Specifically, some models appeared to "encode" factors correctly but failed to "generate" them independently, highlighting the importance of evaluating the generative path.
  • Benefits of Weak Supervision: The SS-FVAE-BB model outperformed unsupervised counterparts on CANDLE, demonstrating that minimal structural guidance (e.g., bounding boxes) can significantly aid the discovery of meaningful causal factors in complex data.

Conclusion

This work highlights the limitations of the independence assumption in representation learning and provides the tools necessary to move toward causally-aware AI. By introducing the UC and CG metrics alongside the CANDLE dataset, we provide a rigorous framework for evaluating disentanglement in the presence of confounders. Future research will focus on extending these methods to "in-the-wild" datasets and exploring more hierarchical causal structures.

Summary updated

Abstract

Representation learners that disentangle factors of variation have already proven to be important in addressing various real world concerns such as fairness and interpretability. Initially consisting of unsupervised models with independence assumptions, more recently, weak supervision and correlated features have been explored, but without a causal view of the generative process. In contrast, we work under the regime of a causal generative process where generative factors are either independent or can be potentially confounded by a set of observed or unobserved confounders. We present an analysis of disentangled representations through the notion of disentangled causal process. We motivate the need for new metrics and datasets to study causal disentanglement and propose two evaluation metrics and a dataset. We show that our metrics capture the desiderata of disentangled causal process. Finally, we perform an empirical study on state of the art disentangled representation learners using our metrics and dataset to evaluate them from causal perspective.
BibTeX
@inproceedings{reddy2022causally,
  author = {Abbavaram Gowtham Reddy and Benin Godfrey L and Vineeth N. Balasubramanian},
  title = {{On Causally Disentangled Representations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=wgE1g4qsUw2},
}
Figure