Disentanglement of Latent Representations via Causal Interventions

Gaël Gendron, Michael Witbrock, Gillian Dobbie

TL;DR

This paper introduces a novel method, the Causal Transition Variational Autoencoder (CT-VAE), that combines causal inference with quantized autoencoders to achieve disentangled representations. It allows for precise interventions on individual factors of variation within an image and can even identify the "action" that caused a change between two images, even with imbalanced data.

What Problem Does This Paper Address?

The core challenge this paper tackles is disentanglement of latent representations, a long-standing problem in machine learning. The goal is to learn representations where each dimension corresponds to an independent, meaningful factor of variation in the data (e.g., object shape, color, pose). While fields like disentanglement, causal representation learning, and independent component analysis have explored this, they often operate in isolation. This paper bridges the gap by proposing that disentanglement can be viewed as identifying interventions that change only one factor of variation. This allows for more robust and interpretable control over generated data.

Methods

The authors propose the Causal Transition Variational Autoencoder (CT-VAE). Key components include:

  • Multi-Codebook Quantized Variational Autoencoder (MCQ-VAE): This is a novel autoencoder architecture that uses multiple codebooks to represent quantized latent vectors. This is inspired by VQ-VAEs but aims to increase the expressivity and disentanglement of local image representations.
  • Causal Transition (CT) Layer: This layer takes the quantized latent representations from the MCQ-VAE and models them within a causal graph. It learns to represent the "action" that causes a transition between two latent representations and the causal dependencies between these variables.
  • Causal Interventions: The CT-VAE can perform interventions on the learned causal graph. This means it can simulate changing a specific factor of variation in the latent space and observe the effect on the generated image.
  • Action Retrieval Task: A novel task is introduced where the model learns to identify the specific "action" (i.e., the factor of variation and its change) that caused a transition from an input image to an output image.
  • Graph Neural Networks (GNNs): GNNs are used within the CT layer to model the causal relationships and perform the transition inference, leveraging their ability to handle graph structures and permutation-equivariance.

Datasets

The authors evaluate their method on several standard synthetic and real-world datasets:

  • Synthetic Datasets:
    • Cars3D: Focuses on car models with factors like type, camera elevation, and azimuth.
  • Shapes3D: Features generated scenes with factors like floor, wall, object colors, scale, shape, and camera orientation.
  • Sprites: Images of animated characters with factors like hair, garments, and pose.
  • DSprites: 2D sprites with factors like color, shape, scale, position (x, y), and rotation.
  • Real-World Dataset:
    • CelebA: A dataset of celebrity faces with 40 attributes. The authors note that these attributes don't fully characterize the images, introducing "exogenous factors of variation."

Key Insights and Ideas

  • Intervention as Disentanglement: The core idea that disentanglement can be framed as identifying interventions that affect only one factor of variation is a powerful conceptual shift.
  • Quantized Representations for Causality: Using quantized latent vectors from VQ-VAEs as causal variables provides a discrete and potentially more interpretable foundation for causal modeling.
  • Handling Imbalanced Data: The causal graph approach helps mitigate issues arising from imbalanced factor distributions in the datasets, as the model learns to disentangle factors based on causal relationships rather than just statistical correlations.
  • Atomic Transitions: The ability to perform "atomic transitions" that modify a single factor without affecting others is crucial for controllable generation and understanding.
  • Action Retrieval as a Novel Task: Introducing the task of recovering the "action" that caused a change between images provides a new benchmark for evaluating causal understanding in generative models.

Key Results

  • Effective Disentanglement and Intervention: The CT-VAE successfully disentangles factors of variation and performs precise interventions on high-level semantic attributes without degrading image quality.
  • Action Retrieval Accuracy: The model achieves good accuracy in retrieving the correct action (factor and direction) for transitions, particularly on datasets with clear factor distinctions (e.g., Cars3D, Shapes3D). For example, on Cars3D, action accuracy reached 0.71 and factor accuracy 0.94.
  • Robustness to Imbalanced Data: Experiments show that the causal graph approach helps the model learn effectively even with highly imbalanced factor distributions.
  • MCQ-VAE Benefits: Using multiple codebooks in the MCQ-VAE architecture shows a slight improvement in factor accuracy, especially with 4 codebooks, suggesting benefits for disentanglement.
  • Causal Masking is Crucial: Ablation studies show that causal masking is essential for accurate disentanglement and action retrieval, with performance dropping significantly without it.

Limitations and Future Work

  • Pre-training/Fine-tuning Split: The current architecture involves a pre-training stage for the MCQ-VAE and a fine-tuning stage for the CT layer. The authors suggest that jointly training both parts could improve performance but would require regularizing latent code distributions.
  • Known Actions: The model currently requires the set of possible actions to be known in advance. Future work aims to learn this set of actions.
  • Disentanglement Metrics: The authors acknowledge that standard disentanglement metrics (like DCI or SAP) are not directly applicable to their quantized latent spaces, leaving the development of precise metrics for this setting as future work.
  • Complex Interventions: While the model handles many interventions well, very complex changes (like significant pose or rotation shifts) are harder to achieve perfectly.
  • Spurious Correlations in Real Data: On the CelebA dataset, the model struggled with spurious associations between attributes (e.g., blonde hair and gender), indicating challenges with real-world data where factors are not perfectly independent.
  • GNN Depth: Reducing the depth of the GNN hurts accuracy, suggesting that multi-step aggregation is important for capturing causal relationships.

Summary updated

Abstract

The process of generating data such as images is controlled by independent and unknown factors of variation. The retrieval of these variables has been studied extensively in the disentanglement, causal representation learning, and independent component analysis fields. Recently, approaches merging these domains together have shown great success. Instead of directly representing the factors of variation, the problem of disentanglement can be seen as finding the interventions on one image that yield a change to a single factor. Following this assumption, we introduce a new method for disentanglement inspired by causal dynamics that combines causality theory with vector-quantized variational autoencoders. Our model considers the quantized vectors as causal variables and links them in a causal graph. It performs causal interventions on the graph and generates atomic transitions affecting a unique factor of variation in the image. We also introduce a new task of action retrieval that consists of finding the action responsible for the transition between two images. We test our method on standard synthetic and real-world disentanglement datasets. We show that it can effectively disentangle the factors of variation and perform precise interventions on high-level semantic attributes of an image without affecting its quality, even with imbalanced data distributions.
BibTeX
@inproceedings{gendron2023disentanglement,
  author = {Gaël Gendron and Michael Witbrock and Gillian Dobbie},
  title = {{Disentanglement of Latent Representations via Causal Interventions}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/361},
}
Figure