Abstract
Disentangling high-level factors of variation in latent representations remains a fundamental challenge in generative modeling. This paper presents CT-VAE, a framework that integrates causality theory with vector-quantized variational autoencoders (VQ-VAEs). By modeling quantized latent vectors as nodes in a causal graph, CT-VAE enables precise "atomic interventions"—such as modifying a single attribute like pose or color—while preserving image fidelity and demonstrating robustness against imbalanced data distributions.
Introduction
Disentanglement aims to map each dimension of a latent space to a single physical factor of variation, such as object color or rotation. While standard Variational Autoencoders (VAEs) have made significant strides, they often struggle with image sharpness and are susceptible to spurious correlations in imbalanced datasets. For instance, if a dataset contains a disproportionate number of red cars, a standard model might entangle "color" with "shape."
The authors propose a shift in perspective: disentanglement is framed as identifying the causal interventions (actions) that transform one image into another by altering exactly one factor. This approach leverages the Independent Causal Mechanisms (ICM) principle, assuming that the mechanism for changing one attribute is independent of others.
Methodology
The CT-VAE architecture consists of two primary components: a Multi-Codebook Quantized VAE (MCQ-VAE) and a Causal Transition (CT) Layer.
Multi-Codebook Quantized VAE (MCQ-VAE)
The MCQ-VAE evolves the standard VQ-VAE by partitioning the latent space into multiple discrete "vocabularies" or codebooks. This modularity is crucial for isolating specific features and provides the discrete nodes necessary for causal modeling.
Causal Transition (CT) Layer
The CT layer, situated between the encoder and decoder, models the transition from an input image $X$ to an output image $Y$. It utilizes a Graph Neural Network (GNN) where latent vectors serve as nodes. The model learns a causal graph (adjacency matrix) that dictates which latent variables are influenced by a specific action.
The framework operates in three distinct modes:
- Standard Mode: Performs basic reconstruction of the input.
- Action Mode: Generates a target image $Y$ given an initial image $X$ and a specified action $a$.
- Causal Mode: Infers the action $a$ that caused the observed transition between $X$ and $Y$.
Experiments and Results
The model was evaluated on synthetic benchmarks (Cars3D, Shapes3D, Sprites, DSprites) and the real-world CelebA dataset.
Action and Factor Recovery
On the Shapes3D dataset, CT-VAE achieved 83% accuracy in identifying specific actions and 90% accuracy in recovering the underlying factors of variation. Qualitative analysis demonstrated that the model could manipulate complex attributes, such as "Orientation," without "leakage" into unrelated attributes like background color.
Robustness to Data Imbalance
By learning invariant mechanisms (e.g., the process of changing a car's elevation), the model remains robust even when certain attribute combinations are overrepresented in the training data. This causal grounding prevents the model from being misled by frequent co-occurrences.
Real-World Application
On CelebA, the model successfully performed interventions like adding "Blond Hair." However, it occasionally reflected dataset biases—for example, changing hair color to blond sometimes unintentionally altered facial features, highlighting the challenges posed by deep-seated spurious associations in real-world data.
Conclusion
CT-VAE demonstrates that treating disentanglement as a dynamic causal process provides a robust path toward high-quality, interpretable representations. Future work includes moving toward "unsupervised causal discovery," where the model learns the set of possible actions directly from raw data, and developing standardized metrics specifically for quantized latent spaces.
Summary updated