Explicit Disentanglement of Appearance and Perspective in Generative Models

Nicki Skafte Detlefsen, Søren Hauberg

Introduction

Disentangled Representation Learning (DRL) aims to develop models that represent independent factors of variation in data through separate, interpretable latent variables. For instance, in a generative model of digits, one might want separate "knobs" for the digit's identity, its rotation, and its thickness. While unsupervised methods like the $\beta$-VAE have made significant strides, theoretical work suggests that true disentanglement is often impossible without inductive biases—prior knowledge about the structure of the physical world.

In this work, Detlefsen and Hauberg (NeurIPS 2019) introduce the Variational Inferred Transformational Autoencoder (VITAE). VITAE explicitly separates an object's intrinsic appearance (the "what") from its spatial perspective (the "where" and "how") by hard-wiring geometric transformations into the model architecture.

The VITAE Architecture

The core innovation of VITAE is the decomposition of the latent space into two distinct components:

  1. Perspective ($z_P$): This space encodes spatial transformations such as rotation, translation, and scaling.
  2. Appearance ($z_A$): This space encodes the "archetype" or canonical form of the object, effectively the image after all spatial transformations have been removed.

To achieve robust separation, the authors proposed Conditional-VITAE (C-VITAE). In this variant, the appearance encoder is conditioned on the predicted perspective. By "undoing" the transformation before encoding the appearance, the model ensures that $z_A$ only needs to represent a normalized version of the data, preventing the appearance space from being polluted by pose information.

Comparison of VAE, Unconditional VITAE, and Conditional VITAE architectures. C-VITAE (c) uses the inferred transformation $\gamma$ to normalize the input $x$ via an inverse transform $(\mathcal{T}_\gamma)^{-1}$ before appearance encoding.

Diffeomorphic Transformations

A critical requirement for VITAE is that the spatial transformations must be smooth and invertible (diffeomorphic). This ensures that the model can map pixels back and forth between the observed image and the canonical archetype without losing information. The authors employ:

  • Affine transformations via matrix exponentials for simpler datasets like MNIST.
  • CPAB (Continuously Piecewise-Affine Based) transformations for complex, non-rigid deformations, such as those found in human body poses and facial expressions.

Experimental Results

The model was evaluated on several benchmarks, including dSprites, Augmented MNIST, and the SMPL human body model.

MNIST Disentanglement

On the Augmented MNIST dataset, C-VITAE successfully separates digit identity from orientation and position. As shown in the latent space visualizations, the perspective space captures the variations in pose (slant and rotation), while the appearance space clusters digits by their canonical forms.

Latent space visualization for MNIST. The perspective space captures transformations (slants/rotations), while the appearance space maps different instances of a digit to a consistent canonical 'archetype' (upright 0s and 1s).

Human Body Modeling (SMPL)

The most striking results were observed on the SMPL dataset, which consists of synthetic 3D human bodies. C-VITAE achieved a disentanglement score ($D_{score}$) of 0.943, significantly outperforming the $\beta$-VAE baseline (0.525). The model demonstrated the ability to independently manipulate body shape (e.g., body fat percentage) and body pose (e.g., arm positions), a task where standard VAEs often fail due to the inherent entanglement of these factors in pixel space.

Conclusion

VITAE represents a shift from purely unsupervised disentanglement towards models with "hard-wired" geometric priors. By incorporating the concept of spatial transformations directly into the architecture, the model achieves superior performance on complex data. Future work may extend this hierarchical approach to disentangle other factors such as color, texture, and lighting, peeling off one layer of variation at a time to reach a fully interpretable representation of the world.

Summary updated

Abstract

Disentangled representation learning finds compact, independent and easy-to-interpret factors of the data. Learning such has been shown to require an inductive bias, which we explicitly encode in a generative model of images. Specifically, we propose a model with two latent spaces: one that represents spatial transformations of the input data, and another that represents the transformed data. We find that the latter naturally captures the intrinsic appearance of the data. To realize the generative model, we propose a Variationally Inferred Transformational Autoencoder (VITAE) that incorporates a spatial ransformer into a variational autoencoder. We show how to perform inference in the model efficiently by carefully designing the encoders and restricting the transformation class to be diffeomorphic. Empirically, our model separates the visual style from digit type on MNIST, separates shape and pose in images of human bodies and facial features from facial shape on CelebA.
BibTeX
@inproceedings{detlefsen2019explicit,
  author = {Nicki Skafte Detlefsen and Søren Hauberg},
  title = {{Explicit Disentanglement of Appearance and Perspective in Generative Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=WSi15o8NY9T},
}
Figure