Disentangling Factors of Variation with Cycle-Consistent Variational Auto-Encoders

Ananya Harsh Jha, Saket Anand, Maneesh Singh, V. S. R. Veeravasarapu

Abstract

Generative models that learn disentangled representations for different factors of variation in an image are crucial for targeted data augmentation. We introduce a novel architecture that disentangles the latent space into two complementary subspaces using only weak supervision via pairwise similarity labels. By integrating cycle-consistency into a variational auto-encoder (VAE) framework, we avoid the instabilities of adversarial training. Our results across three datasets demonstrate superior disentanglement and robustness compared to state-of-the-art adversarial methods.

1. Introduction

Disentangling factors of variation involves training a model to decompose an image into independent components, such as identity (e.g., digit class) and style (e.g., slant). Achieving this allows for precise data manipulation and augmentation. However, learning these representations is difficult when labels are sparse or when models fall into "degenerate" states.

2. The Degeneracy Trap

A common failure mode in auto-encoders is the "degeneracy trap," where the model ignores provided labels and encodes all information into the unspecified latent subspace.

Figure 3: (e) A baseline model failing to disentangle at low latent dimensions, and (f) a diagram of the degenerate solution where the encoder bypasses the intended latent structure.

To combat this, previous works often relied on Generative Adversarial Networks (GANs). While effective, GANs are notoriously difficult to train, prone to mode collapse, and highly sensitive to hyperparameter choices.

3. Methodology: Cycle-Consistent VAEs

We propose a non-adversarial framework that splits the latent space into a specified subspace ($s$) and an unspecified subspace ($z$). The core innovation is the application of cycle-consistency in two directions:

  • Forward Cycle: Given two images of the same class, the model swaps their $s$ vectors and attempts to reconstruct the original images, ensuring $s$ captures shared identity.
  • Reverse Cycle: To prevent identity leakage into $z$, we sample a random $z$ and combine it with two different identity vectors ($s_1, s_2$) to generate new images. The encoder must then extract the same $z$ from both generated images, forcing the $z$ space to be independent of identity.

4. Experimental Results

The model was evaluated on MNIST, 2D Sprites, and LineMod datasets. A key finding is the model's robustness to the dimensionality of the latent space compared to non-adversarial baselines.

Figure 2: Qualitative comparison on MNIST. Our model (a, b) maintains consistent disentanglement across different Z dimensions (16 and 64), whereas the baseline (c, d) shows significant leakage and instability.

Quantitatively, our model showed significantly lower identity leakage in the $z$-space. On MNIST, identity classification accuracy from the $z$-space was reduced to ~17-26%, compared to ~66-98% in competing models. This indicates that the $z$ space contains far less "leaked" identity information.

5. Conclusion

Cycle-Consistent VAEs provide a stable, non-adversarial alternative for learning disentangled representations. By focusing on encoder-centric cycle-consistency, we achieve robust separation of factors with only weak supervision. Future work will explore improving image sharpness and extending the framework to fully unsupervised settings.

Summary updated

Abstract

Generative models that learn disentangled representations for different factors of variation in an image can be very useful for targeted data augmentation. By sampling from the disentangled latent subspace of interest, we can efficiently generate new data necessary for a particular task. Learning disentangled representations is a challenging problem, especially when certain factors of variation are difficult to label. In this paper, we introduce a novel architecture that disentangles the latent space into two complementary subspaces by using only weak supervision in form of pairwise similarity labels. Inspired by the recent success of cycle-consistent adversarial architectures, we use cycle-consistency in a variational auto-encoder framework. Our non-adversarial approach is in contrast with the recent works that combine adversarial training with auto-encoders to disentangle representations. We show compelling results of disentangled latent subspaces on three datasets and compare with recent works that leverage adversarial training.
BibTeX
@inproceedings{jha2018disentangling,
  author = {Ananya Harsh Jha and Saket Anand and Maneesh Singh and V. S. R. Veeravasarapu},
  title = {{Disentangling Factors of Variation with Cycle-Consistent Variational Auto-Encoders}},
  booktitle = {European Conference on Computer Vision},
  year = {2018},
  url = {https://openreview.net/forum?id=r14kq9buZS},
}
Figure