Reference-based Variational Autoencoders

Adrià Ruiz, Oriol Martínez, Xavier Binefa, Jakob Verbeek2019

TL;DR

This paper introduces "reference-based disentangling," a novel approach to learning disentangled representations from visual data with minimal supervision. By leveraging a small set of "reference" images where specific factors are constant, their proposed Reference-based Variational Autoencoder (Rb-VAE) can effectively separate target factors (like facial expression) from common factors (like identity), outperforming existing methods in tasks like conditional image generation and attribute transfer.

What Problem Does This Paper Address?

Learning disentangled representations, where different underlying factors of variation in data (e.g., pose, lighting, identity in faces) are independently encoded in a model's latent space, is a crucial goal in computer vision. This allows for better control over generative models, easier feature extraction, and more interpretable models. However, traditional methods for achieving this often require extensive, explicit labels for all factors of interest during training. This annotation effort can be prohibitively expensive, especially for complex, continuous factors like natural facial expressions. The paper addresses this challenge by proposing a weakly-supervised learning setting that requires only a small set of reference images where the target factors are held constant, significantly reducing the annotation burden.

Methods

The core contribution is the Reference-based Variational Autoencoder (Rb-VAE). This model builds upon the Variational Autoencoder (VAE) framework but introduces a novel way to incorporate weak supervision.

  • Reference-based Disentangling: The key idea is to use a training set of unlabelled images and an auxiliary "reference" set. In the reference set, a specific set of target factors (e.g., facial expression) are constant, while other factors (e.g., identity, pose) may vary. The goal is to learn a representation where these target factors are disentangled from the others.
  • Rb-VAE Architecture: The model uses a VAE structure with an encoder and a decoder. It introduces a conditional distribution for the target factors, p(e|y), where y is a binary variable indicating whether an image comes from the unlabelled (y=0) or reference (y=1) set. For reference images (y=1), the target factors e are modeled as a deterministic delta function (i.e., constant), while for unlabelled images (y=0), e is modeled as a standard Gaussian prior. Common factors z have a shared Gaussian prior for both sets.
  • Symmetric Variational Learning (sRb-VAE): The authors identify a limitation in the standard VAE objective when applied to their setting, which can lead to degenerate solutions where target factors are not fully disentangled. To address this, they propose a "symmetric" objective that includes a reversed KL divergence term. This symmetric objective is then optimized using an adversarial learning procedure, where discriminators are trained to distinguish between real and generated data distributions. This adversarial approach helps to enforce better disentanglement.

Datasets

  • MNIST: A modified version of the MNIST dataset was used for digit style disentangling. The images were upscaled, and synthetic transformations (width, color, size) were applied to create the unlabelled set. A subset of original MNIST training images served as the reference distribution.
  • AffectNet: A subset of the AffectNet dataset was used for facial expression disentangling. A carefully curated reference set of neutral faces was created, and the unlabelled set consisted of images exhibiting various natural expressions.

Key Insights and Ideas

  • Weak Supervision is Powerful: The paper demonstrates that even minimal supervision, in the form of a reference set with constant target factors, can be highly effective for disentangling representations. This significantly reduces the annotation cost compared to fully supervised methods.
  • Standard VAE Objective is Sub-optimal: The authors found that the conventional VAE objective function can lead to degenerate solutions in their reference-based setting, failing to fully disentangle target factors. This highlights the importance of carefully choosing the objective for specific learning tasks.
  • Symmetric KL Divergence and Adversarial Learning for Disentanglement: The proposed symmetric KL divergence objective, optimized via adversarial learning (sRb-VAE), proved crucial for achieving better disentanglement and outperforming the standard Rb-VAE. This suggests that adversarial training can be a powerful tool for enforcing specific structural properties in latent spaces.
  • Disentangled Representations Enable New Tasks: The learned disentangled representations naturally lend themselves to tasks like conditional image synthesis (changing expression while preserving identity) and visual attribute transfer (swapping styles between digits).

Key Results

  • Quantitative Evaluation: In terms of predicting target factors (e.g., facial expressions, digit styles), the proposed sRb-VAE consistently achieved comparable or better results than state-of-the-art unsupervised methods (B-VAE, β-TCVAE, SVAE, DIP-VAE) on both AffectNet and MNIST datasets. For instance, sRb-VAE achieved the best average performance across both datasets.
  • Qualitative Results:
    • Conditional Image Synthesis: sRb-VAE generated more convincing and diverse facial expressions compared to Rb-VAE, demonstrating better disentanglement of expression from identity. Similarly, for MNIST, sRb-VAE showed better control over style variations like scale.
  • Visual Attribute Transfer: Both models could transfer attributes, but sRb-VAE again showed superior performance, particularly in disentangling scale variations in MNIST digits.
  • Latent Variable Analysis: Experiments showed that the latent variable z (common factors) primarily captured information about common variations (like identity), while e (target factors) effectively encoded the desired disentangled attributes.

Limitations and Future Work

  • Degenerate Solutions with Standard VAE Objective: The paper explicitly points out that the standard VAE objective can lead to degenerate solutions, necessitating the use of the symmetric KL divergence and adversarial learning for optimal performance.
  • Hyperparameter Sensitivity: While the paper mentions that dropout probability was set to 0.25 and worked well, it's possible that other hyperparameters might require fine-tuning for optimal performance on different datasets or tasks.
  • Future Work: The authors suggest that their framework can be extended to more complex scenarios and potentially applied to other domains where disentangled representations are beneficial. The success of the symmetric KL divergence and adversarial approach opens avenues for further research in weakly-supervised disentanglement.

Summary updated

BibTeX
@misc{ruiz2019reference,
  author = {Adrià Ruiz and Oriol Martínez and Xavier Binefa and Jakob Verbeek},
  title = {{Reference-based Variational Autoencoders}},
  year = {2019},
}
Figure