Relevance Factor VAE: Learning and Identifying Disentangled Factors

Minyoung Kim, Yuting Wang, Pritish Sahu, Vladimir PavlovicOpenReview 2019 · OpenReview

TL;DR

This paper introduces Relevance Factor VAE (RF-VAE), a novel unsupervised deep learning model that learns disentangled representations of data. Unlike previous methods, RF-VAE can automatically identify and distinguish between truly meaningful factors of variation and less important "nuisance" factors, leading to more robust and interpretable representations.

What Problem Does This Paper Address?

A core goal in machine learning is to learn representations of data that capture the underlying factors of variation. For instance, in images of faces, we might want to disentangle factors like identity, pose, lighting, and expression. Ideally, each latent variable in our model should control a single, independent aspect of the data. While Variational Autoencoders (VAEs) and their extensions have made progress, a key challenge remains: existing unsupervised methods often struggle to differentiate between genuinely important factors and less relevant ones. This can lead to "nuisance" factors degrading the overall disentanglement performance, making the learned representations less interpretable and useful. This paper aims to solve this by developing a method that can automatically identify and focus on the truly relevant factors of variation.

Methods

The proposed model, Relevance Factor VAE (RF-VAE), builds upon the Variational Autoencoder (VAE) framework and incorporates the concept of Total Correlation (TC) minimization, similar to Factor-VAE. The key innovation is the introduction of relevance indicator variables. These are learned alongside the VAE parameters and act as a binary switch (or a continuous probability) for each latent dimension, indicating whether it's considered "relevant" or "nuisance."

The RF-VAE's loss function is designed to:

  1. Reconstruct the input data: Standard VAE objective.
  2. Minimize Total Correlation (TC): This encourages independence among latent variables. Crucially, the TC penalty is applied only to the identified relevant latent dimensions.
  3. Tolerate large prior KL divergences for relevant factors: This is a theoretical justification for allowing relevant factors to deviate more from a simple prior, as they are expected to capture complex variations.
  4. Penalize nuisance factors more heavily: The model is designed to penalize deviations of nuisance factors from a simple prior more strongly.
  5. Regularize the relevance indicators: An L1 norm and an entropy term are used to encourage a sparse and binary selection of relevant factors.

The authors propose two variants: RF-VAE-0, where the set of relevant factors is known beforehand (for evaluation), and RF-VAE, which learns this set automatically from the data.

Datasets

The paper evaluates its approach on several benchmark datasets:

  • 3D-Faces: Synthetic face images with known factors like subject ID, azimuth, elevation, and lighting.
  • Sprites: Binary images of shapes with geometric variations like scale, rotation, and translation.
  • Sprites Oval-Only: A subset of the Sprites dataset to isolate specific factors.
  • Teapots: Images of teapots with variations in azimuth, elevation, and color.
  • Celeb-A: A real-world dataset of celebrity faces (results provided in the supplement).

Key Insights and Ideas

  • The "nuisance factor" problem is critical: The paper highlights that simply applying a disentanglement loss (like TC minimization) to all latent dimensions indiscriminately can lead to poor performance because nuisance factors can "hijack" the disentanglement process.
  • Relevance indicators are key: Automatically learning which latent dimensions are truly important is the core mechanism for overcoming the nuisance factor problem. This allows the model to focus its efforts on disentangling the meaningful variations.
  • Theoretical justification for tolerating KL divergence: The paper provides a theoretical analysis showing why allowing relevant factors to deviate from a simple prior is not only acceptable but necessary for achieving good disentanglement.
  • New disentanglement metric (Metric II): The authors introduce a new metric that measures how well a single factor variation in the latent space corresponds to a single factor variation in the observed data. They argue this metric aligns better with qualitative assessments than previous ones.

Key Results

  • Outperforms existing methods: RF-VAE consistently outperforms Vanilla VAE, β-VAE, and Factor-VAE across various disentanglement metrics (Metric I, II, and III) on benchmark datasets like 3D-Faces and Sprites.
  • RF-VAE (learning R) is competitive with RF-VAE-0 (known R): The model that automatically learns the relevant factors (RF-VAE) performs nearly as well as the version that is given the ground-truth relevant factors (RF-VAE-0), demonstrating the effectiveness of the learned relevance identification.
  • Qualitative evidence supports findings: Latent space traversals in RF-VAE generate images that clearly show variations corresponding to the identified relevant factors, while nuisance factors have minimal impact. This visual evidence strongly supports the model's ability to disentangle.
  • Identifies correct number of factors: On 3D-Faces, RF-VAE correctly identifies 4 relevant factors, matching the ground truth, whereas competing models often identify more, suggesting they learn redundant representations.

Limitations and Future Work

  • Discrete factors with low cardinality: The paper notes that discrete factors with very few possible values (like shape in the Sprites dataset) can still pose a challenge, sometimes becoming entangled with continuous factors. The authors suggest that hybrid models capable of handling both discrete and continuous latent variables might be a solution.
  • Model architecture flexibility: For the Teapots dataset, RF-VAE-0 performed worse than expected, possibly due to the model architecture not being flexible enough to capture all variations. The authors suggest that more complex architectures might be needed for certain datasets.
  • Future work: The authors suggest extending the model to more general, hybrid factor frameworks to better handle discrete factors. They also plan to conduct more rigorous failure analysis for challenging datasets.

Summary updated

Abstract

We propose a novel VAE-based deep auto-encoder model that can learn disentangled latent representations in a fully unsupervised manner, endowed with the ability to identify all meaningful sources of variation and their cardinality. Our model, dubbed Relevance-Factor-VAE, leverages the total correlation (TC) in the latent space to achieve the disentanglement goal, but also addresses the key issue of existing approaches which cannot distinguish between meaningful and nuisance factors of latent variation, often the source of considerable degradation in disentanglement performance. We tackle this issue by introducing the so-called relevance indicator variables that can be automatically learned from data, together with the VAE parameters. Our model effectively focuses the TC loss onto the relevant factors only by tolerating large prior KL divergences, a desideratum justified by our semi-parametric theoretical analysis. Using a suite of disentanglement metrics, including a newly proposed one, as well as qualitative evidence, we demonstrate that our model outperforms existing methods across several challenging benchmark datasets.
BibTeX
@misc{kim2019relevance,
  author = {Minyoung Kim and Yuting Wang and Pritish Sahu and Vladimir Pavlovic},
  title = {{Relevance Factor VAE: Learning and Identifying Disentangled Factors}},
  year = {2019},
  eprint = {1902.01568},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1902.01568},
  howpublished = {OpenReview},
}

Loading related content...

Figure