Discovering Hidden Factors of Variation in Deep Networks

Brian Cheung, Jesse A. Livezey, Arjun K. Bansal, Bruno A. OlshausenICLR 2014 · OpenReview

TL;DR

This paper introduces a novel regularization technique, the cross-covariance penalty (XCov), to an autoencoder framework. This allows standard deep networks to disentangle and explicitly represent factors of variation in data that are not directly related to the primary task (like classification), leading to richer, more interpretable representations.

What Problem Does This Paper Address?

Deep learning models excel at learning features for specific tasks, such as image classification. However, these learned representations often entangle various sources of variation present in the data (e.g., object identity, pose, lighting, style) with the features relevant for the classification task. This means that while a model can classify an image, it might lose or mix information about other important characteristics. The paper aims to address this by developing a method to disentangle these various factors of variation within a deep network, allowing for a more comprehensive understanding and manipulation of the data.

Methods

The core of the paper's contribution lies in augmenting a standard autoencoder with two key components:

  • Semi-supervised Learning: The autoencoder's latent representation is split into two parts: "observed variables" (ŷ) and "latent variables" (z). The observed variables are trained to predict the class label (like digit identity) using a supervised cost (cross-entropy).
  • Cross-Covariance Penalty (XCov): This is the novel regularization term. It penalizes the cross-covariance between the observed variables (ŷ) and the latent variables (z) across a batch of data. The goal is to ensure that the latent variables (z) capture variations other than those related to the class label, effectively disentangling them.

The overall objective function combines a reconstruction cost (to ensure the autoencoder can reconstruct the input), a supervised cost for the observed variables, and the XCov penalty.

Datasets

The authors evaluate their method on three datasets:

  • MNIST Handwritten Digits Database: A standard dataset of handwritten digits (0-9).
  • Toronto Faces Database (TFD): A dataset of grayscale face images with labels for different expressions.
  • Multi-PIE Dataset: A more complex dataset of high-resolution color face images with labels for camera pose and illumination conditions.

Key Insights and Ideas

  • Beyond Classification: Standard deep networks can learn factors of variation beyond what's needed for simple classification, and these factors can be explicitly represented.
  • Disentanglement via XCov: The XCov penalty is an effective way to disentangle class-relevant information from other sources of variation in the latent space of an autoencoder.
  • Latent Space Interpolation: The learned latent space (z) is smooth and continuous, allowing for meaningful interpolation and generation of novel variations of the input data.
  • Extrapolation of Supervised Signals: The model can extrapolate "hidden" variation even from discrete class labels, generating meaningful variations beyond the training distribution. For example, it can generate "negative" facial expressions not explicitly present in the dataset.
  • Hierarchical Representation: As information flows through the decoder, the learned representations become more spatially localized and less semantically meaningful, suggesting a hierarchical decomposition of features.

Key Results

  • Classification Performance: The addition of the XCov regularization negligibly impacted classification performance on MNIST and TFD, showing it doesn't hinder task-specific learning.
  • Disentangled Representations:
    • On MNIST, the latent variables learned to represent digit style (e.g., slant, size).
  • On TFD, the latent variables captured subject identity, allowing for expression manipulation while preserving identity.
  • On Multi-PIE, the latent variables captured identity and other attributes like clothing and skin tone, while observed variables represented camera pose and illumination.
  • Generative Capabilities: The model could generate manipulated images by changing specific factors of variation, such as altering digit style or facial expression while keeping other attributes constant.
  • Extrapolation: The model demonstrated the ability to generate exaggerated or "opposite" facial expressions by extrapolating beyond the range of the supervised labels.

Limitations and Future Work

  • Dimensionality of z: While the paper notes that a low-dimensional z can sometimes learn disentangled representations without XCov, convergence becomes more difficult, and XCov is beneficial for higher-dimensional z and better reconstruction.
  • Hyperparameter Tuning: The choice of $\beta$ (supervised cost multiplier) and $\gamma$ (XCov penalty multiplier) will depend on the specific task, with larger $\beta$ favoring classification and larger $\gamma$ favoring disentanglement.
  • Future Work: The paper suggests that this approach has potential applications in exploratory data analysis and signal denoising, implying further exploration in these areas. The ability to generate novel manipulations also points towards applications in creative content generation.

Summary updated

Abstract

Deep learning has enjoyed a great deal of success because of its ability to learn useful features for tasks such as classification. But there has been less exploration in learning the factors of variation apart from the classification signal. By augmenting autoencoders with simple regularization terms during training, we demonstrate that standard deep architectures can discover and explicitly represent factors of variation beyond those relevant for categorization. We introduce a cross-covariance penalty (XCov) as a method to disentangle factors like handwriting style for digits and subject identity in faces. We demonstrate this on the MNIST handwritten digit database, the Toronto Faces Database (TFD) and the Multi-PIE dataset by generating manipulated instances of the data. Furthermore, we demonstrate these deep networks can extrapolate `hidden' variation in the supervised signal.
BibTeX
@inproceedings{cheung2014discovering,
  author = {Brian Cheung and Jesse A. Livezey and Arjun K. Bansal and Bruno A. Olshausen},
  title = {{Discovering Hidden Factors of Variation in Deep Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2014},
  url = {http://arxiv.org/abs/1412.6583},
}
Figure