Learning Flexible Latent Representations via Encapsulated Variational Encoders

Wenjun Bai, Changqin Quan, Zhi-Wei LuoAAAI 2019 · DOI

TL;DR

The authors introduce the Encapsulated Variational Encoder (EVE), a novel framework that nests two separate encoders within a single scaffold. By tuning a single hyperparameter ($\alpha$), the model can either force these encoders to agree (converged representations) to boost classification or allow them to differ (diverged representations) to increase generative diversity.

What Problem Does This Paper Address?

Variational Auto-Encoders (VAEs) are powerful tools for unsupervised representation learning, but they often lack flexibility. Standard VAEs map data to a single latent space, which might not be optimal for every downstream task. For instance, a "good" representation for classification (which needs to be robust and regularized) might look very different from a "good" representation for image generation (which needs to capture diverse styles and nuances).

The authors identify a gap: there isn't a simple, unified way to control the "tightness" or "diversity" of these latent representations within the VAE framework. They propose EVE to bridge this gap by allowing two encoders to interact and influence each other's learning process.

Methods

The core innovation is the Encapsulated Variational Encoder (EVE). Instead of one encoder, EVE uses two: a base encoder ($q_{\phi_b}$) and a scaffolding encoder ($q_{\phi_s}$).

To manage the relationship between these two, the authors introduce a discrepancy function weighted by a hyperparameter $\alpha$. This function measures the difference between the latent representations produced by the two encoders using Monte-Carlo sampling.

The framework operates in two primary modes:

  1. Converged Representations ($\alpha \to 0$): The model imposes an "equivalence constraint," forcing the two encoders to produce nearly identical latent codes. This acts as a form of strong regularization.
  2. Diverged Representations ($\alpha \to \infty$): The model imposes an "independence constraint," allowing the encoders to capture different aspects of the data.

The authors also extend this to a generative model called the Encapsulated Variational Auto-Encoder (EVAE), which combines the EVE with a probabilistic decoder that takes inputs from both latent spaces.

Datasets

The researchers evaluated their model on two standard benchmarks:

  • MNIST: Used primarily for qualitative evaluation of generative diversity (binarized version).
  • CIFAR-10 (4k): A semi-supervised setup where only 4,000 samples are labeled, used to test the discriminative (classification) performance of converged representations.

Key Insights and Ideas

  • The "Scaffold" as a Regularizer: In the semi-supervised setting, forcing two encoders to converge ($\alpha \to 0$) acts as a powerful regularizer. It ensures the latent space is stable and informative, which directly translates to better classification accuracy.
  • Diversity through Divergence: For generative tasks, "more is more." By allowing the encoders to diverge ($\alpha \to 50$), the model captures more variance in the data. This results in a decoder that can produce the same digit in a wide variety of writing styles rather than a single "average" version.
  • A Single Dial for Flexibility: The beauty of the EVE approach is its simplicity. Rather than redesigning the architecture for different tasks, a researcher can simply "turn the dial" of $\alpha$ to optimize for either discrimination or generation.

Key Results

  • State-of-the-Art Competitive Performance: On the CIFAR-10 (4k) semi-supervised task, the EVE achieved a test error of 18.2% (at $\alpha=0.001$). This outperformed other established methods like Virtual Adversarial Training (VAT) at 24.1% and S3C at 28.1%, coming close to the performance of Ladder Networks (16.5%).
  • Negative Correlation with $\alpha$ in Classification: The paper clearly demonstrates that as $\alpha$ decreases (forcing convergence), the classification error also decreases. For example, error dropped from 46.4% at $\alpha=1$ to 18.2% at $\alpha=0.001$.
  • Qualitative Style Variation: Visualizations on MNIST showed that high $\alpha$ values allowed the model to generate a diverse array of stroke thicknesses and slants for the same digit, whereas low $\alpha$ values produced very uniform, "standardized" digits.

Limitations and Future Work

While the paper presents a robust proof-of-concept, it is limited to a two-encoder setup. The authors suggest that the next logical step is moving toward a non-parametric form of EVE. In this future version, the number of incorporated variational encoders could theoretically reach infinity, potentially allowing the model to automatically determine the optimal number of latent "sub-spaces" needed for a given dataset.

Additionally, the paper focuses on image data; exploring how EVE handles more complex, non-visual data types remains an open question.

Summary updated

BibTeX
@inproceedings{bai2019learning,
  author = {Wenjun Bai and Changqin Quan and Zhi-Wei Luo},
  title = {{Learning Flexible Latent Representations via Encapsulated Variational Encoders}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33019913},
}
Figure