Semiparametric Latent Variable Models for Guided Representation

Jasper Snoek, Ryan P. Adams, Hugo Larochelle

Abstract

Unsupervised discovery of latent representations is essential for density modeling, visualization, and learning features relevant to discriminative tasks. Autoencoders are effective at learning latent codes that reflect meaningful variations in data, but guiding them toward task-specific representations while maintaining invariance to irrelevant transformations remains a challenge. Traditional approaches introduce parametric components to tie latent representations to label information. In this work, we propose a nonparametric guidance mechanism that ensures the existence of a predictive function for labels without explicitly instantiating it. Our approach, the Nonparametrically Guided Autoencoder (NPGA), demonstrates superior performance on the Oil Flow and small NORB datasets, achieving state-of-the-art results for single-layer, non-convolutional networks by effectively incorporating invariance information.

Introduction

Autoencoders are fundamental tools in unsupervised learning, designed to compress high-dimensional data into a low-dimensional latent code. However, they often struggle with "distraction": the model may prioritize visually dominant but task-irrelevant variations, such as background lighting or camera angles, over the core features needed for classification.

Previous attempts to address this via partial supervision typically add a rigid parametric classifier (e.g., logistic regression) atop the latent code. This forces the latent space to conform to a specific functional form—like linear separability—prematurely. We argue that instead of forcing the model to be a specific classifier, we should ensure that a high-quality classifier could exist within the learned representation.

The NPGA Framework

The Nonparametrically Guided Autoencoder (NPGA) combines the standard reconstruction loss of an autoencoder with a nonparametric guidance mechanism based on Gaussian Process Latent Variable Models (GPLVM). The model minimizes a joint cost function:

$$J = (1-\alpha)L_{auto} + \alpha L_{GP}$$

Where $L_{auto}$ is the reconstruction error and $L_{GP}$ is the marginal likelihood of the labels given the latent codes under a Gaussian Process (GP) prior. By marginalizing out the mapping, the GP approach optimizes the latent space to be consistent with a wide family of potential functions.

Latent space visualization showing distinct class clusters.

This flexibility allows for the use of specialized kernels tailored to label types, such as periodic kernels for rotation or RBF kernels for complex identities. Furthermore, the encoder acts as a "back-constraint," ensuring the mapping from input to latent space remains a smooth, efficient parametric function suitable for fast inference at test time.

Experimental Evaluation

We evaluated the NPGA on two primary benchmarks: the 12-dimensional Oil Flow dataset and the challenging small NORB object recognition dataset.

On the Oil Flow data, the NPGA significantly outperformed standard autoencoders guided by logistic regression. The nonparametric "nudge" provided by the GP prior proved more effective than rigid parametric constraints, particularly in small-data regimes.

Classification error comparison between NPGA and parametric models.

On the small NORB dataset, which includes nuisance variables like lighting and elevation, the NPGA achieved a state-of-the-art accuracy of 94.28%. By providing guidance on auxiliary labels, the model effectively partitioned the latent space, "quarantining" nuisance variables into specific neurons. This allowed us to discard the nuisance-tracking neurons post-training while maintaining 94.02% accuracy using only the identity-relevant portion of the representation.

Conclusion

The NPGA offers a powerful alternative to parametric supervision by focusing on the existence of a predictive mapping rather than its specific instantiation. While the computational cost of Gaussian Processes remains a consideration, the model's ability to learn highly discriminative and invariant features in shallow architectures is a significant advancement. Future work will explore scaling this nonparametric guidance to deep, multi-layer architectures.

Summary updated

Abstract

Unsupervised discovery of latent representations, in addition to being useful for density modeling, visualisation and exploratory data analysis, is also increasingly important for learning features relevant to discriminative tasks. Autoencoders, in particular, have proven to be an effective way to learn latent codes that reflect meaningful variations in data. A continuing challenge, however, is guiding an autoencoder toward representations that are useful for particular tasks. A complementary challenge is to find codes that are invariant to irrelevant transformations of the data. The most common way of introducing such problem-specific guidance in autoencoders has been through the incorporation of a parametric component that ties the latent representation to the label information. In this work, we argue that a preferable approach relies instead on a nonparametric guidance mechanism. Conceptually, it ensures that there exists a function that can predict the label information, without explicitly instantiating that function. The superiority of this guidance mechanism is confirmed on two datasets. In particular, this approach is able to incorporate invariance information (lighting, elevation, etc.) from the small NORB object recognition dataset and yields state-of-the-art performance for a single layer, non-convolutional network.
BibTeX
@inproceedings{snoek2012semiparametric,
  author = {Jasper Snoek and Ryan P. Adams and Hugo Larochelle},
  title = {{Semiparametric Latent Variable Models for Guided Representation}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2012},
  url = {https://openreview.net/forum?id=Hg-I5C0kVW9},
}
Figure