Abstract
Unsupervised discovery of latent representations is essential for density modeling, visualization, and learning features relevant to discriminative tasks. Autoencoders are effective at learning latent codes that reflect meaningful variations in data, but guiding them toward task-specific representations while maintaining invariance to irrelevant transformations remains a challenge. Traditional approaches introduce parametric components to tie latent representations to label information. In this work, we propose a nonparametric guidance mechanism that ensures the existence of a predictive function for labels without explicitly instantiating it. Our approach, the Nonparametrically Guided Autoencoder (NPGA), demonstrates superior performance on the Oil Flow and small NORB datasets, achieving state-of-the-art results for single-layer, non-convolutional networks by effectively incorporating invariance information.
Introduction
Autoencoders are fundamental tools in unsupervised learning, designed to compress high-dimensional data into a low-dimensional latent code. However, they often struggle with "distraction": the model may prioritize visually dominant but task-irrelevant variations, such as background lighting or camera angles, over the core features needed for classification.
Previous attempts to address this via partial supervision typically add a rigid parametric classifier (e.g., logistic regression) atop the latent code. This forces the latent space to conform to a specific functional form—like linear separability—prematurely. We argue that instead of forcing the model to be a specific classifier, we should ensure that a high-quality classifier could exist within the learned representation.
The NPGA Framework
The Nonparametrically Guided Autoencoder (NPGA) combines the standard reconstruction loss of an autoencoder with a nonparametric guidance mechanism based on Gaussian Process Latent Variable Models (GPLVM). The model minimizes a joint cost function:
$$J = (1-\alpha)L_{auto} + \alpha L_{GP}$$
Where $L_{auto}$ is the reconstruction error and $L_{GP}$ is the marginal likelihood of the labels given the latent codes under a Gaussian Process (GP) prior. By marginalizing out the mapping, the GP approach optimizes the latent space to be consistent with a wide family of potential functions.
This flexibility allows for the use of specialized kernels tailored to label types, such as periodic kernels for rotation or RBF kernels for complex identities. Furthermore, the encoder acts as a "back-constraint," ensuring the mapping from input to latent space remains a smooth, efficient parametric function suitable for fast inference at test time.
Experimental Evaluation
We evaluated the NPGA on two primary benchmarks: the 12-dimensional Oil Flow dataset and the challenging small NORB object recognition dataset.
On the Oil Flow data, the NPGA significantly outperformed standard autoencoders guided by logistic regression. The nonparametric "nudge" provided by the GP prior proved more effective than rigid parametric constraints, particularly in small-data regimes.
On the small NORB dataset, which includes nuisance variables like lighting and elevation, the NPGA achieved a state-of-the-art accuracy of 94.28%. By providing guidance on auxiliary labels, the model effectively partitioned the latent space, "quarantining" nuisance variables into specific neurons. This allowed us to discard the nuisance-tracking neurons post-training while maintaining 94.02% accuracy using only the identity-relevant portion of the representation.
Conclusion
The NPGA offers a powerful alternative to parametric supervision by focusing on the existence of a predictive mapping rather than its specific instantiation. While the computational cost of Gaussian Processes remains a consideration, the model's ability to learn highly discriminative and invariant features in shallow architectures is a significant advancement. Future work will explore scaling this nonparametric guidance to deep, multi-layer architectures.
Summary updated