Abstract
Unsupervised discovery of latent representations is essential for density modeling, visualization, and feature learning. However, standard autoencoders often prioritize dominant variations in raw input that may be irrelevant to specific downstream tasks. We introduce the Nonparametrically Guided Autoencoder (NPGA), a hybrid model that steers feature learning toward task-relevant information. By employing Gaussian Processes (GPs) to marginalize over potential label-predicting functions, NPGA learns robust representations that separate signal from noise without being constrained by a specific parametric classifier during training.
The "Blind Spot" of Standard Autoencoders
Standard autoencoders are designed to find compressed representations by minimizing reconstruction error. A significant drawback is their tendency to focus on the most dominant variations in the input data. In many real-world scenarios, these variations are "nuisance" factors—such as lighting conditions or camera angles—that are irrelevant to the primary task, like object classification.
Previous attempts to guide autoencoders often involved attaching a parametric classifier (e.g., logistic regression) to the hidden layer. This approach introduces a "moving target" problem: the autoencoder must simultaneously optimize the representation and the specific weights of the classifier. If the classifier is poorly initialized, it can actively hinder the development of useful features.
The NPGA Framework
To overcome the limitations of parametric guidance, the NPGA utilizes a Gaussian Process Latent Variable Model (GPLVM). Instead of training a single classifier, the model uses a GP to effectively "average out" all possible functions that could map the latent space to the labels. This ensures that a predictive structure exists in the latent space without forcing the model to commit to a specific set of parameters early in training.
Key Technical Components
- Hybrid Objective: The model minimizes a combined loss function consisting of the standard reconstruction error and the GP marginal likelihood. This balances the need to represent the input data with the requirement that labels remain predictable from the latent space.
- Back-Constraints: By treating the autoencoder's encoder as a "back-constraint" for the GPLVM, the model can process new data points instantly via a standard neural network pass, avoiding the typical computational bottlenecks associated with GPs.
- Kernel Flexibility: The GP framework allows for specialized kernels. For instance, a periodic kernel can be used for angular data (like azimuth), which is naturally difficult for standard linear classifiers to handle.
Experimental Results
Performance Comparison
The authors evaluated NPGA on the Oil Flow dataset, comparing it against fully parametric models. As shown in the classification error analysis, the nonparametric approach consistently achieves lower error rates across various weighting schemes (Alpha) for the guidance term. By marginalizing over functions, the NPGA avoids the optimization pitfalls that plague parametric guidance.
Disentanglement and Information Separation
On the challenging Small NORB dataset—which contains significant nuisance variations in lighting and elevation—the NPGA achieved a state-of-the-art accuracy of 94.28% for a shallow, non-convolutional network. The model successfully "organized" its latent space, dedicating specific neurons to specific types of information (class, elevation, lighting).
This separation allowed the authors to discard the portions of the representation guided by lighting and elevation while maintaining nearly 94% accuracy using only the class-guided features. This demonstrates the model's ability to effectively ignore irrelevant transformations while preserving task-critical signals.
Conclusion
The Nonparametrically Guided Autoencoder provides a flexible and robust framework for task-specific feature learning. By leveraging the mathematical elegance of Gaussian Processes, it avoids the "moving target" problem of parametric guidance and enables the discovery of highly structured, disentangled latent spaces. Future work will explore extending this nonparametric guidance to deep, multi-layer architectures to handle even more complex data structures.
Summary updated