Abstract
Unsupervised discovery of latent representations, in addition to being useful for density modeling, visualisation and exploratory data analysis, is also increasingly important for learning features relevant to discriminative tasks. Autoencoders, in particular, have proven to be an effective way to learn latent codes that reflect meaningful variations in data. A continuing challenge, however, is guiding an autoencoder toward representations that are useful for particular tasks. A complementary challenge is to find codes that are invariant to irrelevant transformations of the data. The most common way of introducing such problem-specific guidance in autoencoders has been through the incorporation of a parametric component that ties the latent representation to the label information. In this work, we argue that a preferable approach relies instead on a nonparametric guidance mechanism. Conceptually, it ensures that there exists a function that can predict the label information, without explicitly instantiating that function. The superiority of this guidance mechanism is confirmed on two datasets. In particular, this approach is able to incorporate invariance information (lighting, elevation, etc.) from the small NORB object recognition dataset and yields state-of-the-art performance for a single layer, non-convolutional network.
Introduction
Autoencoders are powerful tools for unsupervised learning, capable of finding compressed latent representations that reconstruct original inputs. However, a persistent challenge is the "distraction" problem: autoencoders often prioritize visually dominant variations (like background lighting or sky color) over task-relevant features (like object identity).
Traditional solutions involve adding a parametric classifier, such as a logistic regression layer, to the latent space. This forces the model to learn features predictive of labels. However, this approach requires the model to simultaneously optimize both the feature representation and the specific weights of the classifier. If the classifier is poorly specified or optimization stalls, the learned features suffer.
The Nonparametrically Guided Autoencoder (NPGA)
To address these limitations, the authors introduce the Nonparametrically Guided Autoencoder (NPGA). This model swaps fixed parametric classifiers for Gaussian Processes (GPs) to provide supervised guidance to the latent space.
The NPGA framework relies on three core components:
- The GP Bridge: A Gaussian Process relates the latent representation to the labels.
- Marginalization: Because GPs are nonparametric, they "marginalize out" the specific mapping function. The model ensures that some smooth function can predict the labels from the latent space, without committing to a specific set of weights.
- Hybrid Objective: The training loss balances reconstruction accuracy with the GP marginal likelihood, which measures how well the latent space supports the label information.
Key Insights: Existence over Instantiation
The central innovation of NPGA is the shift from "instantiating" a classifier to ensuring its "existence." By using a GP, the model exerts a "flexible pressure" on the latent space. This allows for better disentanglement of data factors. For instance, on the NORB dataset, separate GPs can be assigned to object class, lighting, and elevation, forcing the autoencoder to partition the latent space into task-relevant and nuisance-invariant zones.
Experimental Results
The NPGA was evaluated on two benchmarks: the Oil Flow dataset and the Small NORB dataset.
- Oil Flow: This served as a validation of the core concept. NPGA consistently outperformed standard parametric guidance (autoencoder + logistic regression), demonstrating that nonparametric guidance is more effective at shaping the latent space for classification.
- Small NORB: NPGA achieved a state-of-the-art 94.28% accuracy. At the time of publication, this was the highest reported result for a shallow, non-convolutional network, surpassing even complex models like Deep Boltzmann Machines.
The results indicate that by guiding the latent space with multiple signals (class, lighting, rotation), the model can effectively discard irrelevant information while maintaining high accuracy.
Limitations and Future Work
Despite its success, NPGA faces challenges in scalability. Gaussian Processes typically scale at $O(N^3)$ due to matrix inversions. While mini-batching was used here, scaling to massive datasets like ImageNet would require sparse GP approximations. Additionally, the model is sensitive to the blending parameter ($\alpha$), which balances reconstruction and guidance. Future research could explore applying this nonparametric guidance to deeper, multi-layer architectures.
Summary updated