Unsupervised Feature Learning through Divergent Discriminative Feature Accumulation

Paul A. Szerlip, Gregory Morse, Justin K. Pugh, Kenneth O. Stanley

Abstract

In deep learning, unsupervised representation learning is typically framed as either a clustering problem or a reconstruction task (e.g., autoencoders). This paper introduces Divergent Discriminative Feature Accumulation (DDFA), a paradigm that abandons reconstruction in favor of a divergent search for discriminative features. By leveraging novelty search and Compositional Pattern Producing Networks (CPPNs), DDFA accumulates a library of features that provide unique signatures for data points. Evaluated on MNIST, DDFA achieves a 1.25% test error, outperforming traditional shallow architectures and demonstrating the potential of open-ended, non-convergent learning.

Introduction: Beyond the Reconstruction Principle

Most unsupervised learning algorithms operate on the "reconstruction principle," assuming that the ability to compress and rebuild an input image implies the discovery of meaningful latent features. However, reconstruction is not the only incentive for learning. Traditional methods are also inherently convergent, utilizing gradient descent to minimize an error function toward a single optimal solution.

Divergent Discriminative Feature Accumulation (DDFA) proposes a shift toward a divergent process. Instead of converging on a fixed set of weights, DDFA "fans out," searching for as many unique ways to differentiate data points as possible. This approach does not require a predefined hidden layer size and continues to discover increasingly complex features as the search progresses.

Methodology: Novelty Search and Geometric Bias

The DDFA pipeline consists of two primary phases: an unsupervised feature discovery phase followed by a supervised classification phase.

Unsupervised Feature Accumulation

DDFA utilizes Novelty Search to reward features for being "different" rather than "accurate." A feature's behavior is defined by its "signature"---a vector of its activations across the training dataset. If a candidate feature produces a signature significantly different from those already in the "accumulator," it is preserved.

To ensure these features capture meaningful spatial information rather than pixel-level noise, the authors employ Compositional Pattern Producing Networks (CPPNs). CPPNs provide an indirect encoding that biases the search toward geometric regularities such as edges, symmetries, and gradients.

Visualizations of diverse features learned by DDFA, showing geometric regularities like edges and gradients.

Supervised Classification

Once a library of features is accumulated, they are frozen. A simple linear classifier is then trained on top of these features using labeled data and standard backpropagation. This separation ensures that the feature learning remains entirely unsupervised.

Experimental Results

The researchers evaluated DDFA on the MNIST benchmark (60,000 training images, 10,000 test images). The performance was compared against two controls: "Random CPPNs" (to isolate the effect of Novelty Search) and "Random Weights" (to isolate the effect of the CPPN geometric bias).

Comparison of test error rates on MNIST for DDFA and control models across different feature counts.

As shown in the table above, DDFA with 3,000 features achieved a 1.25% test error. This significantly outperformed the Random CPPN control (1.61%) and the Random Weight control (2.00%). Notably, this performance nearly matches that of a 4-layer Deep Belief Network (1.2% error), despite DDFA using a shallow, two-layer architecture.

Discussion and Key Insights

The success of DDFA highlights several critical concepts in representation learning:

  • Discrimination vs. Classification: Learning to distinguish between inputs (discrimination) is a powerful proxy for learning what those inputs are (classification).
  • Open-Ended Capacity: Unlike autoencoders, DDFA is an "accumulator." Its capacity is limited only by computational time; it can indefinitely add more features to its library.
  • The Role of Geometric Bias: The disparity between the CPPN-based models and the Random Weight control underscores that the structural representation of weights is as vital as the training algorithm itself.

Limitations and Future Directions

While DDFA is competitive in accuracy, it is computationally intensive, requiring significant time to evolve thousands of features. Furthermore, the current implementation uses shallow features. Future work involves layering DDFA to discover "features of features," potentially enabling a fully divergent deep learning system that integrates convolutional architectures.

By moving away from error minimization and toward curiosity-driven divergence, DDFA provides a promising alternative for open-ended unsupervised learning.

Summary updated

BibTeX
@inproceedings{szerlip2015unsupervised,
  author = {Paul A. Szerlip and Gregory Morse and Justin K. Pugh and Kenneth O. Stanley},
  title = {{Unsupervised Feature Learning through Divergent Discriminative Feature Accumulation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2015},
  doi = {10.1609/aaai.v29i1.9601},
}
Figure