Introduction
Traditional unsupervised feature learning typically relies on generative models, such as autoencoders, which aim to reconstruct input data, or clustering algorithms that group similar data points. This paper proposes Divergent Discriminative Feature Accumulation (DDFA), a paradigm shift that moves away from reconstruction. Instead, DDFA leverages evolutionary "novelty search" to accumulate a diverse set of features that discriminate between training examples in unique ways.
The Challenge of Unsupervised Discrimination
In supervised learning, features are optimized to distinguish between predefined labels (e.g., "dog" vs. "cat"). Without labels, identifying which distinctions are meaningful is difficult. DDFA posits that any feature providing a novel distinction between data points is potentially valuable. The goal is to create a divergent search that avoids converging on a single "best" solution, instead building a rich repertoire of varied perspectives on the data.
Methodology: Novelty Search and CPPNs
DDFA integrates two core concepts from the field of neuroevolution to achieve this divergence:
- Novelty Search: Rather than rewarding a network for accuracy, it is rewarded for being different. Each candidate feature is run across the training set to create a "behavioral signature" (a vector of its outputs). If this signature is sufficiently distinct from the signatures of previously discovered features in a permanent archive, it is considered novel and added to the library.
- Compositional Pattern Producing Networks (CPPNs): To ensure features are not merely random noise, the authors use CPPNs (via the HyperNEAT framework). This indirect encoding biases the search toward geometric regularities such as symmetry, contiguity, and regularity—properties commonly found in biological visual systems like edge detectors.
As the evolutionary process progresses, the system naturally moves from simple edge detectors to more complex structures as simpler forms of "novelty" are exhausted. This allows for complexity growth without a fixed hidden layer size.
Experimental Results
The researchers evaluated DDFA on the MNIST handwritten digit dataset, using a standard split of training, validation, and test images. After accumulating a library of novel features, a simple linear classifier was trained on top of these representations using standard backpropagation.
With 3,000 features, DDFA achieved a 1.25% test error on MNIST. This performance significantly outperformed random control groups and approached the accuracy of much deeper, more complex architectures, such as the original 4-layer RBM-pretrained Deep Belief Network (1.2% error). These results demonstrate that the active search for novelty, rather than the underlying architecture alone, is the primary driver of feature quality.
Discussion and Future Work
While effective, DDFA is computationally intensive; the evolutionary process for 3,000 features required approximately 36 hours on a 12-core machine. Furthermore, this study focused on single-layer features to isolate the core principle of divergent accumulation. Future research could explore "layering" DDFA—building new novel features on top of previously discovered ones—or integrating convolutional constraints to scale the approach to more complex visual tasks like CIFAR or ImageNet.
Summary updated