Introduction
Deep learning has revolutionized visual processing, but most state-of-the-art models rely heavily on supervised or unsupervised gradient-based training, such as backpropagation. In contrast, biological visual systems—like the human brain—are the product of millions of years of evolution.
Traditional NeuroEvolution (NE), which uses evolutionary algorithms to "breed" neural networks, typically fails when scaled to "deep" architectures. The search space simply becomes too vast, a problem known as the "curse of dimensionality." This paper explores a hybrid approach to deep learning by combining NeuroEvolution with backpropagation. It demonstrates that while evolution alone struggles to master complex image classification, it excels at "growing" sophisticated feature extractors that provide a powerful head start for traditional gradient-based learning.
Overcoming the Curse of Dimensionality with HyperNEAT
To tackle the challenge of evolving deep neural structures capable of handling real-world benchmarks like MNIST, the authors utilize Hypercube-based NeuroEvolution of Augmenting Topologies (HyperNEAT).
Unlike standard NE, which evolves every connection weight individually, HyperNEAT uses an indirect encoding called a Compositional Pattern Producing Network (CPPN). Think of the CPPN as a "recipe" or a geometric function. It takes the spatial coordinates of two neurons as input and outputs the weight of the connection between them.
This indirect encoding allows the system to represent massive, complex networks using a very small number of parameters. By doing so, it effectively exploits the geometric regularities—like symmetry and repetition—found in images.
A Hybrid Pipeline: Evolution as a Feature Learner
The paper introduces two key modifications to the standard HyperNEAT pipeline to make it suitable for deep learning tasks:
- Evolution as a Feature Learner: Instead of trying to evolve a complete, end-to-end classifier, HyperNEAT is used to evolve a "feature extractor." The output of this evolved network is then fed into a final layer that is fine-tuned using standard backpropagation.
- CNN Substrates: While HyperNEAT usually works with fully connected layers, the authors extended it to generate weights for Convolutional Neural Network (CNN) architectures, which are inherently better suited for vision tasks.
To test this architecture, the study uses the MNIST dataset of 28x28 pixel grayscale handwritten digits. To keep the evolutionary process computationally feasible, the authors used a subset for fitness evaluation (300 images for training, 1,000 for fitness testing) before evaluating the final champion on the full 10,000-image test set.
Key Insights and Results
The research highlights a powerful division of labor: evolution is fantastic at discovering global organizational principles and robust features (the "what"), while backpropagation is superior at the "fine-tuning" required for precise classification (the "which one").
Because the model uses CPPNs, it "understands" that pixels near each other are related. This spatial awareness allows the evolved feature maps to show distinct, recognizable patterns. Furthermore, simply switching the underlying "substrate" from a traditional feed-forward ANN to a CNN-style structure led to a massive performance boost, even when the evolutionary algorithm remained the same. This pushes NeuroEvolution beyond the typical one-or-two hidden layer limit, demonstrating that indirect encoding can successfully navigate seven-layer architectures.
The results highlight a clear hierarchy of performance:
- HyperNEAT Alone: Struggled significantly, achieving only 23.9% accuracy on MNIST. It could find the "shape" of the problem but couldn't master the nuances of classification.
- HyperNEAT + Backprop (Traditional ANN): Performance jumped to 58.4%. Evolution provided a decent feature set that backprop could actually use.
- HyperNEAT + Backprop (CNN Architecture): This was the clear winner, reaching 92.1% accuracy.
The takeaway is that HyperNEAT is a highly effective "pre-trainer." It discovers features that are more generalizable than those found by evolution alone.
Limitations and Future Work
The primary limitation of this approach is that HyperNEAT still struggles with the "fine-grained" control of individual weights. Because a single change in the CPPN (the genotype) can change weights across the entire network (the phenotype), it is hard for evolution to make the tiny, surgical adjustments needed to reach 99%+ accuracy on its own.
The authors suggest future research should look into ES-HyperNEAT (Evolvable Substrate). In the current paper, the researchers had to manually define where the neurons were located. ES-HyperNEAT would allow the evolutionary process to decide not just the weights of the connections, but the placement and density of the neurons themselves—moving one step closer to how biological brains actually develop.
Summary updated