Surrogate-Assisted Evolution for Efficient Multi-branch Connection Design in Deep Neural Networks

Fergal Stapleton, Daniel García-Núñez, Yanan Sun, Edgar Galván

This paper, recently published by researchers at Maynooth University and Sichuan University, tackles one of the most significant bottlenecks in modern AI: the massive computational cost of finding the "perfect" neural network architecture.

TL;DR

The researchers introduce NeuroLGP-MB, a system that uses Linear Genetic Programming and "semantic" surrogate models to automatically evolve complex, multi-branch neural network architectures. By predicting a network's performance based on its output behavior rather than its raw structure, they reduced search time by up to 16.6% while achieving higher accuracy than traditional, more expensive search methods.

What Problem Does This Paper Address?

In the field of Neuroevolution, we use evolutionary algorithms to "breed" the best neural network architectures. While effective, this is notoriously slow. To know if a candidate network is good, you usually have to train it, which can take hours or days. Multiply that by hundreds of candidates over dozens of generations, and the GPU costs become astronomical.

A common shortcut is using a surrogate model—a "mini-model" that predicts how well a new network will perform without actually training it. However, comparing two networks to see how "similar" they are is hard. Most methods look at the genotype (the "code" or graph structure of the network). This is problematic because two networks might look very different structurally but behave identically, or vice versa. Furthermore, calculating distances between complex, multi-branch graphs is computationally heavy and doesn't scale well.

Methods

The authors propose a two-pronged solution:

  1. NeuroLGP-MB (Linear Genetic Programming - Multi-Branch): They use a specific type of genetic programming that represents a network as a sequence of instructions. This allows the evolution of "multi-branch" connections (where data splits and merges, like in ResNets or Inception modules) much more easily than traditional fixed-length encodings.
  2. Semantic-Based Surrogates: Instead of comparing the structure of two networks, they compare their semantics—the actual output values (softmax vectors) the networks produce when run on a set of images.
  3. They use Kriging Partial Least Squares (KPLS) as the surrogate model. Because the semantic output is always a fixed-length vector (regardless of how deep or complex the network is), the surrogate model can easily calculate distances between any two architectures.
  4. Surrogate-PS (Pre-Selection): They introduced an improved management strategy where the surrogate model is initialized with "high-quality" individuals found during a pre-selection phase, rather than just random samples. This "warms up" the search toward more promising areas of the architecture space.

Datasets

The team evaluated their approach on three medical imaging benchmarks:

  • BreakHis x40 & x200: Histopathological images of breast cancer at two different magnifications.
  • Chest X-Ray: A binary classification dataset for detecting pneumonia.

These datasets were chosen because they represent real-world, complex tasks where high-performing, specialized architectures are highly valuable.

Key Insights and Ideas

  • Behavior Over Structure: The "aha!" moment here is that a network's output is a more reliable metric for similarity than its internal wiring. This allows the surrogate model to handle wildly different topologies (different depths, different branching) using a single, consistent distance metric.
  • Scaling the Sample Size: Traditional neuroevolution surrogates often struggle when moving from dozens of sample points to thousands. By using semantic vectors, the authors successfully scaled the surrogate approach to handle the demands of modern Deep Neural Networks (DNNs).
  • The "Depth" Trend: Interestingly, the analysis showed that the Surrogate-PS method tended to favor deeper networks as generations progressed, whereas the "Expensive" (full training) method stayed relatively shallow. This suggests the surrogate helps the evolution process "dare" to explore more complex, deeper spaces that might otherwise be too costly to evaluate.

Key Results

  • Superior Accuracy: On the BreakHis x40 dataset, the Surrogate-PS model achieved 91.9% accuracy, outperforming the "Expensive" full-evolution approach (90.7%) and the random baseline (88.7%).
  • Efficiency Gains: The surrogate-assisted methods were significantly faster. Compared to the "Expensive" model (which trains every network fully), the surrogate approaches reduced the total search time by 11.8% to 16.6%.
  • Robustness: The surrogate models showed a high correlation (Kendall’s Tau > 0.6) between predicted and actual performance, proving that the semantic distance metric is a reliable proxy for real-world accuracy.

Limitations and Future Work

  • Topological Bias: While the system can evolve multi-branch connections, the authors noted that the final "elite" networks were often relatively narrow (fewer branches than expected). They suggest future work should focus on new mutation and crossover operators that specifically encourage more diverse branching.
  • Dataset Sensitivity: On the Chest X-Ray dataset, all models performed similarly. This suggests that for simpler datasets, the overhead of a complex evolutionary search might not be necessary, and a simpler baseline might suffice.
  • Computational Overhead: While faster than full training, the surrogate model itself requires "semantic" data, which means the networks still need to be run (even if not fully trained) to generate those output vectors. Reducing this "inference overhead" is a potential next step.

Summary updated

Abstract

State-of-the-art Deep Neural Networks (DNNs) often incorporate multi-branch connections, enabling multi-scale feature extraction and enhancing the capture of diverse features. This design improves network capacity and generalisation to unseen data. However, training such DNNs can be computationally expensive. The challenge is further exacerbated by the complexity of identifying optimal network architectures. To address this, we leverage Evolutionary Algorithms (EAs) to automatically discover high-performing architectures, a process commonly known as neuroevolution. We introduce a novel approach based on Linear Genetic Programming (LGP) to encode multi-branch (MB) connections within DNNs, referred to as NeuroLGP-MB. To efficiently design the DNNs, we use surrogate-assisted EAs. While their application in simple artificial neural networks has been influential, we scale their use from dozens or hundreds of sample points to thousands, aligning with the demands of complex DNNs by incorporating a semantic-based approach in our surrogate-assisted EA. Furthermore, we introduce a more advanced surrogate model that outperforms baseline, computationally expensive, and simpler surrogate models.
BibTeX
@inproceedings{stapleton2025surrogate,
  author = {Fergal Stapleton and Daniel García-Núñez and Yanan Sun and Edgar Galván},
  title = {{Surrogate-Assisted Evolution for Efficient Multi-branch Connection Design in Deep Neural Networks}},
  booktitle = {Proceedings of the Genetic and Evolutionary Computation Conference Companion},
  year = {2025},
  doi = {10.1145/3712255.3726649},
}
Figure