Abstract
Neuroevolution and Neural Architecture Search (NAS) typically require training candidate networks from scratch, which is computationally prohibitive. A major hurdle in applying evolutionary crossover to deep neural networks is the "competing conventions" problem, where internal representational differences cause performance collapse when layers are swapped. This paper introduces a novel pipeline to recombine pre-trained networks by inserting trainable "stitching layers" that translate feature representations between different architectures. Our method enables a "warm start" for neuroevolution, allowing for the rapid discovery of efficient hybrid architectures that optimize accuracy and computational cost.
Introduction
The search for optimal neural architectures often involves evaluating thousands of candidates. While evolutionary algorithms utilize crossover to combine successful traits, simply swapping layers between two pre-trained parent networks usually "breaks" the resulting model. Even if two networks perform the same task, their internal layers may order features differently or use different numerical scales. Providing these "wrong" features to a subsequent layer results in a total breakdown of performance. To address this, we propose a method to align these internal "languages" using thin mathematical bridges called stitching layers.
Methodology: The Stitching Pipeline
The proposed framework constructs a "stitched" supernetwork through a four-step process designed to preserve the knowledge of pre-trained parents while allowing for architectural recombination.
- Finding Matches: Potential connection points are identified between Parent A and Parent B based on compatible tensor shapes.
- Acyclic Matching: A branch-and-bound algorithm selects a set of matches that ensures the resulting graph is a directed acyclic graph (DAG), permitting a valid forward pass.
- Constructing the Supernetwork: "Switch" layers are inserted at matching points. These switches can route data from the original parent path or cross over to the other parent via a stitching layer. These layers are typically simple linear transformations or 1x1 convolutions.
- Efficient Training: The original weights of the parent networks are frozen. Only the stitching layers are trained using a Mean-Squared Error (MSE) loss to align the output of the stitch with the expected input of the target layer. This process is significantly faster than standard backpropagation.
Following construction, multi-objective evolutionary algorithms (such as Genetic Algorithms and GOMEA) search for the optimal configuration of switches to maximize accuracy while minimizing computational cost (multiply-adds).
Experimental Results
The approach was validated on ImageNet classification and PASCAL VOC semantic segmentation tasks. Key findings include:
- Efficiency: Training the stitching layers and creating the supernetwork took only 15 to 25 minutes on modern hardware, orders of magnitude faster than training from scratch.
- Performance: The evolutionary search successfully identified "offspring" networks that formed a new Pareto front, sometimes dominating the original parent models or their standard ensembles in efficiency.
- Search Optimization: By skipping evaluations for mutations that did not affect the active subnetwork, the search process was further accelerated, with some algorithms skipping up to 45% of evaluations.
Discussion and Limitations
While effective, the method exhibits a "staircase" effect in the Pareto fronts, where large blocks of layers are toggled at once, leading to coarse jumps in computational cost. Additionally, stitched networks often show "underconfidence" in their predictions, likely due to the MSE loss used during stitching. Future work will explore more granular matching strategies and alternative loss functions, such as cross-entropy, to improve calibration and fill gaps in the accuracy-compute trade-off space.
Summary updated