Abstract
Evolutionary deep intelligence has recently shown great promise for producing small, powerful deep neural network models via the synthesis of increasingly efficient architectures over successive generations. Despite recent research showing the efficacy of multi-parent evolutionary synthesis, little has been done to directly assess architectural similarity between networks during the synthesis process for improved parent network selection. In this work, we present a preliminary study into quantifying architectural similarity via the percentage overlap of architectural clusters. Results show that networks synthesized using architectural alignment (via gene tagging) maintain higher architectural similarities within each generation, potentially restricting the search space of highly efficient network architectures.
Introduction
Deep Neural Networks (DNNs) are powerful tools but often suffer from high computational costs, making them difficult to deploy on edge devices. Evolutionary deep intelligence addresses this by synthesizing smaller, more efficient networks over generations. A significant challenge in "sexual" evolutionary synthesis—where child networks are created from multiple parents—is architectural mismatch. Combining networks with divergent internal structures can lead to dysfunctional offspring. To mitigate this, a method to quantify architectural similarity is required to guide parent selection and ensure structural compatibility.
Methodology
This study introduces Architectural Cluster Overlap as a similarity metric. The framework utilizes:
- m-Parent Evolutionary Synthesis: Synthesizing new networks based on the synaptic probabilities of multiple parents.
- Gene Tagging: A system that tracks the origin of network components from a common ancestor, ensuring "like-with-like" structures are combined during synthesis.
- Similarity Metric: Defined as the percentage overlap of architectural clusters between two networks ($A$ and $B$): $$\%overlap_{AB} = \frac{C_A \cap C_B}{C_A}$$ High overlap indicates shared functional clusters of neurons and synapses.
The researchers evaluated this approach using the MNIST dataset, tracking evolution over seven generations to observe how structures diverged or converged.
Results
The study reveals a clear distinction between networks evolved with and without gene tagging. Gene tagging successfully enforces structural consistency, as evidenced by higher overlap percentages in early-to-mid generations.
As shown in the table above, by Generation 4, networks using gene tagging maintained a 71.81% overlap, compared to 66.64% for those without. This structural alignment, while beneficial for stability, suggests a diversity trade-off: gene tagging may restrict the search space, potentially missing radical architectural efficiencies by "playing it safe."
The performance data highlights a "collapse" in later generations (6 and 7) for the "No Gene Tagging" group. As seen in the scatter plot, accuracy for these models drops to approximately 10% (random guessing). Interestingly, their architectural overlap increases during this phase (reaching 91.05% in Gen 7), not due to convergence on a solution, but because the networks become so sparse and simple that they collapse into similar, non-functional structures.
Discussion and Future Work
The results demonstrate that architectural cluster overlap is a viable metric for assessing population diversity. The highest architectural variability (lowest overlap) occurred in generations 3 and 4, which also represented an optimal balance between storage size and accuracy.
Future research will focus on scaling these findings to larger datasets like ImageNet and exploring more sophisticated information-theoretic measures, such as mutual information. Ultimately, these similarity scores could be used to create automated "mating policies" that select the most compatible parent networks to produce the most efficient offspring.
Summary updated