In standard deep learning, designing a neural network is a decoupled, two-step process: humans (or architecture search algorithms) define a rigid, fixed topology, and gradient descent optimizes the weights. This separation assumes that the ideal structure can be guessed beforehand. However, fixed topologies often suffer from being either overly dense—wasting computational resources—or poorly matched to the underlying problem geometry.
A powerful alternative is to treat both the physical wiring diagram (topology) and the synaptic strengths (weights) as a single, jointly evolving system. By using evolutionary algorithms instead of gradient descent, networks can grow, prune, and reshape themselves dynamically to fit the task at hand.
The Intuition of Joint Optimization
Why optimize structure and weights simultaneously? When connectivity is treated as a variable alongside weight values, the search process can discover highly efficient, sparse architectures tailored to specific data patterns without manual structural engineering (Zanchettin et al. in Hybrid Technique for Artificial Neural Network Architecture and Weight Optimization).
In traditional training, gradients require a fixed, differentiable scaffold. By moving away from derivative-based constraints, algorithms can optimize non-standard, brain-inspired topologies—such as Spiking Neural Networks (SNNs) or Continuous-Time Recurrent Neural Networks (CTRNNs)—where backpropagation is difficult or impossible (Wang et al. in Brain topology improved spiking neural network for efficient reinforcement learning of continuous control; Lotito et al. in A Signal-Centric Perspective on the Evolution of Symbolic Communication).
Furthermore, combining unsupervised biological rules with evolutionary search allows networks to prune unnecessary connections automatically. For instance, Shen et al. demonstrated in Jointly Learning Network Connections and Link Weights in Spiking Neural Networks that joint optimization provides a superior structural scaffold, eliminating roughly 24% of redundant connections while improving accuracy. Similarly, ElSaid et al. showed in Backpropagation-Free 4D Continuous Ant-Based Neural Topology Search that treating weights as spatial coordinates enables the simultaneous discovery of topology and strength without needing separate training phases.
Incremental Complexification: Starting Small
A naive approach to evolving networks might involve generating large, random graphs and mutating them. However, this leads to a computational explosion and a massive search space. The core mechanism that makes topology evolution practical is incremental complexification.
Instead of searching through complex architectures from the start, the process begins with a minimal topology—typically just input and output nodes with no hidden layers (Whiteson et al. in Critical factors in the empirical performance of temporal difference and evolutionary methods for reinforcement learning). Over generations, genetic mutations incrementally add new neurons and connections (Pham et al. in Playing SNES Games With NeuroEvolution of Augmenting Topologies).
This gradual expansion ensures that the algorithm searches the lowest-dimensional spaces first. Complexity is introduced only when a task demands it to achieve better fitness. As a result, the system discovers highly compact, task-specific connectivity patterns. For example, Zhu et al. showed in Spiking Neural Networks as a Controller for Emergent Swarm Agents that this mechanism can produce complex emergent behaviors in robot swarms using minimal controllers consisting of fewer than fifteen neurons.
Topology-Aware Operators
Why did early attempts at evolving neural networks struggle? The answer lies in how genetic operators interact with network structures. If an algorithm treats a network simply as a flat vector of weights, standard crossover operators will randomly swap values between parents. This destroys learned features because the same functional role might be handled by different structural nodes in each parent—a problem known as competing conventions.
To solve this, evolutionary operators must be topology-aware. As early as 1989, Montana et al. recognized in Training Feedforward Neural Networks Using Genetic Algorithms that mutation and crossover must respect the underlying functional units of the network. By manipulating all incoming weights of a specific node as a single block rather than applying random weight-level perturbations, topology-aware operators preserve learned features and allow structural innovations to be passed down meaningfully without destroying functional integrity.
Real-Time Adaptation and Structural Flexibility
Because topology-evolving systems do not rely on storing gradients or activations, they bypass the high memory overhead of backpropagation, making them highly suitable for edge devices (Mannan et al. in CLAN: Continuous Learning using Asynchronous Neuroevolution on Commodity Edge Devices). This structural flexibility opens up capabilities that are difficult to achieve with rigid architectures:
- Real-Time and Lifelong Learning: By updating populations asynchronously, networks can adapt their topologies during live execution without offline training phases (Stanley et al. in The NERO Video Game). To prevent the catastrophic forgetting typical of neural networks during lifelong adaptation, frameworks can introduce genetic distance regularization to protect historical structural innovations (Leuzzi et al. in Lifelong Evolution of Swarms).
- Grafting External Knowledge: Because the architecture is dynamic, external sub-networks—such as human-generated symbolic rules or "if-then" advice—can be directly grafted into an agent's topology without disrupting established functions (Yong et al. in Incorporating Advice into Neuroevolution of Adaptive Agents). Evolution can then seamlessly refine this hybrid structure over time.
By tightly coupling the evolution of form (topology) and function (weights), these algorithms mirror biological development, allowing compact and highly specialized intelligence to emerge organically from minimal beginnings.
Key Foundational Work
Real-time Learning in the NERO Video Game (2005) — contributed 2 concept elements to this direction.
Training Feedforward Neural Networks Using Genetic Algorithms (1989) — contributed 2 concept elements to this direction.
Hybrid Technique for Artificial Neural Network Architecture and Weight Optimization (2005) — contributed 1 concept elements to this direction.