Learning Through Dynamic Synaptic Rewiring

24 concepts22 papers1 proposal

In traditional deep learning, we treat the architecture of a neural network as a fixed container, and learning as the process of filling that container with high-precision numbers. We initialize a dense web of connections and use gradient descent to continuously tweak their weights.

But biological brains and resource-constrained neuromorphic hardware operate under a fundamentally different paradigm. Instead of adjusting the strength of a fixed set of connections, they learn by changing the topology of the network itself—physically creating new connections and pruning obsolete ones. This mechanism, known as structural plasticity, shifts the computational burden of learning from high-precision weight tuning to spatial routing.

The Power of Placement

To understand why changing topology is an effective way to learn, we must look at how individual neurons process information. In a standard artificial neuron, inputs are weighted and summed linearly. However, biological neurons possess complex, branching structures called dendrites that exhibit localized nonlinear processing.

Poirazi et al. demonstrated that by introducing a simple nonlinearity to individual dendritic branches, a neuron's memory capacity increases 15- to 20-fold compared to a standard linear model. Under this framework, the exact strength of a synapse matters far less than its physical location. If a cluster of synapses fires together on the same nonlinear dendritic branch, they trigger a powerful local response.

This architectural feature means that a network can achieve high representational capacity even if its synapses are entirely binary (either fully connected or completely disconnected). Instead of tuning a continuous weight value from $0.0$ to $1.0$, the learning algorithm simply decides whether a connection should exist. As Basu et al. showed, selecting a specific subset of input connections from a larger pool of potential pathways allows binary-synapse networks to match the performance of traditional continuous-weight models while using a fraction of the physical layout.

Escaping Local Minima via Pruning and Regrowth

If we want to train a sparse network, why not just prune away the weakest connections at the start and train the remainder? The problem with static pruning is that it permanently locks the network into its initial, random topology. If the initial configuration lacks the specific pathways needed to solve a complex task, the network becomes trapped in a suboptimal state.

Dynamic synaptic rewiring solves this by turning pruning into a continuous, two-way cycle:

  1. Pruning: Connections that fall below a certain threshold of efficacy or fitness are systematically removed.
  2. Regrowth: New connections are stochastically or deterministically generated in other parts of the network, allowing the model to explore entirely new structural configurations.
[ Initial Sparse Topology ] ──( Train & Evaluate )──> [ Identify Low-Fitness Synapses ]
           ▲                                                        │
           │                                                        ▼
[ Regrow in High-Momentum Regions ] <──( Disconnect )──────── [ Prune Weak Synapses ]

To scale this to deep Spiking Neural Networks (SNNs), researchers use gradients and momentum to guide the regrowth phase. For instance, Chen et al. introduced "Gradient Rewiring" (Grad R), an algorithm that jointly optimizes weights and connectivity by allowing pruned synapses to dynamically regrow during training. If a disconnected pathway begins to exhibit a significant gradient trend, the algorithm re-establishes the connection.

Similarly, Yan et al. and Shen et al. utilize gradient momentum to target regrowth to the regions of the network under the highest demand. By continuously reallocating synaptic resources from low-utility areas to high-momentum pathways, the network self-organizes into an optimal wiring diagram. This competitive interplay allows models to maintain high accuracy even when more than 99% of potential connections are absent.

Hardware Efficiency and Neuromorphic Advantages

The practical motivation for structural plasticity becomes clear when deploying models on neuromorphic hardware, such as Very Large Scale Integration (VLSI) chips. Storing high-precision floating-point weights requires significant memory bandwidth and power, which defeats the energy-saving purpose of spike-based computing.

By leveraging structural plasticity, hardware designers can swap out power-hungry continuous memory for binary or low-resolution synapses. Hussain et al. demonstrated that a spike-based classifier utilizing morphological learning—updating topology based on a margin-maximization fitness metric—could achieve competitive results on MNIST while using up to 90% fewer synaptic resources.

This efficiency extends to recurrent networks like Liquid State Machines (LSMs). Roy et al. and the authors showed that applying an online structural plasticity rule to a reservoir's readout stage and internal structure significantly boosts fading memory and computational power. Rerouting connections based on local spike timings allows the reservoir to adapt to specific input patterns without requiring high-resolution synaptic memory, making it highly compatible with low-power edge devices.

Rewiring as Bayesian Exploration

Why does this stochastic swapping of connections work so robustly from a theoretical standpoint? Hiratani et al. proposed an elegant mathematical framing: individual synapses can be viewed as "particles" in a particle filtering algorithm.

In a particle filter, particles estimate a probability distribution by tracking high-likelihood regions; low-probability particles are discarded, and new ones are sampled near successful candidates. In a neural network, eliminating weak synapses and stochastically generating new ones acts as a physical resampling mechanism. It shifts structural resources away from low-probability configurations toward high-probability ones, accelerating learning and preventing the network from getting stuck in local minima.

This mathematical intuition aligns closely with biological observations. In nature, neural circuits are not static blueprints. Tsimring et al. tracked individual synapses in the mouse visual cortex during developmental critical periods and discovered that circuit formation is characterized by massive synaptic turnover rather than gradual weight refinement. The brain aggressively prunes and replaces a vast majority of its connections to discover optimal configurations for sensory integration.

Furthermore, single-author work by Miner demonstrated that when simple, local structural plasticity rules—such as pruning low-efficacy links and generating new ones based on spatial distance—are applied to spiking networks, complex, non-random wiring motifs identical to those found in the mammalian cortex emerge spontaneously.

Conclusion

Dynamic synaptic rewiring proves that a network's architecture does not need to be a static constraint designed prior to training. By treating the presence of a connection as a dynamic variable, structural plasticity allows networks to continuously explore the space of possible topologies. Whether guided by biological rules like Spike-Timing Dependent Plasticity (Huo et al.), Bayesian sampling (Ozdenizci et al.), or gradient momentum, the ability to dynamically rewrite the wiring diagram enables neural networks to achieve extreme compression, robust adversarial resistance, and high computational capacity with minimal hardware overhead.

Key Foundational Work

Bio-inspired Real Time Sensory Map Realignment in a Robotic Barn Owl (2008) — contributed 1 concept elements to this direction.
Memory Capacity of Linear vs. Nonlinear Models of Dendritic Integration (1999) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_structural_plasticity_b2414958,
  title = {structural plasticity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/structural-plasticity/rdr_b24149587c684ed3aad26f29b28362b9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_structural_plasticity_b2414958_v1,
  title = {structural plasticity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/structural-plasticity/rdr_b24149587c684ed3aad26f29b28362b9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake