Hamiltonian synaptic sampling in a model for reward-gated network plasticity

Zhaofei Yu, David Kappel, Robert Legenstein, Sen Song, Feng Chen, Wolfgang Maass

Abstract

Synaptic plasticity is implemented and controlled through over a thousand different types of molecules in the postsynaptic density and presynaptic boutons. One of the most prominent molecules in the postsynaptic density is CaMKII, described in molecular biology as a "memory molecule" that can integrate calcium-influx signals through autophosphorylation on a time scale of dozens of seconds. We show that the experimental data on CaMKII activation suggest a general functional role in speeding up reward-guided search for network configurations that maximize reward expectation. Our theoretical analysis shows that CaMKII contributes a momentum term that substantially speeds up stochastic search, allowing the network to overcome saddle points of the fitness function. This improved stochastic policy search can be understood as Hamiltonian sampling, one of the most efficient stochastic search methods known.

Introduction

Learning in the brain is inherently noisy and stochastic. Previous theoretical frameworks, such as "synaptic sampling," suggested that the brain optimizes its connections by performing Langevin sampling—essentially a version of stochastic gradient descent where noise is added to help explore the parameter space. However, Langevin sampling is often slow in the high-dimensional "fitness landscapes" of neural networks, where learning frequently gets stuck at saddle points. This paper investigates whether specific biological molecules, like the protein kinase CaMKII, provide a functional "speed boost" to help the brain overcome these optimization hurdles.

The Biological Momentum of CaMKII

The researchers developed a mathematical model integrating the known dynamics of CaMKII into synaptic plasticity rules. Instead of updating synaptic weights ($\theta_i$) directly based on a reward signal, the reward signal updates the concentration of activated CaMKII ($\Gamma_i$), which then slowly drives the change in synaptic weight.

CaMKII activation and temporal integration of calcium pulses.

Because CaMKII stays active for dozens of seconds, it acts as a low-pass filter for calcium signals. In optimization terms, this temporal integration is equivalent to a momentum term. The authors proved using Stochastic Differential Equations that this dynamics implements Hamiltonian sampling with friction, a well-known method in physics and machine learning for faster convergence compared to standard Langevin sampling.

Results and Performance

The model was evaluated across several computational tasks, including sigmoidal emulation, a blind reaching task, and classification on the MNIST dataset.

Network model and performance comparison in a sigmoidal emulation task.

Key results include:

  • Faster Learning: In sigmoidal emulation and reaching tasks, Hamiltonian dynamics (with CaMKII) reached high reward levels significantly faster than standard synaptic sampling.
  • Escaping Plateaus: In the MNIST task, non-Hamiltonian networks frequently hit performance plateaus at saddle points. When switched to Hamiltonian dynamics, the same networks escaped these plateaus almost immediately.
  • Global Optimization: In the XOR task, using a "cooling" schedule (simulated annealing) with Hamiltonian sampling allowed the network to find the optimal solution in 90% of trials, compared to only 40% when the temperature was kept constant.

Conclusion

This work demonstrates that a biological implementation (CaMKII molecules) directly enables a specific algorithm (Hamiltonian sampling) to solve a fundamental computational problem (reward-based optimization). By providing momentum, CaMKII allows the network's parameters to "slide" through flat regions of the fitness landscape rather than getting stuck, significantly accelerating the brain's ability to learn and adapt.

Summary updated

Abstract

Synaptic plasticity is implemented and controlled through over thousand different types of molecules in the postsynaptic density and presynaptic boutons that assume a staggering array of different states through phosporylation and other mechanisms. One of the most prominent molecule in the postsynaptic density is CaMKII, that is described in molecular biology as a "memory molecule" that can integrate through auto-phosporylation Ca-influx signals on a relatively large time scale of dozens of seconds. The functional impact of this memory mechanism is largely unknown. We show that the experimental data on the specific role of CaMKII activation in dopamine-gated spine consolidation suggest a general functional role in speeding up reward-guided search for network configurations that maximize reward expectation. Our theoretical analysis shows that stochastic search could in principle even attain optimal network configurations by emulating one of the most well-known nonlinear optimization methods, simulated annealing. But this optimization is usually impeded by slowness of stochastic search at a given temperature. We propose that CaMKII contributes a momentum term that substantially speeds up this search. In particular, it allows the network to overcome saddle points of the fitness function. The resulting improved stochastic policy search can be understood on a more abstract level as Hamiltonian sampling, which is known to be one of the most efficient stochastic search methods.
BibTeX
@misc{yu2016hamiltonian,
  author = {Zhaofei Yu and David Kappel and Robert Legenstein and Sen Song and Feng Chen and Wolfgang Maass},
  title = {{Hamiltonian synaptic sampling in a model for reward-gated network plasticity}},
  year = {2016},
  eprint = {1606.00157},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1606.00157},
}
Figure