CaMKII activation supports reward-based neural network optimization through Hamiltonian sampling

Zhaofei Yu, David Kappel, Robert Legenstein, Sen Song, Feng Chen, Wolfgang Maass

Abstract

Synaptic plasticity is implemented and controlled through over thousand different types of molecules in the postsynaptic density and presynaptic boutons that assume a staggering array of different states through phosphorylation and other mechanisms. One of the most prominent molecules in the postsynaptic density is CaMKII, described in molecular biology as a "memory molecule" that can integrate Ca-influx signals through auto-phosphorylation on a relatively large time scale of dozens of seconds. The functional impact of this memory mechanism is largely unknown. We show that experimental data on the specific role of CaMKII activation in dopamine-gated spine consolidation suggest a general functional role in speeding up reward-guided search for network configurations that maximize reward expectation. Our theoretical analysis shows that stochastic search could in principle even attain optimal network configurations by emulating simulated annealing. However, this optimization is usually impeded by the slowness of stochastic search. We propose that CaMKII contributes a momentum term that substantially speeds up this search, allowing the network to overcome saddle points of the fitness function. The resulting improved stochastic policy search can be understood as Hamiltonian sampling, one of the most efficient stochastic search methods.

Introduction: The Optimization Challenge in the Brain

Learning in biological neural networks is a high-dimensional optimization problem. Synapses must adjust their weights to maximize reward, a process often modeled as "synaptic sampling." Mathematically, this stochastic search is equivalent to Langevin sampling. While theoretically sound, Langevin sampling is notoriously slow in high-dimensional spaces, frequently stalling at saddle points—regions where the gradient is flat in some directions, preventing the network from finding global optima.

CaMKII as a Molecular Momentum Term

The molecule CaMKII is a major component of the postsynaptic density and is essential for long-term learning. Its unique property is its ability to maintain activation through autophosphorylation long after the initial calcium signal has subsided.

The molecular activation process of CaMKII, showing how autophosphorylation leads to persistent, calcium-independent activation.

The researchers propose that this persistent activation acts as a hidden variable that interacts with the visible synaptic weight. By modeling this interaction through Stochastic Differential Equations (SDEs), they demonstrate that CaMKII activation functions as a low-pass filter on parameter updates. In the language of machine learning, this is equivalent to momentum. This molecular mechanism upgrades the brain's search strategy from simple Langevin sampling to Hamiltonian sampling with friction.

Bridging Biology and Algorithm

This framework connects Marr’s implementational level (CaMKII molecules) to the algorithmic level (Hamiltonian dynamics). Hamiltonian dynamics provide the "inertia" necessary to glide through flat regions of the fitness landscape, such as saddle points, where simpler gradient-based methods would fail.

Furthermore, the authors suggest that by gradually reducing the level of stochastic noise—a process known as "cooling"—the network can perform simulated annealing. This allows the system to settle into globally optimal configurations rather than getting trapped in local minima.

Key Results and Performance Gains

The model was tested across several tasks, including sigmoidal emulation, a blind reaching task, and MNIST classification. In every instance, the inclusion of the Hamiltonian momentum term significantly improved learning efficiency.

A diagram of the neural network task and a graph showing the significantly faster reward acquisition achieved with Hamiltonian sampling compared to standard sampling.

In the sigmoidal emulation and reaching tasks, Hamiltonian dynamics reached higher reward levels much faster than standard sampling. In the blind reaching task, the success rate with Hamiltonian dynamics reached approximately 62%, compared to less than 40% for the non-Hamiltonian version. Notably, in MNIST classification, the Hamiltonian model was able to "break out" of performance plateaus (saddle points) that stalled standard sampling methods. For the XOR task, using a cooling schedule increased the probability of finding the global optimum from 40% to 90%.

Conclusion and Future Directions

This research provides a functional explanation for the long-observed persistence of CaMKII activation. By acting as a momentum term, CaMKII enables biological networks to employ Hamiltonian sampling, a highly efficient optimization strategy. While the model uses a simplified linear representation of CaMKII, the actual 12-subunit structure of the molecule may offer even more complex computational advantages. Future work will need to explore how the brain regulates "synaptic temperature" and how these biological rules scale to larger, more complex neural architectures.

Summary updated

Abstract

Synaptic plasticity is implemented and controlled through over thousand different types of molecules in the postsynaptic density and presynaptic boutons that assume a staggering array of different states through phosporylation and other mechanisms. One of the most prominent molecule in the postsynaptic density is CaMKII, that is described in molecular biology as a "memory molecule" that can integrate through auto-phosporylation Ca-influx signals on a relatively large time scale of dozens of seconds. The functional impact of this memory mechanism is largely unknown. We show that the experimental data on the specific role of CaMKII activation in dopamine-gated spine consolidation suggest a general functional role in speeding up reward-guided search for network configurations that maximize reward expectation. Our theoretical analysis shows that stochastic search could in principle even attain optimal network configurations by emulating one of the most well-known nonlinear optimization methods, simulated annealing. But this optimization is usually impeded by slowness of stochastic search at a given temperature. We propose that CaMKII contributes a momentum term that substantially speeds up this search. In particular, it allows the network to overcome saddle points of the fitness function. The resulting improved stochastic policy search can be understood on a more abstract level as Hamiltonian sampling, which is known to be one of the most efficient stochastic search methods.
BibTeX
@misc{yu2016camkii,
  author = {Zhaofei Yu and David Kappel and Robert Legenstein and Sen Song and Feng Chen and Wolfgang Maass},
  title = {{CaMKII activation supports reward-based neural network optimization through Hamiltonian sampling}},
  year = {2016},
  eprint = {1606.00157},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1606.00157},
}
Figure