Acetylcholine-modulated plasticity in reward-driven navigation: a computational study

Sara Zannone, Zuzanna Brzosko, Ole Paulsen, Claudia ClopathScientific 2018 · DOI

Abstract

Neuromodulation plays a fundamental role in the acquisition of new behaviours. In previous experimental work, we showed that acetylcholine biases hippocampal synaptic plasticity towards depression, and the subsequent application of dopamine can retroactively convert depression into potentiation. We also demonstrated that incorporating this sequentially neuromodulated Spike-Timing-Dependent Plasticity (STDP) rule in a network model of navigation yields effective learning of changing reward locations. Here, we employ computational modelling to further characterize the effects of cholinergic depression on behaviour. We find that acetylcholine, by allowing learning from negative outcomes, enhances exploration over the action space. We show that this results in a variety of effects, depending on the structure of the model, the environment and the task. Interestingly, sequentially neuromodulated STDP also yields flexible learning, surpassing the performance of other reward-modulated plasticity rules.

Introduction

Understanding how the brain bridges the gap between millisecond-level synaptic changes and minute-level behavioral learning remains a central challenge in neuroscience. While Spike-Timing-Dependent Plasticity (STDP) provides a mechanism for synaptic modification, it often fails to solve the "credit assignment problem"—identifying which specific actions in a sequence led to a delayed reward.

Dopamine (DA) is widely recognized as a reward signal that stabilizes "eligibility traces" of recent activity. However, the role of acetylcholine (ACh) in this process has been less defined. This study investigates the functional advantages of ACh-induced synaptic depression, proposing that it serves as a mechanism for "unlearning" unsuccessful actions, thereby facilitating systematic exploration.

The sn-Plast Learning Rule

The researchers implemented a spiking neural network where place cells connect to action neurons. They introduced a "sequentially neuromodulated" plasticity rule (sn-Plast) characterized by two distinct phases:

  1. Cholinergic Depression: During exploration, the presence of ACh causes active synapses to weaken. This effectively marks paths that have not yet yielded a reward.
  2. Dopaminergic Rescue: Upon reaching a reward, a DA burst retroactively converts recent depression into potentiation (strengthening) via an eligibility trace.

Unlike classic asymmetric STDP, this rule utilizes a symmetric window, where the magnitude of activity is more critical than precise spike order, aligning with recent hippocampal findings.

Systematic Exploration in the Radial Maze

The impact of ACh is most evident in discrete navigation tasks like the radial arm maze. In this environment, agents must find a reward hidden in one of eight arms.

Exploration patterns in a radial arm maze with and without acetylcholine (ACh).

Without ACh, agents engage in a random search, often re-visiting arms they have already explored. In contrast, +ACh agents utilize synaptic depression to "cross off" unsuccessful arms. By weakening the connections to arms that do not contain a reward, the model ensures it does not repeat the same mistake within a single trial, leading to a highly efficient, systematic search strategy.

Learning Efficiency and Flexibility

The computational benefits of the sn-Plast rule translate into significantly faster discovery speeds and greater adaptability in dynamic environments.

Discovery speed and success rates for reward-driven navigation tasks.

In the radial maze, +ACh agents found rewards in an average of 4.5 trials—approaching the theoretical optimum—compared to 8 trials for -ACh agents. Furthermore, when reward locations were moved (e.g., in an open field with obstacles), the +ACh model demonstrated superior flexibility. The constant pressure of cholinergic depression allowed the agent to quickly "erase" old, now-unrewarded paths, whereas standard reward-modulated STDP (r-STDP) models often remained stuck in outdated behavioral patterns.

Discussion

This study highlights that learning (driven by DA) and unlearning (driven by ACh) are functionally decoupled processes. This separation allows for a level of behavioral flexibility that single-signal models cannot achieve. While the current model uses a simplified binary dopamine signal, future iterations incorporating Reward Prediction Error (RPE) and localized ACh release could provide even deeper insights into the biological basis of adaptive navigation. By framing ACh as a tool for systematic exploration rather than just "attention," this work offers a robust computational framework for understanding neuromodulated plasticity.

Summary updated

Abstract

Neuromodulation plays a fundamental role in the acquisition of new behaviours. In previous experimental work, we showed that acetylcholine biases hippocampal synaptic plasticity towards depression, and the subsequent application of dopamine can retroactively convert depression into potentiation. We also demonstrated that incorporating this sequentially neuromodulated Spike-Timing-Dependent Plasticity (STDP) rule in a network model of navigation yields effective learning of changing reward locations. Here, we employ computational modelling to further characterize the effects of cholinergic depression on behaviour. We find that acetylcholine, by allowing learning from negative outcomes, enhances exploration over the action space. We show that this results in a variety of effects, depending on the structure of the model, the environment and the task. Interestingly, sequentially neuromodulated STDP also yields flexible learning, surpassing the performance of other reward-modulated plasticity rules.
BibTeX
@inproceedings{zannone2018acetylcholine,
  author = {Sara Zannone and Zuzanna Brzosko and Ole Paulsen and Claudia Clopath},
  title = {{Acetylcholine-modulated plasticity in reward-driven navigation: a computational study}},
  booktitle = {Scientific},
  year = {2018},
  doi = {10.1038/s41598-018-27393-2},
}
Figure