Functional Requirements for Reward-Modulated Spike-Timing-Dependent Plasticity

Nicolas Frémaux, Henning Sprekeler, Wulfram GerstnerJournal of Neuroscience 2010 · DOI

Abstract

Recent experiments have shown that spike-timing-dependent plasticity (STDP) is influenced by neuromodulation. We derive theoretical conditions for successful learning of reward-related behavior for a large class of learning rules where Hebbian synaptic plasticity is conditioned on a global modulatory factor signaling reward. We show that all learning rules in this class can be separated into a term that captures the covariance of neuronal firing and reward and a second term that presents the influence of unsupervised learning. The unsupervised term, which is, in general, detrimental for reward-based learning, can be suppressed if the neuromodulatory signal encodes the difference between the reward and the expected reward—but only if the expected reward is calculated for each task and stimulus separately. If several tasks are to be learned simultaneously, the nervous system needs an internal critic that is able to predict the expected reward for arbitrary stimuli. We show that, with a critic, reward-modulated spike-timing-dependent plasticity is capable of learning motor trajectories with a temporal resolution of tens of milliseconds.

Introduction: Bridging Hebbian Plasticity and Reinforcement Learning

A central challenge in neuroscience is understanding how local synaptic changes lead to global behavioral improvements. Historically, two frameworks have dominated: Hebbian Plasticity, which focuses on local correlations ("neurons that fire together, wire together"), and Reinforcement Learning, which focuses on maximizing a global reward signal.

Reward-modulated spike-timing-dependent plasticity (R-STDP) attempts to bridge this gap using a "three-factor rule." In this model, synaptic weight changes depend on pre-synaptic activity, post-synaptic activity, and a global neuromodulatory signal, such as dopamine, which acts as a "success signal." However, standard STDP is inherently unsupervised—it modifies weights based on timing regardless of the behavioral outcome. This paper investigates the mathematical conditions required for R-STDP to successfully maximize rewards without being overwhelmed by its own unsupervised biases.

The Mechanism of R-STDP and the Success Signal

The researchers modeled neurons using a Spike Response Model ($SRM_0$) with stochastic "escape noise." In their framework, a "candidate" weight change is first generated based on the relative timing of pre- and post-synaptic spikes. This candidate change is stored as an eligibility trace that decays over time. The "actual" weight change occurs only when a global success signal $S(R)$ arrives.

Modulation of the STDP learning window by the success signal $S(R)$. Positive success signals reinforce the Hebbian window, while negative signals invert it, effectively neutralizing unsupervised bias.

As shown in the figure above, the success signal $S(R)$ acts as a multiplier. If $S(R) > 0$, the standard STDP window is applied. If $S(R) < 0$, the window is inverted, punishing the timings that led to a poor outcome. If $S(R) = 0$, no learning occurs. This modulation is critical for steering the neuron toward reward-maximizing behavior.

The Problem of Unsupervised Bias

The authors mathematically demonstrated that the total weight change in R-STDP can be decomposed into two terms:

  1. A Reward-Sensitive Term: This captures the covariance between neuronal firing and the reward, representing the "useful" part of the learning rule.
  2. An Unsupervised Bias Term: This represents the inherent tendency of the STDP rule (e.g., a bias toward shortening spike latencies) multiplied by the average success signal.

If the average success signal is not zero, this bias term can dominate, causing the neuron to follow the "default" STDP behavior rather than the behavior that earns the most reward. For instance, if rewards are always positive, the neuron might simply fire as early as possible, even if the task requires precise late-timing spikes.

To solve this, the brain requires a stimulus-specific critic. A simple global average reward subtraction is insufficient for multi-tasking. If an animal performs Task A (high average reward) and Task B (low average reward), a global average would incorrectly label all Task A trials as successes and Task B trials as failures. A critic must predict the expected reward $\bar{R}$ for each specific stimulus, allowing the success signal to be calculated as $S = R - \bar{R}$.

Results: Learning Precise Motor Control

The effectiveness of this "critic-based" R-STDP was tested on complex tasks, including learning millisecond-precise spike patterns and 3D motor trajectories.

Learning a precise spike pattern over 5000 trials. (Left) Spike trains align with target times (red lines) as reward $R_n$ increases. (Right) Mean reward $\bar{R}_n$ saturates while the success signal $S(R)$ fluctuates around zero.

In the spike-train matching task, a neuron successfully learned to produce specific spike patterns in response to up to 32 interleaved stimuli. As illustrated in the figure above, the reward $R_n$ increases over thousands of trials as the generated spikes align with the target times. Crucially, the success signal $S(R)$ remains centered around zero, indicating that the stimulus-specific critic is effectively neutralizing the unsupervised bias.

Conclusion and Biological Implications

The study concludes that R-STDP is a powerful mechanism for high-precision learning, achieving temporal resolutions of 15–20 ms. However, its success is contingent on the presence of a reward prediction mechanism. While the paper does not detail how this "critic" is learned, it suggests that mechanisms like Temporal Difference (TD) learning might be responsible. Furthermore, the authors found that "block learning"—practicing one task repeatedly—can serve as a biological workaround for a complex critic, as the local average reward naturally adapts to the current task. This research provides a rigorous foundation for how neuromodulation transforms local plasticity into global behavioral adaptation.

Summary updated

Abstract

Recent experiments have shown that spike-timing-dependent plasticity is influenced by neuromodulation. We derive theoretical conditions for successful learning of reward-related behavior for a large class of learning rules where Hebbian synaptic plasticity is conditioned on a global modulatory factor signaling reward. We show that all learning rules in this class can be separated into a term that captures the covariance of neuronal firing and reward and a second term that presents the influence of unsupervised learning. The unsupervised term, which is, in general, detrimental for reward-based learning, can be suppressed if the neuromodulatory signal encodes the difference between the reward and the expected reward-but only if the expected reward is calculated for each task and stimulus separately. If several tasks are to be learned simultaneously, the nervous system needs an internal critic that is able to predict the expected reward for arbitrary stimuli. We show that, with a critic, reward-modulated spike-timing-dependent plasticity is capable of learning motor trajectories with a temporal resolution of tens of milliseconds. The relation to temporal difference learning, the relevance of block-based learning paradigms, and the limitations of learning with a critic are discussed.
BibTeX
@article{fremaux2010functional,
  author = {Nicolas Frémaux and Henning Sprekeler and Wulfram Gerstner},
  title = {{Functional Requirements for Reward-Modulated Spike-Timing-Dependent Plasticity}},
  journal = {Journal of Neuroscience},
  year = {2010},
  doi = {10.1523/jneurosci.6249-09.2010},
}
Figure