A behavioural correlate of the synaptic eligibility trace in the nucleus accumbens

Kenji Yamaguchi, Yoshitomo Maeda, Takeshi Sawada, Yusuke Iino, Mio Tajiri, Ryosuke Nakazato, Shin Ishii, Haruo Kasai, Sho YagishitaScientific 2022 · DOI

TL;DR

Researchers have identified a 1-second "reward-sensitive window" in mouse behavior that perfectly matches the cellular "eligibility trace" previously seen in brain slices. By using precise Pavlovian conditioning and optogenetics, they demonstrated that the Nucleus Accumbens (NAc) acts as a molecular timer, requiring dopamine (reward) to arrive within one second of a stimulus to successfully reinforce learning.

What Problem Does This Paper Address?

In Reinforcement Learning (RL), the "credit assignment problem" asks: how does a brain know which specific action or stimulus led to a reward if that reward arrives seconds or minutes later?

Theoretically, neurons handle this via an eligibility trace—a temporary "tag" left at a synapse after activity that makes it sensitive to a subsequent reward signal (like dopamine). While brain slice experiments have shown that synapses in the Nucleus Accumbens (NAc) have a biochemical window of about 1 second, it wasn't clear if this cellular timer actually dictated animal behavior. Many behavioral studies show animals learning with reward delays of 10–60 seconds, suggesting a disconnect between the 1-second cellular trace and real-world learning.

Methods

The authors used a highly controlled Pavlovian conditioning setup with head-fixed mice to measure the "effective timing" of rewards with sub-second precision.

  1. Precise Pavlovian Conditioning: Mice were presented with a brief pure tone (0.5s) followed by a water reward at varying intervals (from -1s to +2s).
  2. Pharmacology & Viral Vectors: To prove the NAc was the hub, they used an inhibitory peptide (AIP) to block CaMKII (a key plasticity enzyme) and a D1 receptor antagonist (SCH23390) specifically within the NAc.
  3. Optogenetics: To eliminate any "lag" from sensory processing (the time it takes for a sound to reach the brain), they used CSopto. They directly stimulated the glutamatergic inputs from the basolateral amygdala (BLA) to the NAc using blue light, effectively "injecting" the stimulus signal directly into the reward circuit.

Datasets

As a biological study, the "datasets" consist of behavioral and histological data from several cohorts of mice:

  • Wild-type (B6J) mice for standard conditioning.
  • DAT-IRES-Cre mice for specific dopaminergic manipulations.
  • Histological imaging: Confocal microscopy was used to verify the precise placement of optical fibers and the spread of viral expressions (like the AIP-Clover construct) in the NAc vs. the Prefrontal Cortex (PFC).

Key Insights and Ideas

  • The 1-Second Hard Limit: The most striking insight is the rigidity of the window. Learning was robust if the reward arrived within 1 second of the stimulus but dropped to zero at 2 seconds. This suggests the NAc uses a specific molecular timer rather than a general "the sooner, the better" rule.
  • The NAc is the Timer: By blocking CaMKII in the NAc, learning was abolished. Crucially, blocking the same pathway in the Prefrontal Cortex (PFC) had no effect on this rapid learning task, suggesting the NAc is the primary site for this specific 1-second eligibility trace.
  • Bypassing the Senses: The optogenetic experiment (CSopto) confirmed that the 1-second window is an intrinsic property of the NAc synapses. Even when the stimulus was delivered directly to the NAc neurons, the reward still had to arrive within that 1-second window to trigger learning.
  • Resolving the Trade-off: The authors suggest this 1-second window is a biological compromise. It is long enough to allow for natural delays (like the time between smelling food and tasting it) but short enough to maintain high "temporal contiguity," ensuring the brain doesn't accidentally associate a reward with the wrong preceding event.

Key Results

  • Behavioral Window: In the tone-water task, significant learning (measured by anticipatory licking) only occurred at reward delays of +0s, +0.5s, and +1s. At +2s, the mice failed to associate the tone with the water.
  • Molecular Necessity: Injection of the D1 receptor blocker or the CaMKII inhibitor (AIP) into the NAc significantly inhibited the formation of conditioned responses ($P < 0.05$).
  • Optogenetic Confirmation: Direct stimulation of BLA-to-NAc inputs showed a nearly identical reward-sensitive window, peaking at 1 second and failing at 2 seconds.

Limitations and Future Work

The primary limitation is the discrepancy between this 1-second window and complex behaviors where rewards are delayed by much longer periods (e.g., a hunter tracking prey for minutes).

The authors acknowledge that while the NAc handles this "synaptic" eligibility trace, other brain regions like the PFC likely use persistent activity (working memory) to bridge longer gaps. Future work will need to explore how these short-term synaptic traces in the NAc interact with long-term "representative" traces in the cortex to enable complex, multi-step reinforcement learning. Additionally, the study focuses on D1-receptor-expressing neurons; the role of D2-receptor-expressing neurons in this timing window remains a question for future research.

Summary updated

Abstract

Reward reinforces the association between a preceding sensorimotor event and its outcome. Reinforcement learning (RL) theory and recent brain slice studies explain the delayed reward action such that synaptic activities triggered by sensorimotor events leave a synaptic eligibility trace for 1 s. The trace produces a sensitive period for reward-related dopamine to induce synaptic plasticity in the nucleus accumbens (NAc). However, the contribution of the synaptic eligibility trace to behaviour remains unclear. Here we examined a reward-sensitive period to brief pure tones with an accurate measurement of an effective timing of water reward in head-fixed Pavlovian conditioning, which depended on the plasticity-related signaling in the NAc. We found that the reward-sensitive period was within 1 s after the pure tone presentation and optogenetically-induced presynaptic activities at the NAc, showing that the short reward-sensitive period was in conformity with the synaptic eligibility trace in the NAc. These findings support the application of the synaptic eligibility trace to construct biologically plausible RL models.
BibTeX
@inproceedings{yamaguchi2022behavioural,
  author = {Kenji Yamaguchi and Yoshitomo Maeda and Takeshi Sawada and Yusuke Iino and Mio Tajiri and Ryosuke Nakazato and Shin Ishii and Haruo Kasai and Sho Yagishita},
  title = {{A behavioural correlate of the synaptic eligibility trace in the nucleus accumbens}},
  booktitle = {Scientific},
  year = {2022},
  doi = {10.1038/s41598-022-05637-6},
}
Figure