Abstract
Animals remember temporal links between their actions and subsequent rewards. We previously discovered a synaptic mechanism underlying such reward learning in D1 receptor (D1R)-expressing spiny projection neurons (D1 SPN) of the striatum. Dopamine (DA) bursts promote dendritic spine enlargement in a time window of only a few seconds after paired pre- and post-synaptic spiking (pre-post pairing), which is termed as reinforcement plasticity (RP). The previous study has also identified underlying signaling pathways; however, it still remains unclear how the signaling dynamics results in RP. In the present study, we first developed a computational model of signaling dynamics of D1 SPNs. The D1 RP model successfully reproduced experimentally observed protein kinase A (PKA) activity, including its critical time window. In this model, adenylate cyclase type 1 (AC1) in the spines/thin dendrites played a pivotal role as a coincidence detector against pre-post pairing and DA burst. In particular, pre-post pairing (Ca2+ signal) stimulated AC1 with a delay, and the Ca2+-stimulated AC1 was activated by the DA burst for the asymmetric time window. Moreover, the smallness of the spines/thin dendrites is crucial to the short time window for the PKA activity. We then developed a RP model for D2 SPNs, which also predicted the critical time window for RP that depended on the timing of pre-post pairing and phasic DA dip. AC1 worked for the coincidence detector in the D2 RP model as well. We further simulated the signaling pathway leading to Ca2+/calmodulin-dependent protein kinase II (CaMKII) activation and clarified the role of the downstream molecules of AC1 as the integrators that turn transient input signals into persistent spine enlargement. Finally, we discuss how such timing windows guide animals' reward learning.
Introduction
Reinforcement learning hinges on the "credit assignment problem"—the ability of an organism to associate a specific action with a reward that occurs moments later. In the brain's striatum, this process is mediated by reinforcement plasticity (RP), specifically the structural enlargement of dendritic spines in spiny projection neurons (SPNs). Experimental evidence has shown that this plasticity is highly sensitive to timing: a dopamine (DA) burst must follow neural activity within a narrow ~2-second window. This study presents a computational model to elucidate the molecular "clock" governing this temporal contiguity.
The Molecular Architecture of Reinforcement Plasticity
The researchers developed detailed biochemical kinetic models of D1 and D2 SPNs using deterministic ordinary differential equations. A central feature of the model is the identification of Adenylate Cyclase type 1 (AC1) as a critical coincidence detector. Unlike previous models focusing on AC5, this model demonstrates that AC1's sensitivity to both calcium ($Ca^{2+}$) and G-proteins is essential for detecting the overlap between action-induced signals and reward-induced dopamine.
The model highlights that $Ca^{2+}$ stimulates AC1 with a specific delay (~0.3s dead time and ~2s time constant). This delay creates the asymmetric temporal window, ensuring that the "action" (calcium signal) must precede the "reward" (dopamine burst). Furthermore, the high surface-to-volume ratio of dendritic spines allows for the rapid signaling transients necessary for such precise timing.
Results: The Temporal Window of Reward
The D1 RP model successfully reproduced the experimentally observed ~2-second asymmetric time window for Protein Kinase A (PKA) activation and subsequent spine volume changes ($\Delta V_H$). The model predicted peak activity at a DA delay of approximately 1 second, aligning closely with experimental observations.
Beyond D1 neurons, the model also predicted a similar contiguity window for D2 SPNs, which respond to dopamine dips (representing omitted rewards). In these neurons, the dip relieves AC1 from inhibition, allowing for plasticity. While PKA activity is transient, downstream molecules like CaMKII act as integrators, converting these brief signals into long-lasting structural changes.
Discussion and Future Directions
The study provides a theoretical framework for how molecular dynamics in small compartments like dendritic spines enable precise temporal learning. However, the current single-compartment model does not account for spatial diffusion between the spine and the dendrite. Future research utilizing reaction-diffusion models will be necessary to explore how these signals remain localized to specific synapses. Additionally, capturing the effects of basal dopamine levels remains a challenge for future modeling efforts.
Summary updated