Striatal dopamine reflects individual long-term learning trajectories

Samuel Liebana Garcia, Aeron Laffere, Chiara Toschi, Louisa Schilling, Jacek Podlaski, Matthias Fritsche, Peter Zatka-Haas, Yulong Li, Rafał Bogacz, Andrew Saxe, Armin LakbioRxiv 2023 · DOI

TL;DR

This paper reveals that long-term learning in mice involves diverse yet systematic shifts in strategy, with initial biases predicting future behavior. Dopamine signals in the striatum dynamically track these evolving strategies, reflecting stimulus-choice associations. A novel deep reinforcement learning model successfully replicates these behavioral and neural dynamics, suggesting a hierarchical saddle-point structure governs learning.

What Problem Does This Paper Address?

Learning is a fundamental cognitive process, but how brains transition from a naive state to expert performance over extended periods remains a significant mystery. Previous research often focused on comparing naive versus expert states or on fine-tuning already learned tasks, largely overlooking the dynamic, individualistic paths animals take during this learning process. This gap leaves us without a clear understanding of the underlying principles governing long-term learning trajectories, both behaviorally and neurally. Specifically, the role of dopamine, a key neurotransmitter in learning, in shaping these diverse yet systematic learning paths is not well understood.

Methods

The researchers developed a sophisticated visual decision-making task for head-fixed mice. This task allowed them to track individual learning trajectories over several weeks. Key methodological innovations include:

  • Longitudinal Dopamine Imaging: Using fiber photometry with GRAB-DA sensors, they recorded dopamine release in the dorsolateral striatum (DLS) daily throughout the learning process. This provided a high-resolution view of dopamine's dynamic role.
  • Deep Reinforcement Learning Model: They designed a novel "tutor-executor" deep neural network. This model features a layered architecture and a unique learning rule that uses distinct reward prediction errors (RPEs) to train different parts of the network. This approach was specifically designed to mimic the observed dopamine signals.
  • Saddle Point Analysis: The model's learning dynamics were analyzed using saddle point theory to explain the systematic transitions in behavior and neural signals.

Datasets

The study utilized data collected from 28 male wild-type C57/BL6J mice trained on a visual decision-making task. No external datasets were used for training or evaluation; the study relied entirely on the data generated from their experimental setup.

Key Insights and Ideas

  • Individual Learning Paths are Diverse but Systematic: Mice don't learn in lockstep. They adopt varied strategies, but these transitions are remarkably predictable, with early biases dictating later behavior. This suggests underlying organizational principles guiding learning.
  • Dopamine as a Strategy Tracker: Striatal dopamine signals aren't just about immediate rewards; they evolve over time to reflect the specific stimulus-choice associations an individual mouse has learned. This highlights dopamine's role in encoding learned strategies.
  • Saddle Points as Learning Landmarks: The deep RL model's success in replicating the data suggests that learning might navigate a landscape of "saddle points." These points could represent critical junctures or stable states in the learning process, explaining the systematic transitions observed.
  • "Tutor-Executor" Learning Rule: The proposed learning rule, which uses different RPEs for different network components, is a novel concept. It offers a concrete mechanism for how dopamine might drive learning by providing distinct teaching signals.

Key Results

  • Behavioral Diversity: Mice showed significant individual variability in their learning strategies, with some developing left-biased, right-biased, or balanced approaches.
  • Predictive Early Biases: Initial biases in choice behavior strongly predicted the later development of psychometric slopes and overall strategy.
  • Dopamine Encoding of Associations: Dopamine responses to stimuli emerged and strengthened as mice learned stimulus-choice associations, mirroring the development of their behavioral strategies.
  • Model Success: The "tutor-executor" deep RL model accurately reproduced both the diverse behavioral trajectories and the dynamic dopamine signals observed in mice.
  • Saddle Point Dynamics: The model's analysis revealed a hierarchy of saddle points that explained the systematic progression through learning strategies.

Limitations and Future Work

  • Task Specificity: The findings are based on a specific visual decision-making task. Future work could explore if these principles generalize to other types of learning and tasks.
  • Causality of Dopamine: While the study shows a strong correlation between dopamine signals and learning, future research could directly investigate the causal role of these dopamine signals in driving long-term learning.
  • Curriculum Design: The study used a fixed, relatively difficult task from the start. Investigating how different training curricula might influence learning trajectories and dopamine dynamics would be valuable.
  • Other Brain Regions: The study focused on the dorsal striatum. Future research could explore dopamine's role in other brain regions and circuits during long-term learning.

Summary updated

Abstract

Abstract Learning from naïve to expert occurs over long periods of time, accompanied by changes in the brain’s neuronal signals. The principles governing behavioural and neuronal dynamics during long-term learning remain unknown. We developed a psychophysical visual decision task for mice that allowed for studying learning trajectories from naïve to expert. Mice adopted sequences of strategies that became more stimulus-dependent over time, showing substantial diversity in the strategies they transitioned through and settled on. Remarkably, these transitions were systematic; the initial strategy of naïve mice predicted their strategy several weeks later. Longitudinal imaging of dopamine release in dorsal striatum demonstrated that dopamine signals evolved over learning, reflecting stimulus-choice associations linked to each individual’s strategy. A deep neural network model trained on the task with reinforcement learning captured behavioural and dopamine trajectories. The model’s learning dynamics accounted for the mice’s diverse and systematic learning trajectories through a hierarchy of saddle points. The model used prediction errors mirroring recorded dopamine signals to update its parameters, offering a concrete account of striatal dopamine’s role in long-term learning. Our results demonstrate that long-term learning is governed by diverse yet systematic transitions through behavioural strategies, and that dopamine signals exhibit key characteristics to support this learning.
BibTeX
@misc{garcia2023striatal,
  author = {Samuel Liebana Garcia and Aeron Laffere and Chiara Toschi and Louisa Schilling and Jacek Podlaski and Matthias Fritsche and Peter Zatka-Haas and Yulong Li and Rafał Bogacz and Andrew Saxe and Armin Lak},
  title = {{Striatal dopamine reflects individual long-term learning trajectories}},
  year = {2023},
  doi = {10.1101/2023.12.14.571653},
  howpublished = {bioRxiv},
}
Figure