The role of the thalamus in human reinforcement learning

Antoine Collomb‐Clerc, Maëlle C. M. Gueguen, Lorella Minotti, Philippe Kahane, Vincent Navarro, Fabrice Bartoloméi, Romain Carron, Regis Jean, Stéphan Chabardès, Stefano Palminteri, Julien BastinbioRxiv 2022 · DOI

TL;DR

This study provides the first direct evidence of the thalamus's role in human reinforcement learning by recording electrical activity from deep brain stimulation electrodes in patients. They discovered that specific low-frequency brain oscillations in the thalamus encode crucial learning signals, particularly for punishment avoidance, offering new insights into how our brains learn from negative consequences.

What Problem Does This Paper Address?

Reinforcement learning, the process by which we learn from rewards and punishments, is fundamental to survival and decision-making. While we know that certain brain circuits, like the fronto-striatal system and dopamine pathways, are involved, the specific role of the thalamus, a key relay station in the brain, has remained largely unclear in humans. Previous research has primarily focused on reward-based learning, leaving a significant gap in our understanding of how the thalamus contributes to learning from negative outcomes, especially punishment avoidance. This paper aims to fill this gap by investigating thalamic activity during both reward-seeking and punishment-avoidance learning tasks in humans.

Methods

The researchers utilized a unique approach by recording intracranial electrophysiological signals (local field potentials or LFPs) from eight patients undergoing deep brain stimulation (DBS) for epilepsy. These DBS electrodes, strategically placed in the anterior and dorsomedial thalamic nuclei, allowed for rare, direct recordings of neural activity within this subcortical structure.

The patients performed a well-validated instrumental learning task where they had to learn associations between abstract visual cues and monetary rewards or punishments. This task was designed to disentangle reward-seeking from punishment-avoidance learning.

To analyze the neural data, the researchers employed time-frequency analysis to examine spectral power in different frequency bands. They then used a general linear model to regress this neural activity against computational variables derived from a Q-learning model. These variables included expected values (Q-values) and prediction errors, which represent the difference between expected and actual outcomes. This allowed them to pinpoint which aspects of learning were encoded by specific thalamic oscillations.

Datasets

Not discussed in the paper. The study utilized data collected from eight individual patients undergoing DBS surgery.

Key Insights and Ideas

  • Thalamus is a key player in punishment avoidance learning: Contrary to previous research focusing on reward, this study highlights the thalamus's significant involvement in learning to avoid negative outcomes.
  • Low-frequency oscillations are the language of learning in the thalamus: The study found that specific low-frequency oscillations (4-12 Hz) in the thalamus are crucial for encoding learning signals, rather than higher frequencies often associated with other brain functions.
  • Asymmetry in learning signals: While both reward and punishment expectations are encoded, punishment expectations (Qp) are more strongly represented by thalamic LFPs, particularly early after cue presentation. This aligns with the behavioral observation that punishment avoidance learning is often slower.
  • Prediction errors are encoded: Thalamic LFPs also encode prediction errors, which are essential for updating learning models. Specifically, they seem to reflect cortical inputs related to reward prediction errors.
  • Neural basis for behavioral inhibition: The findings suggest that the thalamus might play a role in the behavioral inhibition observed during punishment avoidance, potentially by modulating motor action thresholds.

Key Results

  • Punishment expectation encoding: Low-frequency oscillations (4-12 Hz) in the thalamus were significantly correlated with punishment expectations (Qp) early after cue onset (0.36-1.14s window). This association was stronger for punishment than for reward expectations.
  • Prediction error encoding: At the time of outcome display, thalamic LFPs positively encoded reward prediction errors (Qr/R) and negatively encoded outcome values.
  • Behavioral asymmetry mirrored: The stronger encoding of punishment expectations in the thalamus aligns with the observed behavioral asymmetry where subjects are slower to respond in punishment avoidance tasks.
  • No significant dissociation for prediction errors: While reward prediction errors were clearly encoded, the evidence for punishment prediction error encoding in the thalamus was weaker and less consistent.

Limitations and Future Work

The authors acknowledge that the study was conducted on a small sample of patients with epilepsy, and the presence of epilepsy and DBS electrodes might influence the findings. Future work could involve investigating these processes in healthy individuals using non-invasive methods or exploring the precise neural circuits through which the thalamus interacts with other brain regions during reinforcement learning. The study also suggests that understanding the thalamus's role could be crucial for characterizing cognitive deficits in neuropsychiatric disorders.

Summary updated

Abstract

Abstract Although the thalamus is supposed to be involved in reinforcement-based decision-making, there is no direct evidence regarding the involvement of this subcortical structure in humans. To fill this gap, we leveraged rare intra-thalamic electrophysiological recordings in patients and found that temporally structured thalamic oscillations encode key learning signals. Our findings also provide neural insight into the computational mechanisms of action inhibition in punishment avoidance learning.
BibTeX
@misc{collombclerc2022role,
  author = {Antoine Collomb‐Clerc and Maëlle C. M. Gueguen and Lorella Minotti and Philippe Kahane and Vincent Navarro and Fabrice Bartoloméi and Romain Carron and Regis Jean and Stéphan Chabardès and Stefano Palminteri and Julien Bastin},
  title = {{The role of the thalamus in human reinforcement learning}},
  year = {2022},
  doi = {10.1101/2022.11.23.517731},
  howpublished = {bioRxiv},
}
Figure