Human thalamic low-frequency oscillations correlate with expected value and outcomes during reinforcement learning

Antoine Collomb‐Clerc, Maëlle C. M. Gueguen, Lorella Minotti, Philippe Kahane, Vincent Navarro, Fabrice Bartoloméi, Romain Carron, Jean Régis, Stéphan Chabardès, Stefano Palminteri, Julien BastinNature Communications 2023 · DOI

TL;DR

This study provides the first direct evidence that the human thalamus, specifically its anterior (ATN) and dorsomedial (DMTN) nuclei, plays a crucial role in reinforcement learning. They found that low-frequency oscillations (LFOs) in these thalamic regions encode expected values and prediction errors, both for rewards and punishments, shedding light on how we learn to make decisions in complex environments.

What Problem Does This Paper Address?

Reinforcement learning (RL) is fundamental to how we learn from experience, guiding us to maximize rewards and minimize punishments. While it's well-established that RL involves a network of brain regions including the prefrontal cortex and striatum, the specific role of the thalamus, a key relay station in this network, has remained unclear in humans. Previous research, primarily from animal studies, hinted at its involvement, particularly in punishment avoidance, but direct electrophysiological evidence in humans was lacking. This paper aims to fill that gap by investigating whether thalamic neural activity, specifically low-frequency oscillations, reflects the computational signals associated with reinforcement learning.

Methods

The researchers utilized a sophisticated approach combining intra-thalamic electrophysiological recordings from eight patients with pharmacoresistant epilepsy (who had electrodes implanted for deep brain stimulation) with computational modeling of their behavior.

  • Electrophysiology: Local Field Potentials (LFPs) were recorded directly from the anterior thalamic nucleus (ATN) and dorsomedial thalamic nucleus (DMTN) using implanted electrodes. Time-frequency analysis was then applied to these signals to examine oscillatory activity across different frequency bands.
  • Computational Modeling: A Q-learning model was used to analyze the participants' behavioral data. This model estimates trial-wise "expected values" (how good a choice is predicted to be) and "prediction errors" (the difference between expected and actual outcomes). These estimated values served as computational regressors for the electrophysiological data.
  • Statistical Analysis: Regression analyses were performed to correlate the power of thalamic LFOs (4-12 Hz) with the estimated expected values and prediction errors at different stages of the learning task (after cue presentation and after outcome delivery).

Datasets

Not discussed in the paper. The study used data collected from eight human participants performing a reinforcement learning task.

Key Insights and Ideas

  • Thalamus as a RL Hub: The study firmly establishes the human thalamus (ATN/DMTN) as an active participant in reinforcement learning, not just a passive relay.
  • LFOs as the Neural Code: Low-frequency oscillations (4-12 Hz) in the thalamus appear to be the primary neural signal encoding key RL variables like expected value and prediction errors. This is a significant finding, as other frequency bands were less consistently associated with these signals.
  • Dual Encoding of Prediction Errors: The thalamus seems to encode both components of the "teaching signal" in RL: the difference between what was expected and what actually happened. This is crucial for updating future behavior.
  • Punishment Sensitivity: Interestingly, thalamic LFOs showed a stronger correlation with punishment expectations compared to reward expectations, particularly before a choice was made. This might explain why we are often slower to learn from or respond to potential negative outcomes.
  • Action Inhibition Link: The observed differences in how rewarding and punishing conditions were represented in the thalamus suggest a neural mechanism underlying action inhibition, which is critical for avoiding negative consequences.

Key Results

  • Expected Value Encoding: Thalamic LFOs positively correlated with expected value (both for rewards and punishments) after the cue was presented and before the choice was made.
  • Prediction Error Encoding: Thalamic LFOs were negatively correlated with outcomes, meaning they reflected the difference between expected and actual outcomes, a core component of prediction errors. This was observed for both reward and punishment prediction errors.
  • Behavioral Asymmetry: Participants exhibited slower reaction times in punishment avoidance tasks compared to reward-seeking tasks, consistent with previous findings and suggesting a role for motor inhibition in punishment learning.
  • Frequency Band Specificity: LFOs (4-12 Hz) were the most consistently and significantly associated with prediction errors, outperforming other frequency bands like delta, beta, and gamma.

Limitations and Future Work

  • Patient Population: The study was conducted on a small sample of patients with pharmacoresistant epilepsy. While their implanted electrodes provided a unique opportunity for direct recordings, the findings might not generalize perfectly to a healthy population. The authors acknowledge that the epilepsy itself or the deep brain stimulation treatment could potentially influence neural activity.
  • ATN vs. DMTN Dissociation: The study did not find significant functional differences between the anterior thalamic nucleus (ATN) and the dorsomedial thalamic nucleus (DMTN) in their encoding of learning signals. The authors suggest this might be due to the ATN and DMTN being recruited together for flexible decision-making, or potentially a lack of statistical power to detect subtle differences. Future work could explore this further with larger sample sizes or more targeted analyses.
  • Cortical vs. Thalamic Dynamics: While thalamic LFOs showed similar prediction error encoding to cortical regions, there was an initial positive association in cortical areas that was absent in the thalamus. This suggests a potential difference in the temporal dynamics or processing pathways between these regions.
  • Future Directions: The authors suggest that understanding the thalamus's role in reinforcement learning is crucial for characterizing cognitive deficits in neuropsychiatric disorders. Further research could explore how these thalamic signals are modulated in conditions like addiction or anxiety. Investigating the precise connectivity and interactions between the thalamus and other RL-related brain regions (like the prefrontal cortex and striatum) is also a promising avenue.

Summary updated

Abstract

Reinforcement-based adaptive decision-making is believed to recruit fronto-striatal circuits. A critical node of the fronto-striatal circuit is the thalamus. However, direct evidence of its involvement in human reinforcement learning is lacking. We address this gap by analyzing intra-thalamic electrophysiological recordings from eight participants while they performed a reinforcement learning task. We found that in both the anterior thalamus (ATN) and dorsomedial thalamus (DMTN), low frequency oscillations (LFO, 4-12 Hz) correlated positively with expected value estimated from computational modeling during reward-based learning (after outcome delivery) or punishment-based learning (during the choice process). Furthermore, LFO recorded from ATN/DMTN were also negatively correlated with outcomes so that both components of reward prediction errors were signaled in the human thalamus. The observed differences in the prediction signals between rewarding and punishing conditions shed light on the neural mechanisms underlying action inhibition in punishment avoidance learning. Our results provide insight into the role of thalamus in reinforcement-based decision-making in humans.
BibTeX
@inproceedings{collombclerc2023human,
  author = {Antoine Collomb‐Clerc and Maëlle C. M. Gueguen and Lorella Minotti and Philippe Kahane and Vincent Navarro and Fabrice Bartoloméi and Romain Carron and Jean Régis and Stéphan Chabardès and Stefano Palminteri and Julien Bastin},
  title = {{Human thalamic low-frequency oscillations correlate with expected value and outcomes during reinforcement learning}},
  booktitle = {Nature Communications},
  year = {2023},
  doi = {10.1038/s41467-023-42380-6},
}
Figure