Striatal response to negative feedback in a stop signal task operates as a multi-value learning signal

Benjamin J. Smith, M. B. Lipsett, Danielle Cosme, Victoria A. Braun, Anastasia M. Browning O’Hagan, Elliot T. BerkmanImaging 2023 · DOI

TL;DR

This study reveals that the striatum, a brain region crucial for learning, acts as a "multi-value learning signal" during a stop-signal task. It not only tracks changes in the probability of needing to stop but also learns to predict the timing of those stop signals, demonstrating a sophisticated form of reinforcement learning in inhibitory control.

What Problem Does This Paper Address?

The ability to inhibit responses is fundamental for goal-directed behavior, and understanding how this inhibitory control is learned and regulated is a key area of neuroscience. The stop-signal task (SST) is a common tool for studying this, but it involves learning multiple cues simultaneously: the probability of a stop signal occurring (P(Stop)) and the delay before the stop signal (Stop Signal Delay, SSD). Previous research has shown that the striatum is involved in learning from errors in the SST, but it was unclear if it could simultaneously track and learn from these two distinct, dynamic parameters. This paper aims to characterize the neural signals in the striatum that reflect this complex, multi-faceted learning process.

Methods

The researchers used functional magnetic resonance imaging (fMRI) to examine brain activity in 217 participants performing a stop-signal task. They employed a mixed-effects model to analyze the relationship between behavioral learning signals and striatal activity. Specifically, they looked at:

  • Post-error slowing: The tendency to respond slower after making an error, which is thought to reflect an adaptive learning process.
  • Reward Prediction Error (RPE): A signal that indicates the difference between expected and actual outcomes, crucial for reinforcement learning.
  • Multi-value learning: The ability to learn from multiple predictive cues simultaneously.

They analyzed brain activity in different time windows relative to the stop signal, including during the trial, the inter-trial interval (ITI), and the subsequent trial. They also used a novel "trial-average time course" analysis that avoided assumptions of a standard hemodynamic response function (HRF) to get a more direct view of BOLD signal changes.

Datasets

The study utilized data from 217 participants who completed a stop-signal task. This data was collected as part of a larger intervention study on healthy eating. The task itself was calibrated to ensure participants made errors on approximately 50% of stop trials, a standard practice for maintaining task difficulty and eliciting learning signals.

Key Insights and Ideas

  • Striatum as a Multi-Value Learner: The striatum doesn't just learn one thing at a time; it simultaneously tracks and learns from both the probability of a stop signal (P(Stop)) and the timing of that signal (SSD). This is a more sophisticated form of learning than previously demonstrated in this context.
  • Distinct Neural Signals: The study identified two distinct neural signals in the striatum following a stop signal:
  • A salience signal (1-3 seconds post-stop) that appears to be a general response to the stop cue, regardless of success or failure.
  • A valenced RPE signal (4-10 seconds post-stop) that differentiates between successful and failed stops, reflecting a prediction error related to the outcome.
  • Post-Error Slowing and RPE: The observed deactivation in the striatum following failed stops, and activation following successful stops, aligns with the concept of RPE, suggesting that error processing in the SST is indeed a form of reinforcement learning.
  • Dynamic Learning: The striatum's activity evolves over time, showing an initial salience response and then a more nuanced RPE signal that reflects learned expectations about the stop signal.

Key Results

  • Striatal Activity Correlates with Learning: Mixed-effects models showed that striatal activity in the 4-10 second window post-stop signal was significantly predicted by both changes in reaction time (representing SSD learning) and changes in the probability of a stop trial (P(Stop) learning).
  • Differential Responses:
    • Following Correct Stop trials, there was an increase in striatal activity.
  • Following Failed Stop trials, there was a decrease in striatal activity.
  • Specific Regions Involved: The RPE signal was most prominent in the striatal regions (putamen, accumbens, and caudate), but not in cortical regions like the ACC or FOC, suggesting a specialized role for the striatum in this type of learning.
  • Failed Stop Spike: A distinct, early (1-3 second) spike in activity was observed specifically after failed stop trials, which was not correlated with reaction time changes, suggesting it's more of a salience or surprise signal rather than an RPE.

Limitations and Future Work

  • HRF Assumptions: While the study used a novel time-course analysis to mitigate HRF assumptions, the interpretation of BOLD signals is still inherently indirect.
  • Correlation vs. Causation: The study demonstrates strong correlations between striatal activity and learning parameters, but direct causal links would require further investigation.
  • Generalizability: While the study involved a large sample size, future work could explore these findings in different populations or with different types of inhibitory control tasks.
  • Applied Contexts: The authors suggest that a better understanding of these neural-computational mechanisms could inform interventions in applied habit contexts, such as reducing unhealthy food consumption, by optimizing inhibitory training strategies.

Summary updated

Abstract

A mixed-effects model indicates the striatal activity might be a learning signal that encodes reaction time change and the current expected probability of a stop trial occuring. This extends existing evidence that the striatum encodes a reward prediction error signal for learning within the stop signal task, and demonstrates for the first time that this signal seems to encode both change in stop signal probability and in stop signal delay.
BibTeX
@inproceedings{smith2023striatal,
  author = {Benjamin J. Smith and M. B. Lipsett and Danielle Cosme and Victoria A. Braun and Anastasia M. Browning O’Hagan and Elliot T. Berkman},
  title = {{Striatal response to negative feedback in a stop signal task operates as a multi-value learning signal}},
  booktitle = {Imaging},
  year = {2023},
  doi = {10.1162/imag\_a\_00024},
}
Figure