Valence-dependent influence of serotonin depletion on model-based choice strategy

Yulia Worbe, Stefano Palminteri, George Savulich, Nathaniel D. Daw, Emilio Fernández-Egea, Trevor W. Robbins, Valerie VoonMolecular 2015 · DOI

TL;DR

This study reveals that serotonin, a key neurotransmitter, plays a crucial role in how we balance goal-directed and habitual decision-making. By temporarily reducing serotonin levels, researchers found that it impairs our ability to plan ahead for rewards but surprisingly enhances it when avoiding losses, suggesting a valence-dependent effect of serotonin on our cognitive strategies.

What Problem Does This Paper Address?

Human decision-making is a complex interplay between two systems: a "reflective" or goal-directed system that plans based on future outcomes, and a "reflexive" or habitual system that relies on past experiences. Computationally, these are often modeled as "model-based" (goal-directed) and "model-free" (habitual) learning strategies. While dopamine's role in these systems is well-studied, the influence of serotonin, another critical neuromodulator, on this balance has been less clear. This paper aims to fill that gap by investigating how manipulating serotonin levels affects an individual's reliance on model-based versus model-free strategies, particularly in the context of both reward and punishment. Understanding this could shed light on disorders characterized by impaired decision-making, like OCD and addiction.

Methods

The researchers employed a dietary tryptophan depletion (TD) method to temporarily reduce serotonin levels in healthy volunteers. Tryptophan is an essential amino acid that serves as a precursor to serotonin. By having participants consume a liquid mixture lacking tryptophan but containing other competing amino acids, the study aimed to selectively and transiently lower central serotonin.

The core of the experimental design was a novel two-stage decision-making task. This task is designed to differentiate between model-based and model-free strategies. In the first stage, participants made a choice between two stimuli, which then probabilistically led to one of two pairs of stimuli in the second stage. Each of these second-stage stimuli was associated with either a monetary reward or a monetary loss (depending on the task version). The key to distinguishing the strategies lies in how participants' choices in the first stage are influenced by the outcomes of the second stage and the transition probabilities between stages.

Specifically, a model-free strategy predicts choices based on past rewards or punishments, while a model-based strategy integrates information about the transition probabilities and outcomes to predict future consequences. The researchers analyzed participants' "stay" and "shift" probabilities in the first stage, looking for specific interaction patterns between outcome and transition probability that are signatures of model-based learning. They also used a computational modeling approach to fit participants' choices to a hybrid model that estimates parameters for both model-free and model-based learning, including a "weighting factor" (w) that reflects the relative engagement of each strategy.

Datasets

Not discussed in the paper. The study involved experimental data collected from human participants performing a task.

Key Insights and Ideas

  • Serotonin's Dual Role: The most striking finding is that serotonin's influence on goal-directed behavior is not uniform; it depends on the affective valence of the outcome. This challenges a simple, overarching role for serotonin in cognitive control.
  • Impaired Reward Planning, Enhanced Loss Avoidance: When serotonin levels were reduced, participants showed a deficit in model-based planning for rewards. Conversely, they became more goal-directed when trying to avoid losses, suggesting serotonin might normally suppress goal-directedness in aversive situations.
  • Potential Mechanism: Average Reward Representation: The authors hypothesize that serotonin might modulate the representation of average reward rate. This could explain the valence-dependent effects: in reward scenarios, a higher average reward signal might favor model-free responses, while in punishment scenarios, the "opportunity cost" of time spent deliberating might invert, favoring model-based strategies when serotonin is low.
  • Clinical Relevance: The findings have significant implications for understanding psychiatric disorders characterized by impaired goal-directed control, such as obsessive-compulsive disorders and addictions, where decision-making processes are disrupted.

Key Results

  • Factorial Analysis:
    • In the reward condition, tryptophan depletion (TD) led to a shift towards a model-free strategy, indicated by a significant main effect of outcome but a lack of the crucial outcome × transition probability interaction that signifies model-based learning.
  • In the punishment condition, TD promoted a model-based strategy, evidenced by a significant outcome × transition probability interaction in the TD group, suggesting enhanced goal-directed behavior in avoiding losses.
  • Computational Modeling:
    • In the reward condition, the TD group showed a significantly lower weighting factor (w) compared to the control (BAL) group, indicating a reduced engagement of the model-based strategy.
  • In the punishment condition, no significant differences were found between the groups in the computational model parameters, which the authors note as a point for further investigation, suggesting the factorial analysis might be more sensitive to detecting these effects.
  • Biochemical Confirmation: Post-procedure analysis confirmed that TD robustly decreased the TRP:ELNAAs ratio, a marker of central serotonergic function, in the TD group compared to the BAL group.

Limitations and Future Work

  • Specificity of TD: While TD is believed to selectively affect serotonin, the authors acknowledge that it's difficult to definitively rule out broader effects on other neurotransmitter systems or general cognitive functions. Future work could involve using more specific pharmacological agents to target serotonin receptors.
  • Computational Model Nuances: The authors note that the computational model fitting in the punishment condition did not fully corroborate the factorial analysis findings. This suggests potential limitations in the model or the need for further refinement to capture the nuances of behavior in aversive contexts.
  • Habit Formation: The study focused on the balance between model-based and model-free learning. However, the authors suggest that the two-stage task might not fully capture "habit expression." Further studies involving over-training and extinction testing could provide a clearer picture of serotonin's role in habitual responses.
  • Interaction with Dopamine: The paper touches upon the potential interaction between serotonin and dopamine systems, especially given that dopamine administration has shown opposite effects on model-based learning. Future research could explore these complex interactions more directly.
  • Clinical Translation: While the findings are clinically relevant, further studies are needed to clarify how enhanced serotonin neurotransmission (e.g., via SSRIs) might influence these decision-making strategies, particularly in clinical populations.

Summary updated

Abstract

Human decision-making arises from both reflective and reflexive mechanisms, which underpin goal-directed and habitual behavioural control. Computationally, these two systems of behavioural control have been described by different learning algorithms, model-based and model-free learning, respectively. Here, we investigated the effect of diminished serotonin (5-hydroxytryptamine) neurotransmission using dietary tryptophan depletion (TD) in healthy volunteers on the performance of a two-stage decision-making task, which allows discrimination between model-free and model-based behavioural strategies. A novel version of the task was used, which not only examined choice balance for monetary reward but also for punishment (monetary loss). TD impaired goal-directed (model-based) behaviour in the reward condition, but promoted it under punishment. This effect on appetitive and aversive goal-directed behaviour is likely mediated by alteration of the average reward representation produced by TD, which is consistent with previous studies. Overall, the major implication of this study is that serotonin differentially affects goal-directed learning as a function of affective valence. These findings are relevant for a further understanding of psychiatric disorders associated with breakdown of goal-directed behavioural control such as obsessive-compulsive disorders or addictions.
BibTeX
@inproceedings{worbe2015valence,
  author = {Yulia Worbe and Stefano Palminteri and George Savulich and Nathaniel D. Daw and Emilio Fernández-Egea and Trevor W. Robbins and Valerie Voon},
  title = {{Valence-dependent influence of serotonin depletion on model-based choice strategy}},
  booktitle = {Molecular},
  year = {2015},
  doi = {10.1038/mp.2015.46},
}
Figure