Extraversion differentiates between model-based and model-free strategies in a reinforcement learning task

Anya Skatova, Patricia A. Chan, Nathaniel D. DawFrontiers in Human Neuroscience 2013 · DOI

Abstract

Trait extraversion has long been associated with increased reward sensitivity, likely mediated by the dopaminergic system. However, reinforcement learning (RL) involves two distinct computational strategies: model-free (habitual, error-driven) and model-based (goal-directed, planning). This study investigates whether extraversion selectively predicts performance in one of these systems. Using a two-step decision task, researchers found that while extraversion can lead to overall performance declines due to task disengagement, it specifically correlates with enhanced model-free learning among highly engaged participants. These findings suggest that personality traits reflect specific tuning parameters within neural learning algorithms.

Extraverts are characterized by their sensitivity to rewards, a trait often linked to the brain's dopamine-driven reward prediction error (RPE) signals. Computational neuroscience distinguishes between two primary RL strategies:

  1. Model-free RL: A computationally efficient strategy that reinforces actions based on past rewards, effectively forming habits.
  2. Model-based RL: A more complex strategy that utilizes a mental map of the environment to plan actions based on expected future outcomes.

The central question of this research is whether the extraversion-reward link is a general enhancement of all learning or if it is selective to the model-free, dopamine-dependent system.

Methods: The Two-Step Decision Task

To differentiate these strategies, 97 participants performed a "two-step decision task." In the first stage, participants choose between two options that lead to one of two second-stage "rooms" with specific probabilities (70% "common" vs. 30% "rare"). In the second stage, they choose between two options to receive a monetary reward.

Drifting reward probabilities for the four second-stage options in the two-step task.

The reward probabilities for the four second-stage options drift independently over time (as seen in the figure above), requiring participants to continuously update their values. A model-free learner repeats a first-stage choice if it led to a reward, regardless of the transition type. In contrast, a model-based learner accounts for the task structure; a reward following a rare transition would lead them to switch their first-stage choice to reach the same rewarded room more reliably.

Key Results and Insights

The "Engagement Mask"

Initial analysis of the full sample showed that higher extraversion scores actually predicted worse performance across both learning strategies. The researchers hypothesized that extraverts might be more prone to boredom or disengagement during repetitive laboratory tasks, which could mask their underlying computational strengths.

Selective Facilitation in Engaged Participants

To test this, the authors introduced an "engagement index" based on how well a participant's choices fit any learning model. When focusing on the top 20% of the most engaged participants, the original hypothesis was confirmed: extraversion significantly predicted better model-free learning ($Z = 2.02, p = 0.043$) but showed no relationship with model-based planning.

Discussion: Personality as a Computational Parameter

These results suggest that personality traits like extraversion are not merely descriptive but may reflect the "tuning" of specific neural algorithms. Extraversion appears to act as a proxy for the sensitivity or gain of the model-free, dopamine-mediated system. However, the tendency for extraverts to disengage from repetitive tasks highlights the importance of considering motivational factors when measuring cognitive performance.

Limitations and Future Directions

While the study provides a computational link between personality and RL, it does not directly measure dopamine levels. Future research using pharmacological interventions could confirm the biological mechanism. Additionally, exploring how other traits like neuroticism or impulsivity interact with these learning systems could lead to a more comprehensive "computational phenotyping" of human personality.

Summary updated

Abstract

Prominent computational models describe a neural mechanism for learning from reward prediction errors, and it has been suggested that variations in this mechanism are reflected in personality factors such as trait extraversion. However, although trait extraversion has been linked to improved reward learning, it is not yet known whether this relationship is selective for the particular computational strategy associated with error-driven learning, known as model-free reinforcement learning, vs. another strategy, model-based learning, which the brain is also known to employ. In the present study we test this relationship by examining whether humans' scores on an extraversion scale predict individual differences in the balance between model-based and model-free learning strategies in a sequentially structured decision task designed to distinguish between them. In previous studies with this task, participants have shown a combination of both types of learning, but with substantial individual variation in the balance between them. In the current study, extraversion predicted worse behavior across both sorts of learning. However, the hypothesis that extraverts would be selectively better at model-free reinforcement learning held up among a subset of the more engaged participants, and overall, higher task engagement was associated with a more selective pattern by which extraversion predicted better model-free learning. The findings indicate a relationship between a broad personality orientation and detailed computational learning mechanisms. Results like those in the present study suggest an intriguing and rich relationship between core neuro-computational mechanisms and broader life orientations and outcomes.
BibTeX
@inproceedings{skatova2013extraversion,
  author = {Anya Skatova and Patricia A. Chan and Nathaniel D. Daw},
  title = {{Extraversion differentiates between model-based and model-free strategies in a reinforcement learning task}},
  booktitle = {Frontiers in Human Neuroscience},
  year = {2013},
  doi = {10.3389/fnhum.2013.00525},
}
Figure