Neural and computational underpinnings of biased confidence in human reinforcement learning

Chih-Chung Ting, Nahuel Salem-Garcia, Stefano Palminteri, Jan B. Engelmann, Maël LebretonNature Communications 2023 · DOI

TL;DR

This study reveals a fascinating split in how our brains process confidence during decision-making. Using fMRI and a reinforcement learning task, researchers found that the ventromedial prefrontal cortex (VMPFC) tracks overall confidence, including a bias towards gains, while dorsal prefrontal regions track context-specific confidence. This suggests the VMPFC is crucial for our subjective feeling of certainty, even when it's influenced by biases.

What Problem Does This Paper Address?

We constantly make decisions in uncertain environments and, crucially, we have a sense of how confident we are in those decisions. This "metacognitive" ability is vital for learning, adapting our strategies, and even knowing when to change our minds. While previous research has linked confidence judgments to activity in two distinct prefrontal networks – a positive one (VMPFC) and a negative one (dorsolateral and dorsomedial prefrontal cortex) – their precise roles and how they interact remained unclear. This paper aims to disentangle these roles by investigating how different aspects of confidence, like objective uncertainty and subjective biases, are represented in the brain during reinforcement learning. Specifically, they leverage the known phenomenon that people are more confident when seeking rewards (gains) than when trying to avoid losses, even if the objective difficulty of the task is the same.

Methods

The researchers employed a combination of behavioral experiments, computational modeling, and fMRI.

  • Reinforcement Learning Task: Participants performed a task where they learned to choose between pairs of symbols, each probabilistically associated with monetary gains or losses. The experiment manipulated two factors: outcome valence (gain vs. loss) and information feedback (partial vs. complete). After each choice, participants reported their confidence.
  • fMRI: While participants performed the task, their brain activity was measured using fMRI. This allowed the researchers to correlate neural activity with different aspects of the task and confidence judgments.
  • Computational Modeling: They used reinforcement learning models to estimate latent variables like chosen option values and context values. These models helped to quantify aspects of decision-making and confidence.
  • Statistical Analysis: Various statistical methods, including ANOVAs, t-tests, and Bayesian model selection, were used to analyze behavioral data and fMRI signals. They also employed generalized linear models (GLMs) to investigate the relationship between neural activity and different confidence-related variables.

Datasets

The study used data collected from 40 human participants performing the described reinforcement learning task in an MRI scanner. No new benchmarks were introduced; rather, existing behavioral data from previous studies were reanalyzed and integrated.

Key Insights and Ideas

  • Functional Dissociation: The core finding is a clear division of labor between prefrontal networks in processing confidence. The VMPFC appears to represent a "task-wide" confidence signal that incorporates biases (like the gain bias), while dorsal prefrontal regions track "condition-specific" confidence, reflecting uncertainty within a particular context.
  • VMPFC and Biased Confidence: The VMPFC's positive correlation with confidence, especially its sensitivity to the gain bias, suggests it's a key player in generating our subjective feeling of certainty, even when that feeling is influenced by affective factors.
  • Challenging Dominant Models: The results challenge the prevailing view that the VMPFC primarily encodes option values in reinforcement learning. Instead, this study suggests VMPFC activity is better explained by confidence judgments, even when value-related variables are also considered.
  • Confidence as a "Global Feeling": The VMPFC seems to integrate various decision variables and contextual biases to form a global, subjective sense of confidence, akin to our overall feeling of how right we are.

Key Results

  • Behavioral: Participants showed a significant "valence-induced confidence bias," reporting higher confidence when seeking gains compared to avoiding losses, even with similar accuracy. This bias was somewhat mitigated by complete information feedback.
  • fMRI:
    • A positive correlation between confidence and VMPFC activity was observed across all conditions.
  • A negative correlation between confidence and activity in dorsolateral and dorsomedial prefrontal cortices was also found.
  • VMPFC activity was better explained by confidence judgments than by option values derived from reinforcement learning models.
  • Dorsal prefrontal network activity correlated with condition-specific confidence, not task-wide confidence.

Limitations and Future Work

  • Null Hypothesis Testing: The study acknowledges that demonstrating the absence of an effect in the negative network (i.e., no valence or information effect on condition-specific confidence) relies on accepting the null hypothesis, which has inherent limitations.
  • Information Effect on Confidence: Despite clear effects of information on accuracy, the study did not find a significant brain region sensitive to both confidence and the information manipulation. The authors suggest this might be due to the smaller effect size of information on confidence or participants' ability to infer counterfactual outcomes.
  • Mechanistic Interactions: The precise mechanisms by which task-wide and condition-specific confidence signals interact remain to be elucidated.
  • Future Directions: The authors suggest future research should explore the conditions under which the polarity of confidence signals in the VMPFC might change (e.g., from exploration to exploitation) and how different confidence variables influence future behavior and decision-making. They also highlight the importance of considering confidence biases in clinical applications.

Summary updated

Abstract

While navigating a fundamentally uncertain world, humans and animals constantly evaluate the probability of their decisions, actions or statements being correct. When explicitly elicited, these confidence estimates typically correlates positively with neural activity in a ventromedial-prefrontal (VMPFC) network and negatively in a dorsolateral and dorsomedial prefrontal network. Here, combining fMRI with a reinforcement-learning paradigm, we leverage the fact that humans are more confident in their choices when seeking gains than avoiding losses to reveal a functional dissociation: whereas the dorsal prefrontal network correlates negatively with a condition-specific confidence signal, the VMPFC network positively encodes task-wide confidence signal incorporating the valence-induced bias. Challenging dominant neuro-computational models, we found that decision-related VMPFC activity better correlates with confidence than with option-values inferred from reinforcement-learning models. Altogether, these results identify the VMPFC as a key node in the neuro-computational architecture that builds global feeling-of-confidence signals from latent decision variables and contextual biases during reinforcement-learning.
BibTeX
@inproceedings{ting2023neural,
  author = {Chih-Chung Ting and Nahuel Salem-Garcia and Stefano Palminteri and Jan B. Engelmann and Maël Lebreton},
  title = {{Neural and computational underpinnings of biased confidence in human reinforcement learning}},
  booktitle = {Nature Communications},
  year = {2023},
  doi = {10.1038/s41467-023-42589-5},
}
Figure