Valence biases factual and counterfactual learning in opposite directions

Stefano Palminteri, Germain Lefebvre, Emma J. Kilford, Sarah‐Jayne BlakemorebioRxiv 2016 · DOI

TL;DR

This paper reveals that our brains learn differently from what actually happened versus what could have happened. While we tend to focus on positive outcomes when learning from our own choices (factual learning), we actually pay more attention to negative outcomes when considering what we missed out on (counterfactual learning). This suggests a "confirmation bias" where we prioritize information that validates our past decisions.

What Problem Does This Paper Address?

We constantly make decisions and learn from their consequences. This learning process is crucial for adapting our behavior. Previous research has shown that when learning from the outcomes of our chosen actions (factual learning), people exhibit a "valence bias," meaning they learn more from positive outcomes than negative ones. This is often framed as an "optimistic bias." However, a significant question remained: does this valence bias also apply to learning from forgone outcomes (counterfactual learning) – what we could have experienced if we had made a different choice? Understanding this is key because much of our decision-making involves considering alternatives.

Methods

The researchers designed two experiments using a reinforcement learning task. Participants had to choose between two options, and the outcomes were either winning or losing points.

  • Experiment 1: Participants only saw the outcome of their chosen option, allowing for the study of factual learning.
  • Experiment 2: Participants saw the outcomes of both their chosen and unchosen options, enabling the study of both factual and counterfactual learning.

They employed a computational model, a modified Rescorla-Wagner model, to analyze the learning rates. This model allowed them to estimate separate learning rates for positive and negative prediction errors, and for both factual (chosen) and counterfactual (unchosen) outcomes. Model comparison was used to determine the most parsimonious explanation for the observed learning patterns.

Datasets

Not discussed in the paper. The study involved two cohorts of human participants.

Key Insights and Ideas

  • Opposing Biases: The most striking finding is that factual and counterfactual learning are biased in opposite directions. This challenges the idea of a single, overarching "positive valence bias."
  • Confirmation Bias: The results strongly support a "confirmation bias" hypothesis. This means we preferentially learn from outcomes that confirm our previous choices, regardless of whether they were factual or counterfactual. Positive outcomes from chosen options and negative outcomes from unchosen options are weighted more heavily.
  • Adaptive or Maladaptive? The study suggests these biases might be maladaptive, particularly in situations requiring rapid adaptation to changing reward contingencies (like in the "Reversal" condition). Participants with stronger biases performed worse in these scenarios.
  • Unified Learning System: The model comparison results suggest that factual and counterfactual learning might be processed by the same underlying learning system, rather than separate ones.

Key Results

  • Factual Learning: Replicated previous findings, showing a "positive valence bias" where participants learned more from positive prediction errors than negative ones (e.g., learning more from winning a point than losing one when it was the chosen outcome).
  • Counterfactual Learning: Revealed an opposite bias. Participants learned more from negative prediction errors than positive ones when considering forgone outcomes (e.g., paying more attention to the fact that they could have won a point but didn't, compared to the fact that they could have lost a point and didn't).
  • Model Comparison: The "Confirmation" model, which accounts for biases based on whether an outcome confirms or contradicts a choice, provided a better fit to the data than models focusing solely on valence or outcome type.
  • Behavioral Impact: Participants with higher learning biases showed a tendency to stick with a preferred choice even when there was no objectively "correct" option and performed worse in conditions requiring adaptation to changing reward probabilities.

Limitations and Future Work

  • Maladaptive Nature: While the biases appear maladaptive in certain contexts, the authors speculate they might have an overall adaptive value by contributing to self-esteem and confidence (e.g., "self-serving" biases). Further research could explore this trade-off.
  • Neurobiological Basis: The study provides behavioral and computational evidence. Future work could investigate the specific neural mechanisms underlying these opposing biases in factual and counterfactual learning.
  • Generalizability: The study used specific task conditions. Exploring these biases in more complex, real-world decision-making scenarios would be valuable.
  • Explicit vs. Implicit Confirmation: The authors note that their definition of confirmation bias extends to implicitly reinforcing current choices, which differs from previous work focusing on explicit advice or instructions. Further research could delineate these different forms of confirmation bias.

Summary updated

Abstract

Abstract Previous studies suggest that factual learning, that is, learning from obtained outcomes, is biased, such that participants preferentially take into account positive, as compared to negative, prediction errors. However, whether or not the prediction error valence also affects counterfactual learning, that is, learning from forgone outcomes, is unknown. To address this question, we analysed the performance of two cohorts of participants on reinforcement learning tasks using a computational model that was adapted to test if prediction error valance influences learning. Concerning factual learning, we replicated previous findings of a valence-induced bias, whereby participants learned preferentially from positive, relative to negative, prediction errors. In contrast, for counterfactual learning, we found the opposite valence-induced bias: negative prediction errors were preferentially taken into account relative to positive ones. When considering valence-induced bias in the context of both factual and counterfactual learning, it appears that people tend to preferentially take into account information that confirms their current choice
BibTeX
@misc{palminteri2016valence,
  author = {Stefano Palminteri and Germain Lefebvre and Emma J. Kilford and Sarah‐Jayne Blakemore},
  title = {{Valence biases factual and counterfactual learning in opposite directions}},
  year = {2016},
  doi = {10.1101/090654},
  howpublished = {bioRxiv},
}
Figure