Robust Pavlovian-to-Instrumental and Pavlovian-to-Metacognitive Transfers in human reinforcement learning

Chih-Chung Ting, Stefano Palminteri, Jan B. Engelmann, Maël LebretonbioRxiv 2019 · DOI

TL;DR

This study investigates how the emotional valence (gain vs. loss) of outcomes affects human decision-making speed and confidence. They found that while both are generally affected by valence, confidence biases can persist even when decision speed differences are eliminated, suggesting these two effects are partially dissociable. This has implications for understanding how emotions influence our judgments and learning.

What Problem Does This Paper Address?

In reinforcement learning, humans are generally good at both seeking rewards and avoiding punishments. However, research has shown that the context of these outcomes (whether it's a gain or a loss) can influence behavior in subtle ways. Specifically, two phenomena have been observed:

  1. Pavlovian-to-Instrumental Transfer (PIT): Decisions in loss-inducing contexts tend to be slower. This is thought to be because the negative emotional association with losses, learned through Pavlovian conditioning, transfers to the instrumental action of making a choice, slowing down motor execution.
  2. Pavlovian-to-Metacognitive Transfer (PMT): Loss contexts also lead to decreased confidence in one's choices. This is a metacognitive effect, meaning it relates to our awareness and judgment of our own cognitive processes.

The key question this paper addresses is whether these two effects (slower responses and lower confidence in loss contexts) are driven by the same underlying mechanism or if they can be separated. Specifically, can we observe the confidence bias (PMT) even when the response time bias (PIT) is experimentally disrupted? Understanding this dissociation is crucial for building more accurate models of decision-making and how emotions influence our judgments.

Methods

The researchers designed six experiments to disentangle PIT and PMT. The core of their approach involved a probabilistic instrumental learning task where participants learned to choose symbols associated with gains or losses. They then employed several strategies to disrupt the PIT effect (slowing of response times in loss contexts):

  • Manipulating Option-Action Mapping: They altered how participants' button presses mapped to symbol choices. This was done to decouple the decision-making process from the motor execution of the response, aiming to break the link that might cause response slowing.
  • Imposing Time Pressure: In one experiment, participants were given a strict time limit to make their choices, forcing them to respond quickly regardless of the valence context.

They measured choice accuracy, confidence ratings (on a scale of 50-100% probability of being correct), and response times (RTs). Statistical analyses included repeated-measures ANOVAs and generalized linear mixed-effects models to assess the effects of valence, information, and their interactions on these behavioral measures. They also used inter-individual regression analyses to examine the relationship between individual differences in RT and confidence biases.

Datasets

The study utilized data from six experiments. Experiments 1 and 2 were based on previously published data, while experiments 3-6 were newly conducted for this study. The total number of participants across all experiments was 108. No new benchmarks were introduced; the study focused on analyzing existing behavioral paradigms.

Key Insights and Ideas

  • Robustness of Valence Effects: Both PIT (slower RTs in loss contexts) and PMT (lower confidence in loss contexts) are remarkably robust phenomena, persisting even when experimental manipulations attempt to disrupt them.
  • Partial Dissociability: The most significant insight is that PMT can be observed even when PIT is successfully disrupted. This suggests that while these effects often co-occur, they are not entirely dependent on each other.
  • Confidence as More Than Just RT: The findings challenge the idea that confidence judgments are simply a byproduct of response times. The fact that confidence is affected by valence even when RTs are not strongly differentiated implies that other mechanisms are at play in generating subjective confidence.
  • Implications for Learning Models: The results highlight the need for more sophisticated computational models of reinforcement learning that can account for both motor and metacognitive effects of emotional valence, and their potential dissociation.

Key Results

  • PIT Resilience: Despite attempts to disrupt it, the slowing of response times in loss contexts (PIT) was observed in most experiments, though its effect size was sometimes attenuated.
  • PMT Persistence: Participants consistently showed lower confidence in loss contexts compared to gain contexts across all experiments, with similar effect sizes.
  • Dissociation in Experiment 5: In one experiment (Experiment 5), the researchers were able to significantly reduce or eliminate the valence effect on RTs. Crucially, they still observed a significant valence effect on confidence in this experiment, providing strong evidence for the partial dissociability of PMT and PIT.
  • Inter-individual Correlations: Analyses revealed that individuals who showed stronger PIT effects also tended to show stronger PMT effects, indicating a link between the two, but the intercept of these regressions was significantly positive, meaning even individuals with no RT difference still showed a confidence difference.

Limitations and Future Work

  • Computational Modeling: The authors note that their current data, due to the experimental manipulations, cannot be directly used with combined reinforcement-learning and decision-diffusion modeling approaches to fully test their hypotheses about the underlying mechanisms. Further experiments would be needed for this.
  • Decision Time vs. Response Time: While they attempted to dissociate decision from motor execution, they acknowledge that future studies could investigate whether participants track an internal "decision time" that might influence confidence, beyond the measured response time.
  • Generalizability: While the study used multiple experiments and a decent sample size, further research could explore these effects in different types of tasks and populations.

Summary updated

Abstract

Abstract In simple instrumental-learning tasks, humans learn to seek gains and to avoid losses equally well. Yet, two effects of valence are observed. First, decisions in loss-contexts are slower, which is consistent with the Pavlovian-instrumental transfer (PIT) hypothesis. Second, loss contexts decrease individuals’ confidence in their choices – a bias akin to a Pavlovian-to-metacognitive transfer (PMT). Whether these two effects are two manifestations of a single mechanism or whether they can be partially dissociated is unknown. Here, across six experiments, we attempted to disrupt the PIT effects by manipulating the mapping between decisions and actions and imposing constraints on response times (RTs). Our goal was to assess the presence of the metacognitive bias in the absence of the RT bias. Were observed both PIT and PMT despite our disruption attempts, establishing that the effects of valence on motor and metacognitive responses are very robust and replicable. Nonetheless, within- and between-individual inferences reveal that the confidence bias resists the disruption of the RT bias. Therefore, although concomitant in most cases, PMT and PIT seem to be – partly – dissociable. These results highlight new important mechanistic constraints that should be incorporated in learning models to jointly explain choice, reaction times and confidence.
BibTeX
@misc{ting2019robust,
  author = {Chih-Chung Ting and Stefano Palminteri and Jan B. Engelmann and Maël Lebreton},
  title = {{Robust Pavlovian-to-Instrumental and Pavlovian-to-Metacognitive Transfers in human reinforcement learning}},
  year = {2019},
  doi = {10.1101/593368},
  howpublished = {bioRxiv},
}

Loading related content...

Figure