Decomposing the effects of context valence and feedback information on speed and accuracy during reinforcement learning: a meta-analytical approach using diffusion decision modeling

Laura Fontanesi, Stefano Palminteri, Maël Lebreton

TL;DR

This paper uses a novel combination of reinforcement learning (RL) and diffusion decision models (DDM) to analyze how humans learn to seek rewards versus avoid punishments. By jointly modeling choice accuracy and response times across four experiments, they reveal that learning to avoid losses, while resulting in similar accuracy to reward learning, significantly slows down decision-making through mechanisms beyond simple task difficulty.

What Problem Does This Paper Address?

Traditional reinforcement learning models excel at predicting what action an agent will choose to maximize rewards or minimize punishments. However, they often overlook the how and when of these decisions, specifically how response times (RTs) and accuracy interact, and how contextual factors influence these dynamics. This is a significant gap because, in fields like perceptual decision-making, the relationship between speed and accuracy is crucial for understanding underlying cognitive processes. The authors aim to bridge this gap by investigating how two key contextual factors in RL – outcome valence (gains vs. losses) and feedback information (partial vs. complete) – influence not just choices, but also the speed and accuracy of those choices, and to disentangle the specific cognitive mechanisms at play.

Methods

The researchers employed a multi-pronged approach:

  1. Bayesian Meta-Analysis: They first aggregated data from four existing RL experiments using a Bayesian meta-analytic approach. This allowed them to robustly assess the main effects and interactions of valence and feedback on overall accuracy and RTs, treating each experiment as a fixed effect. This is a standard but powerful technique for synthesizing findings across studies.
  2. Diffusion Decision Model (DDM): To go beyond simple accuracy and RT measures, they fitted a hierarchical Bayesian DDM to the combined data. The DDM is a well-established sequential sampling model that decomposes decision-making into key parameters:
  3. Drift rate (v): The speed of evidence accumulation towards a decision.
  4. Threshold (a): The amount of evidence needed to make a decision (reflecting response caution).
  5. Non-decision time (NDT): Time spent on processes other than evidence accumulation (e.g., motor execution, stimulus encoding). This allowed them to test how valence and feedback affected these specific cognitive components.

  6. Reinforcement Learning Diffusion Decision Model (RLDDM): To capture the dynamic, trial-by-trial learning process, they integrated the DDM with a "RELATIVE" RL model. This RLDDM allowed them to link latent learning variables (like learned value differences and contextual conflict) directly to the DDM parameters (drift rate, threshold, and NDT). This is a novel and sophisticated approach that allows for a more mechanistic understanding of how learning unfolds and influences decision dynamics over time.

Datasets

The study utilized data from four previously conducted behavioral experiments. These experiments shared a common reinforcement learning paradigm with manipulations of outcome valence (reward vs. punishment) and feedback information (partial vs. complete). The combined dataset comprised 89 participants. No new datasets were introduced.

Key Insights and Ideas

  • Losses Slow Down Decisions, Not Just Choices: While participants learned to seek rewards and avoid losses with similar accuracy, they consistently took longer to make decisions in loss-avoidance contexts. This suggests that the cognitive processes involved in avoiding punishment are inherently slower, even if the outcome accuracy is comparable.
  • Feedback is Key for Both Speed and Accuracy: Complete feedback (knowing the outcome of both chosen and unchosen options) consistently led to faster and more accurate decisions compared to partial feedback. This effect was broad, influencing all DDM parameters (drift rate, threshold, and non-decision time).
  • Valence Primarily Affects Decision Speed (NDT & Threshold): Outcome valence (gains vs. losses) primarily impacted response times, not accuracy. Specifically, losses increased non-decision time (suggesting motor inhibition or a "hold your horses" response) and, in some cases, increased the decision threshold (indicating more caution).
  • Contextual Learning Modulates Decision Dynamics: The RLDDM revealed that learned contextual values (how rewarding or punishing a context is perceived) directly influenced non-decision time, explaining the slower RTs in punishing contexts. Similarly, trial-specific decision conflict (uncertainty about which option is better) modulated the decision threshold, making participants more cautious when conflict was high.
  • The Power of Joint Modeling: The study strongly advocates for jointly modeling choice and response time data. Analyzing them separately misses crucial insights into the underlying cognitive mechanisms, particularly how learning and context influence the speed-accuracy trade-off.

Key Results

  • Accuracy: Feedback information significantly affected accuracy, with complete feedback leading to higher accuracy. Valence did not have a significant main effect on accuracy.
  • Response Times (RTs): Both valence and feedback information, as well as their interaction, significantly affected RTs. Participants were slower in the loss domain and faster in complete feedback contexts.
  • DDM Parameters:
    • Feedback: Complete feedback increased drift rate and threshold, and decreased non-decision time.
  • Valence: Loss contexts showed higher thresholds and non-decision times compared to gain contexts. Valence did not affect drift rate.
  • RLDDM: Learned value differences positively modulated drift rate. Learned conflict negatively modulated the threshold. Learned context values negatively modulated non-decision time.

Limitations and Future Work

  • Experimental Specificity: While the meta-analytic approach and DDM/RLDDM fitting across four experiments aimed for generalizability, the authors note that the valence effect on the threshold was not entirely consistent across experiments, potentially due to variations in time pressure.
  • Non-Decision Time Effects: The authors highlight that effects on non-decision time are less commonly observed in SSM literature, and their interpretation (e.g., motor inhibition) is still an area for further exploration.
  • Future Directions: The study suggests that future work could explore how meta-cognitive judgments, such as decision confidence, are influenced by these learning contexts, as losses have been shown to reduce confidence. The RLDDM framework also opens avenues for investigating more complex learning scenarios and their impact on decision dynamics.

Summary updated

Abstract

Reinforcement learning (RL) models describe how humans and animals learn by trial-and-error to select actions that maximize rewards and minimize punishments. Traditional RL models focus exclusively on choices, thereby ignoring the interactions between choice preference and response time (RT), or how these interactions are influenced by contextual factors. However, in the field of perceptual decision-making, such interactions have proven to be important to dissociate between different underlying cognitive processes. Here, we investigated such interactions to shed new light on overlooked differences between learning to seek rewards and learning to avoid losses. We leveraged behavioral data from four RL experiments, which feature manipulations of two factors: outcome valence (gains vs. losses) and feedback information (partial vs. complete feedback). A Bayesian meta-analysis revealed that these contextual factors differently affect RTs and accuracy: While valence only affects RTs, feedback information affects both RTs and accuracy. To dissociate between the latent cognitive processes, we jointly fitted choices and RTs across all experiments with a Bayesian, hierarchical diffusion decision model (DDM). We found that the feedback manipulation affected drift rate, threshold, and non-decision time, suggesting that it was not a mere difficulty effect. Moreover, valence affected non-decision time and threshold, suggesting a motor inhibition in punishing contexts. To better understand the learning dynamics, we finally fitted a combination of RL and DDM (RLDDM). We found that while the threshold was modulated by trial-specific decision conflict, the non-decision time was modulated by the learned context valence. Overall, our results illustrate the benefits of jointly modeling RTs and choice data during RL, to reveal subtle mechanistic differences underlying decisions in different learning contexts.
BibTeX
@inproceedings{fontanesi2019decomposing,
  author = {Laura Fontanesi and Stefano Palminteri and Maël Lebreton},
  title = {{Decomposing the effects of context valence and feedback information on speed and accuracy during reinforcement learning: a meta-analytical approach using diffusion decision modeling}},
  booktitle = {Cognitive},
  year = {2019},
  doi = {10.3758/s13415-019-00723-1},
}
Figure