Trading off the cost of conflict against expected rewards

Nura Sidarus, Stefano Palminteri, Valérian ChambonbioRxiv 2018 · DOI

TL;DR

This study reveals that the subjective cost of conflict influences our decisions, sometimes even overriding our learned value representations. Interestingly, the impact of conflict on learning depends on its type: motivational conflicts (internal drives vs. instructions) hinder learning, while conflicts with external distractors do not appear to affect it.

What Problem Does This Paper Address?

We constantly make decisions, weighing potential rewards against the effort or costs involved. While we know that physical and mental effort act as costs, influencing our choices, it's less clear if experiencing conflict itself is perceived as a cost in value-based decision-making. This paper investigates whether the aversive nature of conflict, particularly when it arises from external stimuli conflicting with our internal value assessments, influences our choices and subsequent learning. The authors aim to disentangle the effects of different types of conflict (internal vs. external) on both decision-making and instrumental learning.

Methods

The researchers employed a combination of behavioral experiments and computational modeling.

  • Behavioral Task: Participants engaged in a reversal-learning task where they had to learn and adapt to changing reward probabilities associated with left/right actions. This task was augmented with a flanker task, where irrelevant arrow stimuli (distractors) were presented alongside the target. Trials were either "free" (participants chose their action) or "instructed" (participants followed a specific direction). This design allowed for the manipulation of conflict between external distractors and actions, as well as internal motivational conflicts (instruction vs. subjective value).
  • Computational Modeling: A reinforcement learning framework, specifically Q-learning, was used to model participants' behavior. Key parameters were estimated, including:
  • Distractor Bias (φ): This parameter captured the tendency to avoid conflict by favoring choices congruent with distractors.
  • Learning Rates (α): Separate learning rates were estimated for different conditions (free vs. instructed, congruent vs. incongruent, and for motivational conflicts).
  • Inverse Temperature (β): This parameter captured choice stochasticity.

Model comparison was used to determine the best-fitting model and to assess the influence of different conflict types on decision-making and learning.

Datasets

Not discussed in the paper. The study involved data collected from human participants.

Key Insights and Ideas

  • Conflict as a Cost: The study provides strong evidence that conflict itself is perceived as a cost. Participants were biased to follow distractor suggestions, especially when their internal evidence for an action was weak, suggesting a trade-off between expected reward and the cost of conflict.
  • Motivational Conflict Impedes Learning: When participants had to suppress their preferred action (based on learned values) to follow an instruction (a motivational conflict), their learning rates were significantly reduced. This suggests that the effort to resolve internal conflicts can devalue the learning from outcomes.
  • External Conflict's Limited Impact on Learning: Interestingly, conflict arising from external distractors (action vs. distractor) did not appear to significantly impact learning rates, unlike motivational conflicts. This suggests that the source of the conflict is crucial for its effect on learning.
  • Choice Matters for Learning: Learning rates were generally higher in "free" choice trials compared to "instructed" trials, suggesting that agency and self-directed learning enhance the learning process.

Key Results

  • Decision Stage: Participants showed a "distractor bias," meaning they were more likely to choose actions congruent with irrelevant distractors, especially when their internal value signals were weak. This bias was associated with slower reaction times when participants went against the distractor's suggestion, indicating a conflict cost.
  • Learning Stage:
    • Learning rates were lower in instructed trials compared to free trials.
  • Motivational conflict (instructed-low value trials) led to a significant reduction in learning rates compared to instructed-high value trials.
  • Conflict between actions and external distractors did not show a robust effect on learning rates in the best-fitting model.
  • Model Performance: The winning computational model (m8) successfully captured these behavioral patterns, incorporating a distractor bias parameter for decisions and separate learning rates for different types of instructed trials.

Limitations and Future Work

  • Sample Size: The authors acknowledge that the relatively small sample size (N=20) warrants caution in interpreting some results, particularly correlations.
  • Binary Conflict Categorization: The categorization of trials as either "congruent" or "incongruent" might oversimplify the continuous nature of conflict. Future work could explore more nuanced measures.
  • Distractor Facilitation vs. Avoidance: The study couldn't definitively distinguish whether distractors facilitated decisions by breaking ties between equally valued options or solely by promoting conflict avoidance. Future studies with neutral distractors could help disentangle these effects.
  • Generalizability: While the study demonstrates conflict adaptation effects in a reinforcement learning context, further research is needed to explore the generalizability of these findings across different tasks and populations.
  • Neuroimaging: The authors suggest that future studies employing neuroimaging techniques (e.g., M/EEG) could provide deeper insights into the neural mechanisms underlying conflict processing and its integration with decision-making and learning.

Summary updated

Abstract

Abstract Value-based decision-making involves trading off the cost associated with an action against its expected reward. Research has shown that both physical and mental effort constitute such subjective costs, biasing choices away from effortful actions, and discounting the value of obtained rewards. Facing conflicts between competing action alternatives is considered aversive, as recruiting cognitive control to overcome conflict is effortful. Yet, it remains unclear whether conflict is also perceived as a cost in value-based decisions. The present study investigated this question by embedding irrelevant distractors (flanker arrows) within a reversal-learning task, with intermixed free and instructed trials. Results showed that participants learned to adapt their choices to maximize rewards, but were nevertheless biased to follow the suggestions of irrelevant distractors. Thus, the perceived cost of being in conflict with an external suggestion could sometimes trump internal value representations. By adapting computational models of reinforcement learning, we assessed the influence of conflict at both the decision and learning stages. Modelling the decision showed that conflict was avoided when evidence for either action alternative was weak, demonstrating that the cost of conflict was traded off against expected rewards. During the learning phase, we found that learning rates were reduced in instructed, relative to free, choices. Learning rates were further reduced by conflict between an instruction and subjective action values, whereas learning was not robustly influenced by conflict between one’s actions and external distractors. Our results show that the subjective cost of conflict factors into value-based decision-making, and highlights that different types of conflict may have different effects on learning about action outcomes.
BibTeX
@misc{sidarus2018trading,
  author = {Nura Sidarus and Stefano Palminteri and Valérian Chambon},
  title = {{Trading off the cost of conflict against expected rewards}},
  year = {2018},
  doi = {10.1101/412809},
  howpublished = {bioRxiv},
}
Figure