TL;DR
This paper investigates how humans adapt their reinforcement learning (RL) strategies based on context, specifically by adjusting outcome values relative to a reference point and the range of experienced outcomes. They found that while this "state-dependent valuation" improves learning efficiency, it leads to seemingly irrational choices when options are evaluated outside their original learning contexts.
What Problem Does This Paper Address?
In everyday decision-making, the value we assign to an outcome isn't absolute; it's influenced by our recent experiences and expectations. This is well-established in economics and perception, but its role in reinforcement learning (RL) has been less explored. Standard RL models often assume absolute value learning. However, real-world scenarios involve contextual effects like "reference-point dependence" (evaluating outcomes as gains or losses relative to a baseline) and "range-adaptation" (adjusting sensitivity based on the magnitude of experienced outcomes). This paper aims to understand how these contextual factors influence human RL behavior and to develop computational models that capture this state-dependent valuation.
Methods
The researchers designed a probabilistic instrumental learning task where they orthogonally manipulated outcome valence (reward/loss) and magnitude (big/small) across different "contexts" (fixed pairs of abstract stimuli). Participants learned to associate stimuli with outcomes. They then tested participants in a "transfer test" where options were presented in novel combinations, allowing them to assess how learned values generalized.
To model the observed behavior, they compared several RL algorithms:
- ABSOLUTE: A standard Q-learning model that learns context-independent, absolute outcome values.
- RELATIVE: A model incorporating both reference-point dependence and range-adaptation, encoding outcomes as "correct" or "incorrect" within a context.
- HYBRID: A model that combines ABSOLUTE and RELATIVE learning, with a subject-specific weight parameter (w) balancing the influence of absolute and relative value encoding.
- UTILITY: A model based on economic theory of marginally decreasing utility for larger magnitudes.
- POLICY: A model that normalizes values at the decision stage (softmax) rather than during outcome evaluation.
Model comparison and simulation were used to determine the best-fitting model.
Datasets
The study used data collected from 60 healthy human participants performing the described probabilistic instrumental learning task. The task was conducted in two variants (Experiment 1 and Experiment 2), with Experiment 2 including an additional factor of feedback information (partial vs. complete). The data collected included choice behavior during learning sessions and a transfer test. No new benchmark datasets were introduced.
Key Insights and Ideas
- Context is King (for Learning): Humans don't learn absolute values; they dynamically adjust them based on the context of rewards and losses. This "state-dependent valuation" is crucial for efficient learning.
- Two Flavors of Context: The study highlights two key contextual adaptations:
- Reference-point dependence: Outcomes are perceived as gains or losses relative to a learned baseline.
- Range-adaptation: The perceived magnitude of an outcome is influenced by the range of magnitudes previously experienced.
- The "Irrational" Trade-off: While state-dependent valuation makes learning locally adaptive (e.g., better learning in high-magnitude contexts), it leads to "irrational" choices in the transfer test. This means participants might prefer an option that was objectively less rewarding in its original context when faced with novel combinations.
- Progressive Adaptation: State-dependent valuation isn't an all-or-nothing phenomenon; it emerges and strengthens progressively throughout the learning process.
- Understanding Matters: Participants who explicitly understood the task structure (i.e., recognized the fixed pairs of stimuli as contexts) showed a stronger tendency towards relative value learning.
Key Results
- Model Comparison: The HYBRID model, which combines absolute and relative value learning, provided the best fit to the behavioral data, outperforming models that relied solely on absolute or fully relative valuation.
- Magnitude Effect: Outcome magnitude significantly affected learning performance, with larger magnitudes leading to better learning. Valence did not have a significant effect.
- Transfer Test Inversions: Participants showed preference reversals in the transfer test, choosing options that were objectively less favorable based on absolute expected value. For example, a favorable option in a "reward/small" context was preferred over an unfavorable option in a "reward/big" context.
- Learning Progression: Trial-by-trial analysis showed that early in learning, absolute value models fit better, while later in learning, relative value models became more predictive.
- Feedback Influence: Complete feedback (showing the outcome of the unchosen option) led to a stronger reliance on relative value learning compared to partial feedback.
- Correlation with Understanding: A positive correlation was found between participants' explicit understanding of the task's contextual structure and their reliance on relative value learning.
Limitations and Future Work
- Partial Adaptation: While the HYBRID model captures a mix of absolute and relative learning, it suggests that neither extreme (fully absolute or fully relative) is optimal. This implies that humans operate under a form of "bounded rationality" where they balance different valuation strategies.
- Future Directions: The authors suggest future research could explore how this state-dependent valuation is implemented in the brain (e.g., dual valuation vs. integrated brain regions) and how it might predict real-world outcomes like socio-economic achievements or psychiatric conditions. They also propose investigating how factors like cognitive load or unconscious processing might impair this normalization process. The study also notes that while the HYBRID model accurately captures behavior, it's descriptive, and further work could explore more mechanistic models.
Summary updated