TL;DR
This paper challenges the assumption that model-free learning only uses outcome-relevant information. They demonstrate that a model-free system can assign value to irrelevant spatial-motor aspects of a task, influencing decisions even when these features don't predict rewards. This suggests a more complex and potentially less regulated credit assignment process in model-free learning than previously thought.
What Problem Does This Paper Address?
In reinforcement learning, agents learn to make decisions by associating actions with outcomes. Two primary learning strategies are model-free (learning directly from experience without building an explicit model of the environment) and model-based (learning an internal model of the environment's structure to predict future outcomes). A key assumption in model-free learning is that it efficiently focuses on features that are predictive of rewards. For instance, when learning the value of a food item, a model-free system should ideally learn the value of the food itself, not the spatial position of the food on the table or the specific hand used to pick it up, as these are typically irrelevant to the food's taste or nutritional value.
However, real-world environments are complex, and sometimes irrelevant features can become intertwined with task performance. This paper investigates whether a model-free system can "know" what to ignore, specifically whether it assigns value to task representations that are irrelevant to the outcome, even when it has minimal knowledge of the environment's overall structure. They question if this "obligatory" credit assignment to irrelevant features can be regulated.
Methods
The researchers employed a well-established 2-step decision task with a large sample of 769 healthy adolescents. This task involves navigating between two stages to earn rewards. Crucially, in this specific task design, the stimulus identity (fractal images) predicted the reward, while the spatial-motor aspects (e.g., left/right key presses, fractal location on the screen) did not. This dissociation allowed them to disentangle the influence of outcome-relevant (fractal identity) versus outcome-irrelevant (spatial-motor) information.
They used two main analytical approaches:
- Model-Agnostic Sequential Analysis: This involved examining participants' choices across consecutive trials. By analyzing patterns of response key selection (e.g., repeating a key press after a reward vs. no reward) in different contexts (within the same task state, across different states, and across different task stages), they could infer whether value was being assigned to spatial-motor features.
- Computational Modeling: They fitted five different reinforcement learning models to the participants' data. These models varied in how they incorporated spatial-motor information into both model-free and model-based learning components. By comparing the model fits (using Bayesian Information Criterion - BIC), they could determine which model best explained the observed behavior and quantify the contribution of spatial-motor learning.
Additionally, they used structural equation modeling to investigate the relationship between the strength of spatial-motor value associations and the deployment of model-based strategies.
Datasets
The study utilized data from 769 healthy adolescents (ages 14-24) collected at three time points over approximately 18 months as part of the NSPN (Neuroscience in Psychiatry Network) study. This longitudinal aspect allowed for the examination of learning over time and across different measurements. No new datasets were introduced; rather, existing behavioral data from a well-defined cognitive task was analyzed.
Key Insights and Ideas
- Obligatory Credit Assignment: The most striking finding is that the model-free system appears to assign value to spatial-motor representations regardless of whether these features predict outcomes. This suggests that credit assignment in model-free learning might not be as strictly regulated by outcome relevance as commonly assumed.
- State-Independent Value: The influence of these spatial-motor value associations extended across different task states and even across different task stages, indicating that the learned values were not tied to specific environmental contexts.
- Model-Based Control as a Regulator: Individuals who exhibited stronger model-based decision-making strategies showed a reduced influence of these spatial-motor value associations. This suggests that a more sophisticated, goal-directed strategy might act as a top-down regulator, dampening the impact of irrelevant learned values.
- Implications for Disorders: The findings could have implications for understanding disorders characterized by compulsive or habitual behaviors (e.g., addiction, obesity), where less goal-directed control is often observed, potentially leading to stronger influences of irrelevant motor-action associations.
Key Results
- Within-State Analysis: Participants were significantly more likely to repeat a fractal choice after a rewarded trial when the response mapping (fractal to key press) was the same as the previous trial, compared to when it was flipped. This effect was stronger (37.26% vs. 21.39% increase in fractal stay probability) when the mapping was repeated, indicating that reward influenced not just the fractal choice but also the response key itself.
- Between-States and Between-Stages Analysis: Reward history at one trial significantly influenced response key selection in subsequent trials, even when the fractal stimuli changed or when moving between task stages. For example, repeating a rewarded response key at the second stage increased the probability of selecting the same key at the first stage of the next trial by 4.13%.
- Computational Modeling: The best-fitting model (Model 2) included a separate learning system for spatial-motor values, significantly improving its ability to predict participants' choices compared to models without this component.
- Relationship Analysis: Structural equation modeling revealed a significant negative correlation between the strength of spatial-motor value associations and model-based learning (-0.32, P < 0.001), supporting the idea that model-based control mitigates the influence of irrelevant learned values.
Limitations and Future Work
- Confounding of Spatial Location and Motor Response: The study acknowledges that spatial location and motor response were perfectly confounded in their task design. Future research should aim to disentangle the specific contributions of these two aspects.
- Nature of the Association: The study couldn't definitively determine whether the observed effect was due to unregulated learning (formation of associations), unregulated decision-making (influence of associations on behavior), or both.
- Stimulus-Motor Combination: Participants might also assign credit to a combination of stimulus and motor response/object location, which was not fully explored.
- Future Directions: The authors suggest further investigation into how states are structured and represented within the model-free system, and how memory resource limitations might influence the ability to differentiate between intrinsic and extrinsic state information, potentially leading to greater involvement of outcome-irrelevant task representations. They also propose exploring whether impairments in certain patient populations are more domain-specific than previously thought, potentially relating to these motor-outcome associations.
Summary updated