Credit assignment to state-independent task representations and its relationship with model-based decision making

Nitzan Shahar, Rani Moran, Tobias U. Hauser, Rogier Kievit, Daniel McNamee, Michael Moutoussis, Raymond J. Dolan, Edward T. Bullmore, Ian Goodyer, Peter Fonagy, Peter B. Jones, Sharon Neufeld, Rafael Romero-García, Michelle St Clair, Petra E. Vértes, Kirstie Whitaker, Becky Inkster, Gita Prabhu, Cinly Ooi, Umar Toseeb, Barry Widmer, Junaid Bhatti, Laura Villis, Ayesha Alrumaithi, Sarah Birt, Aislinn Bowler, Kalia Cleridou, Hina Dadabhoy, Emma Davies, Ashlyn Firkins, Sian Granville, Elizabeth Harding, Alexandra Hopkins, Daniel Isaacs, Janchai King, Danae Kokorikou, Christina Maurice, Cleo McIntosh, Jessica Memarzia, Harriet L. Mills, Ciara O’Donnell, Sara Pantaleone, Jenny Scott, Pasco Fearon, John Suckling, Anne‐Laura van HarmelenProceedings of the National Academy of Sciences 2019 · DOI

TL;DR

This paper challenges the assumption that model-free learning only uses outcome-relevant information. They demonstrate that a model-free system can assign value to irrelevant spatial-motor aspects of a task, influencing decisions even when these features don't predict rewards. This suggests a more complex and potentially less regulated credit assignment process in model-free learning than previously thought.

What Problem Does This Paper Address?

In reinforcement learning, agents learn to make decisions by associating actions with outcomes. Two primary learning strategies are model-free (learning directly from experience without building an explicit model of the environment) and model-based (learning an internal model of the environment's structure to predict future outcomes). A key assumption in model-free learning is that it efficiently focuses on features that are predictive of rewards. For instance, when learning the value of a food item, a model-free system should ideally learn the value of the food itself, not the spatial position of the food on the table or the specific hand used to pick it up, as these are typically irrelevant to the food's taste or nutritional value.

However, real-world environments are complex, and sometimes irrelevant features can become intertwined with task performance. This paper investigates whether a model-free system can "know" what to ignore, specifically whether it assigns value to task representations that are irrelevant to the outcome, even when it has minimal knowledge of the environment's overall structure. They question if this "obligatory" credit assignment to irrelevant features can be regulated.

Methods

The researchers employed a well-established 2-step decision task with a large sample of 769 healthy adolescents. This task involves navigating between two stages to earn rewards. Crucially, in this specific task design, the stimulus identity (fractal images) predicted the reward, while the spatial-motor aspects (e.g., left/right key presses, fractal location on the screen) did not. This dissociation allowed them to disentangle the influence of outcome-relevant (fractal identity) versus outcome-irrelevant (spatial-motor) information.

They used two main analytical approaches:

  1. Model-Agnostic Sequential Analysis: This involved examining participants' choices across consecutive trials. By analyzing patterns of response key selection (e.g., repeating a key press after a reward vs. no reward) in different contexts (within the same task state, across different states, and across different task stages), they could infer whether value was being assigned to spatial-motor features.
  2. Computational Modeling: They fitted five different reinforcement learning models to the participants' data. These models varied in how they incorporated spatial-motor information into both model-free and model-based learning components. By comparing the model fits (using Bayesian Information Criterion - BIC), they could determine which model best explained the observed behavior and quantify the contribution of spatial-motor learning.

Additionally, they used structural equation modeling to investigate the relationship between the strength of spatial-motor value associations and the deployment of model-based strategies.

Datasets

The study utilized data from 769 healthy adolescents (ages 14-24) collected at three time points over approximately 18 months as part of the NSPN (Neuroscience in Psychiatry Network) study. This longitudinal aspect allowed for the examination of learning over time and across different measurements. No new datasets were introduced; rather, existing behavioral data from a well-defined cognitive task was analyzed.

Key Insights and Ideas

  • Obligatory Credit Assignment: The most striking finding is that the model-free system appears to assign value to spatial-motor representations regardless of whether these features predict outcomes. This suggests that credit assignment in model-free learning might not be as strictly regulated by outcome relevance as commonly assumed.
  • State-Independent Value: The influence of these spatial-motor value associations extended across different task states and even across different task stages, indicating that the learned values were not tied to specific environmental contexts.
  • Model-Based Control as a Regulator: Individuals who exhibited stronger model-based decision-making strategies showed a reduced influence of these spatial-motor value associations. This suggests that a more sophisticated, goal-directed strategy might act as a top-down regulator, dampening the impact of irrelevant learned values.
  • Implications for Disorders: The findings could have implications for understanding disorders characterized by compulsive or habitual behaviors (e.g., addiction, obesity), where less goal-directed control is often observed, potentially leading to stronger influences of irrelevant motor-action associations.

Key Results

  • Within-State Analysis: Participants were significantly more likely to repeat a fractal choice after a rewarded trial when the response mapping (fractal to key press) was the same as the previous trial, compared to when it was flipped. This effect was stronger (37.26% vs. 21.39% increase in fractal stay probability) when the mapping was repeated, indicating that reward influenced not just the fractal choice but also the response key itself.
  • Between-States and Between-Stages Analysis: Reward history at one trial significantly influenced response key selection in subsequent trials, even when the fractal stimuli changed or when moving between task stages. For example, repeating a rewarded response key at the second stage increased the probability of selecting the same key at the first stage of the next trial by 4.13%.
  • Computational Modeling: The best-fitting model (Model 2) included a separate learning system for spatial-motor values, significantly improving its ability to predict participants' choices compared to models without this component.
  • Relationship Analysis: Structural equation modeling revealed a significant negative correlation between the strength of spatial-motor value associations and model-based learning (-0.32, P < 0.001), supporting the idea that model-based control mitigates the influence of irrelevant learned values.

Limitations and Future Work

  • Confounding of Spatial Location and Motor Response: The study acknowledges that spatial location and motor response were perfectly confounded in their task design. Future research should aim to disentangle the specific contributions of these two aspects.
  • Nature of the Association: The study couldn't definitively determine whether the observed effect was due to unregulated learning (formation of associations), unregulated decision-making (influence of associations on behavior), or both.
  • Stimulus-Motor Combination: Participants might also assign credit to a combination of stimulus and motor response/object location, which was not fully explored.
  • Future Directions: The authors suggest further investigation into how states are structured and represented within the model-free system, and how memory resource limitations might influence the ability to differentiate between intrinsic and extrinsic state information, potentially leading to greater involvement of outcome-irrelevant task representations. They also propose exploring whether impairments in certain patient populations are more domain-specific than previously thought, potentially relating to these motor-outcome associations.

Summary updated

Abstract

Model-free learning enables an agent to make better decisions based on prior experience while representing only minimal knowledge about an environment's structure. It is generally assumed that model-free state representations are based on outcome-relevant features of the environment. Here, we challenge this assumption by providing evidence that a putative model-free system assigns credit to task representations that are irrelevant to an outcome. We examined data from 769 individuals performing a well-described 2-step reward decision task where stimulus identity but not spatial-motor aspects of the task predicted reward. We show that participants assigned value to spatial-motor representations despite it being outcome irrelevant. Strikingly, spatial-motor value associations affected behavior across all outcome-relevant features and stages of the task, consistent with credit assignment to low-level state-independent task representations. Individual difference analyses suggested that the impact of spatial-motor value formation was attenuated for individuals who showed greater deployment of goal-directed (model-based) strategies. Our findings highlight a need for a reconsideration of how model-free representations are formed and regulated according to the structure of the environment.
BibTeX
@inproceedings{shahar2019credit,
  author = {Nitzan Shahar and Rani Moran and Tobias U. Hauser and Rogier Kievit and Daniel McNamee and Michael Moutoussis and Raymond J. Dolan and Edward T. Bullmore and Raymond J. Dolan and Ian Goodyer and Peter Fonagy and Peter B. Jones and Michael Moutoussis and Tobias U. Hauser and Sharon Neufeld and Rafael Romero-García and Michelle St Clair and Petra E. Vértes and Kirstie Whitaker and Becky Inkster and Gita Prabhu and Cinly Ooi and Umar Toseeb and Barry Widmer and Junaid Bhatti and Laura Villis and Ayesha Alrumaithi and Sarah Birt and Aislinn Bowler and Kalia Cleridou and Hina Dadabhoy and Emma Davies and Ashlyn Firkins and Sian Granville and Elizabeth Harding and Alexandra Hopkins and Daniel Isaacs and Janchai King and Danae Kokorikou and Christina Maurice and Cleo McIntosh and Jessica Memarzia and Harriet L. Mills and Ciara O’Donnell and Sara Pantaleone and Jenny Scott and Pasco Fearon and John Suckling and Anne‐Laura van Harmelen and Rogier Kievit},
  title = {{Credit assignment to state-independent task representations and its relationship with model-based decision making}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2019},
  doi = {10.1073/pnas.1821647116},
}

Loading related content...

Figure