Human subjects exploit a cognitive map for credit assignment

Rani Moran, Peter Dayan, Raymond J. DolanProceedings of the National Academy of Sciences 2021 · DOI

TL;DR

This paper reveals how humans use internal "cognitive maps" of task structure to improve how they learn from rewards. They found that this map helps both "model-free" (habitual) and "model-based" (planning) learning systems assign credit more effectively, with model-free learning benefiting from knowing which rewards are directly caused by an action, and model-based learning benefiting from knowing which rewards are most informative for future decisions.

What Problem Does This Paper Address?

Reinforcement learning (RL) theories often describe decision-making as a competition or collaboration between two systems: model-free (MF) and model-based (MB). MF learning is like building habits based on direct reward experience, while MB learning uses a "cognitive map" (CM) of the environment to plan ahead and predict outcomes. While much research has focused on how these systems interact during decision-making, less is known about how the CM influences credit assignment – how we learn from the outcomes of our choices. Specifically, how does knowing the task's structure (via a CM) help us decide which past actions or states deserve credit (or blame) for a received reward, especially when rewards are complex or not directly tied to a single action? This paper aims to fill this gap by investigating how CM knowledge shapes credit assignment in both MF and MB learning.

Methods

The researchers designed a novel dual-outcome bandit task where participants had to choose between different "people" (representing options) who grew pairs of "vegetables" (representing outcomes). Crucially, participants were given full information about the task structure, allowing them to build a cognitive map. They then analyzed participant behavior using two main approaches:

  • Model-agnostic analyses: This involved looking at how choices on one trial predicted choices on the next, and how past rewards influenced these predictions. They used logistic mixed-effects models to disentangle the effects of different types of outcomes (choice-related vs. unrelated, important vs. unimportant).
  • Computational modeling: They developed a sophisticated hybrid RL model that incorporated both MF and MB learning with parameters for credit assignment. This model was fitted to the entire dataset, allowing for a more detailed examination of how different aspects of credit assignment contributed to behavior. They also performed model comparisons to determine which components of their model were essential for explaining the data.

Datasets

The study used data collected from 42 human participants performing a custom-designed dual-outcome bandit task. No pre-existing datasets were used for training or evaluation. The task itself, while inspired by previous work, was a novel variant designed to specifically probe the influence of cognitive maps on credit assignment.

Key Insights and Ideas

  • Cognitive Maps Guide Habitual Learning (MFCA): Even the seemingly simple, habit-based MF system benefits from a cognitive map. Knowing the task structure allows the MF system to assign credit more effectively by distinguishing between rewards directly caused by an action ("related") and those that are not ("unrelated").
  • Negative Credit for Unrelated Rewards: Surprisingly, rewards not directly related to a chosen action actually led to negative credit assignment for that action in the MF system. This suggests a mechanism for actively down-weighting or even penalizing actions that lead to irrelevant outcomes, potentially promoting exploration.
  • Importance Matters for Planning (MBCA): The MB system, which uses the cognitive map for planning, prioritizes rewards that are "important" for distinguishing between choices. Outcomes that offer more unique information for decision-making receive greater credit assignment.
  • Model-Based Learning Outperforms Model-Free: Simulations showed that, in this task, a purely model-based approach to credit assignment generally led to higher earnings than a purely model-free approach.
  • Hybrid Systems are Common: While pure MF or MB learners exist, a significant portion of participants (around 32%) showed evidence of using a hybrid approach, combining both strategies.

Key Results

  • MF Credit Assignment: Participants showed enhanced MF credit assignment for choice-related outcomes compared to choice-unrelated outcomes. Choice-unrelated rewards led to negative MF credit assignment.
  • MB Credit Assignment: MB credit assignment was significantly boosted for "important" outcomes (those that helped differentiate between choices) compared to "unimportant" outcomes.
  • Model Comparison: The full hybrid model, incorporating both MF and MB credit assignment modulated by cognitive map information, provided the best fit to the data, significantly outperforming simpler models that lacked these components.
  • Simulations: Pure MB agents achieved higher simulated earnings than pure MF agents. CM guidance for MFCA, particularly negative credit for unrelated outcomes, was found to be instrumental for simulated earnings.

Limitations and Future Work

  • Ecological Validity of Importance: The study notes that the "importance" of outcomes in their task was somewhat arbitrary and not temporally consistent. In more naturalistic environments where importance might be more stable, importance-based MBCA could be more beneficial. Future work could explore tasks with temporally autocorrelated importance.
  • Individual Differences: The study acknowledges that not all participants will use all CM-based strategies simultaneously. Future research could investigate how individuals arbitrate between these strategies and what situational factors influence this arbitration.
  • Cognitive Costs: The paper suggests that a critical next step is to characterize and quantify the cognitive costs associated with MB and MF control, as well as CM-based modulation of learning. This would provide a more complete understanding of the trade-offs involved.
  • Neural Mechanisms: While the study provides behavioral evidence, future work could explore the neural underpinnings of these CM-guided credit assignment processes.

Summary updated

Abstract

An influential reinforcement learning framework proposes that behavior is jointly governed by model-free (MF) and model-based (MB) controllers. The former learns the values of actions directly from past encounters, and the latter exploits a cognitive map of the task to calculate these prospectively. Considerable attention has been paid to how these systems interact during choice, but how and whether knowledge of a cognitive map contributes to the way MF and MB controllers assign credit (i.e., to how they revaluate actions and states following the receipt of an outcome) remains underexplored. Here, we examine such sophisticated credit assignment using a dual-outcome bandit task. We provide evidence that knowledge of a cognitive map influences credit assignment in both MF and MB systems, mediating subtly different aspects of apparent relevance. Specifically, we show MF credit assignment is enhanced for those rewards that are related to a choice, and this contrasted with choice-unrelated rewards that reinforced subsequent choices negatively. This modulation is only possible based on knowledge of task structure. On the other hand, MB credit assignment was boosted for outcomes that impacted on differences in values between offered bandits. We consider mechanistic accounts and the normative status of these findings. We suggest the findings extend the scope and sophistication of cognitive map-based credit assignment during reinforcement learning, with implications for understanding behavioral control.
BibTeX
@inproceedings{moran2021human,
  author = {Rani Moran and Peter Dayan and Raymond J. Dolan},
  title = {{Human subjects exploit a cognitive map for credit assignment}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2021},
  doi = {10.1073/pnas.2016884118},
}
Figure