TL;DR
This paper reveals how humans use internal "cognitive maps" of task structure to improve how they learn from rewards. They found that this map helps both "model-free" (habitual) and "model-based" (planning) learning systems assign credit more effectively, with model-free learning benefiting from knowing which rewards are directly caused by an action, and model-based learning benefiting from knowing which rewards are most informative for future decisions.
What Problem Does This Paper Address?
Reinforcement learning (RL) theories often describe decision-making as a competition or collaboration between two systems: model-free (MF) and model-based (MB). MF learning is like building habits based on direct reward experience, while MB learning uses a "cognitive map" (CM) of the environment to plan ahead and predict outcomes. While much research has focused on how these systems interact during decision-making, less is known about how the CM influences credit assignment – how we learn from the outcomes of our choices. Specifically, how does knowing the task's structure (via a CM) help us decide which past actions or states deserve credit (or blame) for a received reward, especially when rewards are complex or not directly tied to a single action? This paper aims to fill this gap by investigating how CM knowledge shapes credit assignment in both MF and MB learning.
Methods
The researchers designed a novel dual-outcome bandit task where participants had to choose between different "people" (representing options) who grew pairs of "vegetables" (representing outcomes). Crucially, participants were given full information about the task structure, allowing them to build a cognitive map. They then analyzed participant behavior using two main approaches:
- Model-agnostic analyses: This involved looking at how choices on one trial predicted choices on the next, and how past rewards influenced these predictions. They used logistic mixed-effects models to disentangle the effects of different types of outcomes (choice-related vs. unrelated, important vs. unimportant).
- Computational modeling: They developed a sophisticated hybrid RL model that incorporated both MF and MB learning with parameters for credit assignment. This model was fitted to the entire dataset, allowing for a more detailed examination of how different aspects of credit assignment contributed to behavior. They also performed model comparisons to determine which components of their model were essential for explaining the data.
Datasets
The study used data collected from 42 human participants performing a custom-designed dual-outcome bandit task. No pre-existing datasets were used for training or evaluation. The task itself, while inspired by previous work, was a novel variant designed to specifically probe the influence of cognitive maps on credit assignment.
Key Insights and Ideas
- Cognitive Maps Guide Habitual Learning (MFCA): Even the seemingly simple, habit-based MF system benefits from a cognitive map. Knowing the task structure allows the MF system to assign credit more effectively by distinguishing between rewards directly caused by an action ("related") and those that are not ("unrelated").
- Negative Credit for Unrelated Rewards: Surprisingly, rewards not directly related to a chosen action actually led to negative credit assignment for that action in the MF system. This suggests a mechanism for actively down-weighting or even penalizing actions that lead to irrelevant outcomes, potentially promoting exploration.
- Importance Matters for Planning (MBCA): The MB system, which uses the cognitive map for planning, prioritizes rewards that are "important" for distinguishing between choices. Outcomes that offer more unique information for decision-making receive greater credit assignment.
- Model-Based Learning Outperforms Model-Free: Simulations showed that, in this task, a purely model-based approach to credit assignment generally led to higher earnings than a purely model-free approach.
- Hybrid Systems are Common: While pure MF or MB learners exist, a significant portion of participants (around 32%) showed evidence of using a hybrid approach, combining both strategies.
Key Results
- MF Credit Assignment: Participants showed enhanced MF credit assignment for choice-related outcomes compared to choice-unrelated outcomes. Choice-unrelated rewards led to negative MF credit assignment.
- MB Credit Assignment: MB credit assignment was significantly boosted for "important" outcomes (those that helped differentiate between choices) compared to "unimportant" outcomes.
- Model Comparison: The full hybrid model, incorporating both MF and MB credit assignment modulated by cognitive map information, provided the best fit to the data, significantly outperforming simpler models that lacked these components.
- Simulations: Pure MB agents achieved higher simulated earnings than pure MF agents. CM guidance for MFCA, particularly negative credit for unrelated outcomes, was found to be instrumental for simulated earnings.
Limitations and Future Work
- Ecological Validity of Importance: The study notes that the "importance" of outcomes in their task was somewhat arbitrary and not temporally consistent. In more naturalistic environments where importance might be more stable, importance-based MBCA could be more beneficial. Future work could explore tasks with temporally autocorrelated importance.
- Individual Differences: The study acknowledges that not all participants will use all CM-based strategies simultaneously. Future research could investigate how individuals arbitrate between these strategies and what situational factors influence this arbitration.
- Cognitive Costs: The paper suggests that a critical next step is to characterize and quantify the cognitive costs associated with MB and MF control, as well as CM-based modulation of learning. This would provide a more complete understanding of the trade-offs involved.
- Neural Mechanisms: While the study provides behavioral evidence, future work could explore the neural underpinnings of these CM-guided credit assignment processes.
Summary updated