TL;DR
This paper proposes a unified computational model where the hippocampus and dorsal striatum (specifically DLS) handle distinct but complementary learning strategies for decision-making. The model explains how the brain flexibly switches between rapid, stimulus-response (model-free) learning in the striatum and more deliberate, relational (model-based) planning in the hippocampus, based on the reliability of each system's predictions. This framework successfully accounts for a wide range of behavioral findings in both spatial navigation and non-spatial decision tasks.
What Problem Does This Paper Address?
Neuroscience and reinforcement learning (RL) research have long observed that animals and humans employ multiple strategies for making decisions, particularly when aiming to maximize rewards. Two prominent distinctions are:
- Model-Free (MF) vs. Model-Based (MB) RL: MF learning relies on trial-and-error to directly associate states with actions and their expected rewards. MB learning, on the other hand, builds an internal model of the environment to simulate future outcomes and plan actions. MF is fast but less flexible, while MB is flexible but computationally expensive.
- Egocentric vs. Allocentric Navigation: In spatial navigation, "response learning" uses egocentric cues (relative to the agent's body) and is linked to the dorsal striatum (DLS). "Place learning" uses allocentric cues (fixed in the environment) and is associated with the hippocampus (HPC).
While these dichotomies seem parallel, their precise neural underpinnings and how they interact have been a subject of ongoing research. Specifically, it's unclear how the HPC's role in place learning relates to its potential contribution to MB planning, and how these systems are integrated. This paper aims to bridge this gap by proposing a unified model that explains how the HPC and DLS, through distinct learning mechanisms, contribute to both spatial and non-spatial decision-making, and how their outputs are arbitrated.
Methods
The core of this paper is a novel computational model that integrates two distinct RL systems:
- Dorsal Striatal (DLS) System: Implements model-free (MF) reinforcement learning. It learns direct associations between egocentric sensory inputs (e.g., landmark features) and actions. This is achieved using a Q-learning algorithm with eligibility traces, which allows for efficient updates based on reward prediction errors.
- Hippocampal (HPC) System: Implements a successor representation (SR), which is a form of model-based (MB) learning. The SR represents the expected discounted future occupancy of each state. In this model, place cells (driven by boundaries) are used to encode this SR, allowing the HPC to learn a "cognitive map" of relational structures between states. This enables flexible planning and re-evaluation of rewards.
Arbitration Mechanism: The model arbitrates between the DLS and HPC outputs by weighting their estimated action values based on their reliability. Reliability is measured by the recent average of prediction errors (reward prediction error for DLS, successor state prediction error for HPC). This means the system relies more on the prediction from the system that has been more accurate recently. This arbitration is inspired by previous work and is suggested to be supported by the medial prefrontal cortex.
The model was tested by simulating various behavioral experiments involving spatial navigation (water maze, plus maze) and non-spatial sequential decision tasks.
Datasets
The paper does not introduce new datasets. Instead, it uses existing experimental paradigms and their associated behavioral data from published studies to validate its model simulations. These include:
- Water Maze Experiments: Adapted versions of the Morris water maze, involving landmark and boundary cues, and manipulations like hippocampal lesions.
- Plus Maze Experiments: Tasks designed to distinguish between place and response learning, and to test for reward devaluation sensitivity.
- Two-Step Decision Tasks: Sequential decision-making tasks where participants choose between options with probabilistic transitions and varying reward probabilities.
Key Insights and Ideas
- Unified Role of HPC and DLS: The model proposes that the HPC's role in allocentric place learning is fundamentally the same as its role in MB planning – both involve learning the relational structure of the environment via a successor representation. Similarly, the DLS's role in egocentric response learning is analogous to its role in MF RL.
- Reliability-Based Arbitration: The brain doesn't just pick one strategy; it dynamically weighs the outputs of different systems based on their recent predictive accuracy. This explains how behavior can shift depending on the task and learning history.
- Boundary-Driven Place Cells and Blocking: A key insight is how modeling place cells as being driven by boundaries (rather than just landmarks) explains why boundary-guided navigation is not susceptible to the "blocking effect" seen in landmark-guided navigation. This is because the SR learned from boundaries captures relational structure that is more robust to changes in individual cues.
- SR as a Bridge: The Successor Representation (SR) is presented as a crucial intermediate mechanism. It allows for flexible MB-like planning at a lower computational cost than full MB models, by decoupling learning about transitions from learning about rewards.
- Generalization Across Domains: The model's ability to explain behavior in both spatial and non-spatial tasks suggests that the underlying computational principles of the hippocampal-striatal system are generalizable for decision-making.
Key Results
- Spatial Navigation: The model accurately reproduced behavioral findings from hippocampal and DLS lesion studies in water maze tasks, showing impaired intrasession learning with HPC lesions and preserved learning across sessions, consistent with a shift to response strategies.
- Plus Maze: The model simulated the shift from place to response strategies over training, and how DLS inactivation prevented this shift while HPC inactivation promoted it early on. It also captured devaluation sensitivity, a hallmark of MB learning, which was preserved in the full model but impaired in the HPC-lesioned simulation.
- Non-Spatial Two-Step Task: The model successfully mimicked human behavior, showing a blend of MF (striatal) and MB (hippocampal) influences. The striatal component increased "stay probability" after rewards, while the hippocampal SR component allowed for more flexible, goal-directed choices.
- Blocking Phenomenon: The model demonstrated blocking in landmark-guided navigation (consistent with DLS involvement) but not in boundary-guided navigation (consistent with HPC involvement and SR learning).
- Correlation between MB and Spatial Memory: Simulations showed a significant correlation between MB planning ability and allocentric spatial memory in the full model, which was absent in the HPC-lesioned model, mirroring human experimental findings.
Limitations and Future Work
- SR vs. Full MB: The paper acknowledges that the SR model, while powerful, has limitations. It can be policy-dependent and may not be as flexible as a full MB model when transition structures change rapidly. Future work could explore how hippocampal replay or other mechanisms might address these limitations.
- Arbitration Mechanism Details: While the model proposes reliability-based arbitration, the precise neural implementation (e.g., involvement of the medial prefrontal cortex) is not fully detailed and remains an area for future investigation.
- Generalization of SR: The paper suggests that the SR's ability to learn temporal abstractions is key to its role in MB planning, especially for multi-step tasks. Further research could explore how this SR mechanism generalizes to more complex, real-world decision-making scenarios.
- OFC and State Representation: The role of the orbitofrontal cortex (OFC) in representing states is mentioned as distinct from and potentially complementary to the HPC's role. Future work could investigate how OFC and HPC interact for flexible decision-making.
- Boundary-Driven Place Cells: The model's explanation for the absence of blocking in boundary navigation relies on place cells being driven by boundaries. Further experimental validation of this specific neural mechanism would be valuable.
Summary updated