Abstract
The hippocampus and the striatum support episodic and procedural memory, respectively, and "place" and "response" learning within spatial navigation. Recently this dichotomy has been linked to "model-based" and "model-free" reinforcement learning. Here we present a well-constrained neural model of how both systems support spatial navigation, and apply the same model to more abstract problems such as sequential decision making. In particular, we show that if a task can be transformed into a Markov Decision Process, the machinery provided by the hippocampus and striatum can be utilized to solve it. These results show how the hippocampal complex can represent non-spatial problems, including context, probabilities and action-dependent information, in support of "model-based" reinforcement learning to complement learning within the striatum.
Introduction
Neuroscientists have long observed a functional dichotomy in how the brain learns: the hippocampus facilitates "model-based" learning by constructing flexible cognitive maps of the world, while the striatum governs "model-free" learning through the formation of rigid, automatic habits. While this division is well-documented in spatial navigation—such as a rat finding its way through a physical maze—it remains less clear how these systems handle abstract, non-spatial tasks. This paper proposes that our "spatial" brain machinery is actually a versatile architecture designed to solve any Markov Decision Process (MDP), effectively treating abstract choices as paths on a mental map.
Model Architecture
The authors developed a biologically constrained neural model that integrates three primary modules to simulate decision-making processes.
- Hippocampal Module: Utilizes "place cells" to create allocentric (map-like) representations. In abstract tasks, these cells represent sensory cues as "landmarks."
- Striatal Module: Learns stimulus-response (S-R) associations through reinforcement learning (trial and error).
- Prefrontal Cortex (PFC) Integration: Acts as a selector that compares the outputs of both systems to produce a final overt action.
A key technical innovation is the use of "splitter place cells." In non-deterministic tasks where outcomes are probabilistic, the model "unwraps" these probabilities into independent neural chains. This allows the brain to mentally simulate and "probe" the value of a path as if it were a deterministic route in a physical maze.
Experimental Validation and Results
The model was validated against two landmark experimental datasets. First, the spatial navigation behavior of rats in a "plus maze" (Packard and McGaugh, 1996) provided a baseline. The model successfully mirrored the observed transition from hippocampal "place" strategies (flexible navigation) to striatal "response" strategies (habitual behavior) over time.
Second, the model was applied to a two-step abstract decision task in humans involving digital cards with varying reward probabilities (Daw et al., 2011). The hippocampal component correctly accounted for the task's transition structure, reproducing the specific "stay" probabilities seen in humans, where subjects adjust their choices based on whether a previous transition was "common" or "rare."
Discussion and Conclusion
The core insight of this work is that the brain does not distinguish between a physical hallway and a logical choice. It treats sensory cues—whether they are physical walls or digital symbols—as landmarks in a "mental maze." Computationally, navigating a logic tree is identical to navigating a city. The model suggests the PFC serves as an arbiter, choosing between the flexible, map-making hippocampus and the fast, efficient striatum.
While the model provides a unified architecture for spatial and abstract tasks, limitations remain regarding high-dimensional problems (tasks with dozens of variables) and the representation of "hidden" states, such as aesthetics or complex numbers, where there is no immediate visual landmark. Future research may explore how the hippocampus represents body part motion or the actions of other agents.
Summary updated