Abstract
Studies suggest that dopaminergic neurons report a unitary, global reward prediction error signal. However, learning in complex real-life tasks, in particular tasks that show hierarchical structure, requires multiple prediction errors that may coincide in time. We used functional neuroimaging to measure prediction error signals in humans performing such a hierarchical task involving simultaneous, uncorrelated prediction errors. Analysis of signals in a priori anatomical regions of interest in the ventral striatum and the ventral tegmental area indeed evidenced two simultaneous, but separable, prediction error signals corresponding to the two levels of hierarchy in the task. This result suggests that suitably designed tasks may reveal a more intricate pattern of firing in dopaminergic neurons. Moreover, the need for downstream separation of these signals implies possible limitations on the number of different task levels that we can learn about simultaneously.
Introduction
In the field of reinforcement learning (RL), the Reward Prediction Error (RPE)—the difference between expected and actual outcomes—is the fundamental signal used to update future expectations. For decades, the prevailing view in neuroscience has been that dopaminergic neurons in the basal ganglia broadcast a single, "unitary" RPE signal throughout the brain. This global signal is thought to provide a universal teaching signal for all types of learning.
However, human behavior often involves hierarchical structures. For example, when visiting a specific casino to play a specific slot machine, a learner must track two distinct values: the quality of that specific machine and the overall quality of the casino. If a single global signal were used, it would be difficult to distinguish which level of the hierarchy should be updated when an outcome occurs. This study investigates whether the human brain can represent multiple, simultaneous, and uncorrelated prediction errors to support hierarchical learning.
Methods and Experimental Design
To test this hypothesis, the researchers designed a "Casino Task" and monitored 28 healthy participants using high-resolution functional MRI (fMRI).
In this task, participants chose between two "casinos." Upon entering, they played two different slot machines in sequence. Each machine yielded a certain number of points (Slot Level). To "win" the casino and earn a monetary reward (10 cents), participants had to reach a specific point target using the sum of the two slot machines (Casino Level).
The critical moment occurred when the second slot machine finished: participants learned both the points from that machine and whether they had won the overall casino reward. The task was specifically designed so that these two outcomes were uncorrelated; for instance, a participant could receive a high point value from a slot machine but still fail to win the casino if the target was very high, or vice versa. This allowed the researchers to separate the RPEs for the slot level and the casino level.
Key Insights and Results
Behavioral analysis confirmed that participants were "smart" hierarchical learners. Their subsequent casino choices were driven by previous casino wins rather than raw point totals, while their slot machine choices were driven by points rather than the incidental casino outcome. This double dissociation provides strong evidence that humans naturally organize learning into hierarchical structures.
Computational modeling supported this, showing that a hierarchical "Outcome Model" significantly outperformed simpler, non-hierarchical models in explaining participant behavior ($p < 0.002$).
The fMRI results provided the most striking evidence against the unitary signal theory. In the Ventral Striatum (VS), BOLD activity correlated significantly with both the slot-level RPE ($p = 0.0269$) and the casino-level RPE ($p < 0.0001$) at the exact same moment in time. The Ventral Tegmental Area (VTA), where dopamine neurons originate, also showed significant correlation with the slot RPE and a strong trend for the casino-level RPE ($p = 0.051$). Crucially, the mean correlation between these two RPE regressors was nearly zero (-0.029), proving the brain was tracking two distinct streams of information.
Discussion
While the study demonstrates that the basal ganglia can handle multiple RPEs, it also highlights an anatomical overlap. The researchers found that these signals were interspersed within the same regions rather than being spatially segregated into distinct "casino" or "slot" zones.
This finding raises the "spatial credit assignment problem": if these signals are physically overlapping, how do downstream brain areas know which error belongs to which level of the hierarchy? The authors suggest this may represent a cognitive bottleneck, potentially limiting the number of hierarchical levels humans can learn about simultaneously. Future research using higher temporal or spatial resolution may reveal whether these signals are separated by specific neural channels or temporal frequencies that fMRI cannot currently detect.
Summary updated