Abstract
How do humans learn which actions are responsible for a reward that arrives much later? This study investigates the "credit assignment problem" using a novel sequential decision-making task. By analyzing behavior and pupil dilation across visual, auditory, and spatial modalities, researchers found that human learning is best explained by model-free reinforcement learning with "eligibility traces"—a decaying memory of past actions. This mechanism allows for one-shot reinforcement of entire action sequences, a process reflected in both choice patterns and physiological responses after a single reward.
Introduction
A fundamental mystery of human cognition is how we learn which actions are responsible for a reward that only arrives much later. Imagine learning a new video game: you perform a sequence of ten moves and finally reach a treasure chest. How does your brain know that the third move was the crucial one? This is known as the credit assignment problem.
In Reinforcement Learning (RL) theory, two main mechanisms address this:
- Temporal-Difference (TD) Learning (e.g., TD-0): Reward information "travels back" only one step at a time. Learning a long sequence requires many repetitions.
- Eligibility Traces (ET): The brain maintains a temporary "tag" on recently visited states and actions. When a reward occurs, all "tagged" states are updated simultaneously, enabling one-shot learning of entire sequences.
While ETs are a staple of machine learning, direct evidence for them in human sequential decision-making has been elusive until now.
Methods and Experimental Design
The researchers designed a novel sequential learning task where participants moved through states (represented by stimuli) to reach a goal (G). The critical path was: Start → D2 → D1 → Goal. To ensure the effect was general, the experiment used three modalities: visual clip-art (12 participants), acoustic sounds (15 participants), and spatial locations (22 participants).
The "One-Shot" Test
The study focused on the very second episode, immediately following the first reward. Under TD-0, participants should only learn about state D1 (one step from the goal). If humans use ETs, they should already show a preference for the correct action at state D2 (two steps from the goal).
Physiological Proxy: Pupil Dilation
Pupil size is a known correlate of Reward Prediction Error (RPE) and state value. The researchers hypothesized that if a state’s value is updated via an eligibility trace, the pupil should dilate more upon encountering that state in the second episode compared to the first.
Key Insights
- The "Aha!" Moment at D2: Participants repeated the correct action at state D2 in approximately 85% of cases during the second episode. This rapid reinforcement of a non-adjacent state is a clear signature of one-shot learning that classic TD-0 models cannot explain.
- The Pupil as a Value Tracker: Even before making a choice, participants' pupils dilated significantly more when encountering state D2 in the second episode. This suggests the brain assigned a higher "value" to that state after just a single reward.
- The 10-Second Window: By fitting RL models (such as $Q(\lambda)$ and $SARSA(\lambda)$), the authors estimated the eligibility trace's time constant ($\tau$) to be approximately 10 seconds.
- Model-Free Supremacy: Model-free RL with eligibility traces explained behavior better than more complex "model-based" algorithms involving internal maps, suggesting ETs are a robust, possibly "hard-wired" feature of human learning.
Key Results
The study found significant behavioral and physiological evidence supporting the eligibility trace hypothesis across all three sensory modalities.
- Behavioral Bias: Choice probability at D2 was significantly higher than chance ($p < 0.001$) after a single reward.
- Model Comparison: Using the Akaike Information Criterion (AIC), models incorporating eligibility traces significantly outperformed those without (like $Q-0$).
- Pupillary Signature: Pupil dilation was significantly stronger in Episode 2 vs. Episode 1 for both D1 and D2 across all modalities, confirming a physiological update of state values.
Limitations and Future Work
While these results are compelling, questions remain regarding whether the eligibility trace decays strictly over time or over the number of steps taken. Additionally, while pupillometry implicates neuromodulators like norepinephrine or dopamine, the specific brain regions (e.g., striatum vs. prefrontal cortex) implementing these traces are yet to be definitively identified. Future research could also explore how eligibility traces scale in more complex, deeper decision trees where credit might be spread thin.
Summary updated