Abstract: This work has been funded partially by the EU project MirrorBot, grant IST-2001-35282, and NEST-043374 coordinated by SW. CW, JT and AF are supported by the Hertie Foundation, and the EU projects PLICON, grant MEXT-CT-2006-042484, and Daisy, grant FP6-2005015803. Urs Bergmann provided feedback on the manuscript.
Introduction
While modern computers excel at discrete tasks like chess, robots often struggle with the basic motor and adaptive capabilities of a human child. A primary reason for this discrepancy is the challenge of state representation. Standard Reinforcement Learning (RL) algorithms typically require discrete states to function, whereas robots must navigate high-dimensional, continuous sensory data.
This paper explores how the architecture of the mammalian brain—specifically the basal ganglia and cortex—can serve as a blueprint for creating adaptive robots. By mapping RL components onto biological structures, the authors propose a framework that combines unsupervised representation learning with RL-driven action selection.
Mapping RL to Biological Anatomy
The authors map the "Actor-Critic" framework onto the Basal Ganglia (BG). In this biological model, the Striatum functions as the state representation. It is described as a "silent structure" where sparse activity prevents reward signals from interfering with unrelated states, ensuring stable learning. The GPi/SNr nuclei represent the action output, while Dopamine (from the SNc/VTA) acts as the Temporal Difference (TD) prediction error ($\delta$).
Crucially, dopamine is not a simple "pleasure" signal; it is a predictor. Once a task is learned, dopamine neurons fire at the cue predicting a reward rather than the reward itself. This mapping provides a biological basis for how agents learn goal-directed behaviors through interaction with their environment.
Computational Algorithms in the Brain
The paper compares two primary RL algorithms: TD-Learning and SARSA. While both are effective in robotics, the authors argue that SARSA is more biologically plausible. SARSA updates values based on the chosen next action, which aligns more closely with neural firing patterns observed in primates.
Additionally, the paper introduces AGREL (Attention-Gated RL) for classification tasks. AGREL mimics how the visual cortex tunes itself to "diagnostic features" relevant to a specific task, using a reward signal to guide weight updates in a multilayer perceptron. This hierarchical approach allows lower centers to handle basic survival while higher centers (BG and Cortex) manage complex, learned tasks.
Experimental Validation
The theories were validated across several robotic and biological benchmarks:
- Khepera Robot: Used to test a basal ganglia model for behavior selection, such as switching between "cylinder-pickup" and "wall-follow" tasks.
- PeopleBot Robot: Successfully learned complex docking behaviors by combining unsupervised visual feature extraction with a TD-learning actor-critic network.
- 1-2-AX Task: Demonstrated how the Prefrontal Cortex (PFC) and BG interact to manage nested working memory tasks.
- Biological Consistency: The models align with existing neuroscientific data from rat T-mazes and monkey saccade (eye movement) tasks.
Conclusion and Future Directions
Despite the success of these models, challenges remain. A significant hurdle is the Single Reward Signal Problem: in complex environments, multiple learning strategies run in parallel, yet there is only one global dopamine signal. The authors suggest "reward sharing" as a potential solution.
Future work should move beyond reactive "behaviorism" toward Neuroconstructivism, where systems are proactive and responses depend on internal context. Integrating language input to "coach" motor sequences also represents a promising frontier for human-robot interaction, potentially allowing robots to learn from natural language instructions.
Summary updated