Reinforcement Learning Embedded in Brains and Robots

Cornelius Weber, Mark Elshaw, Stefan Wermter, Jochen Triesch, Christopher WillmotOpenalex Unknown 2008 · DOI · OpenAlex

Abstract: This work has been funded partially by the EU project MirrorBot, grant IST-2001-35282, and NEST-043374 coordinated by SW. CW, JT and AF are supported by the Hertie Foundation, and the EU projects PLICON, grant MEXT-CT-2006-042484, and Daisy, grant FP6-2005015803. Urs Bergmann provided feedback on the manuscript.

Introduction

While modern computers excel at discrete tasks like chess, robots often struggle with the basic motor and adaptive capabilities of a human child. A primary reason for this discrepancy is the challenge of state representation. Standard Reinforcement Learning (RL) algorithms typically require discrete states to function, whereas robots must navigate high-dimensional, continuous sensory data.

This paper explores how the architecture of the mammalian brain—specifically the basal ganglia and cortex—can serve as a blueprint for creating adaptive robots. By mapping RL components onto biological structures, the authors propose a framework that combines unsupervised representation learning with RL-driven action selection.

Mapping RL to Biological Anatomy

The authors map the "Actor-Critic" framework onto the Basal Ganglia (BG). In this biological model, the Striatum functions as the state representation. It is described as a "silent structure" where sparse activity prevents reward signals from interfering with unrelated states, ensuring stable learning. The GPi/SNr nuclei represent the action output, while Dopamine (from the SNc/VTA) acts as the Temporal Difference (TD) prediction error ($\delta$).

Crucially, dopamine is not a simple "pleasure" signal; it is a predictor. Once a task is learned, dopamine neurons fire at the cue predicting a reward rather than the reward itself. This mapping provides a biological basis for how agents learn goal-directed behaviors through interaction with their environment.

The primary basal ganglia-thalamus-cortex loop and its pathways.

Computational Algorithms in the Brain

The paper compares two primary RL algorithms: TD-Learning and SARSA. While both are effective in robotics, the authors argue that SARSA is more biologically plausible. SARSA updates values based on the chosen next action, which aligns more closely with neural firing patterns observed in primates.

Additionally, the paper introduces AGREL (Attention-Gated RL) for classification tasks. AGREL mimics how the visual cortex tunes itself to "diagnostic features" relevant to a specific task, using a reward signal to guide weight updates in a multilayer perceptron. This hierarchical approach allows lower centers to handle basic survival while higher centers (BG and Cortex) manage complex, learned tasks.

Comparison of TD-Learning and SARSA architectures, learnt weights, and algorithms.

Experimental Validation

The theories were validated across several robotic and biological benchmarks:

  • Khepera Robot: Used to test a basal ganglia model for behavior selection, such as switching between "cylinder-pickup" and "wall-follow" tasks.
  • PeopleBot Robot: Successfully learned complex docking behaviors by combining unsupervised visual feature extraction with a TD-learning actor-critic network.
  • 1-2-AX Task: Demonstrated how the Prefrontal Cortex (PFC) and BG interact to manage nested working memory tasks.
  • Biological Consistency: The models align with existing neuroscientific data from rat T-mazes and monkey saccade (eye movement) tasks.

Conclusion and Future Directions

Despite the success of these models, challenges remain. A significant hurdle is the Single Reward Signal Problem: in complex environments, multiple learning strategies run in parallel, yet there is only one global dopamine signal. The authors suggest "reward sharing" as a potential solution.

Future work should move beyond reactive "behaviorism" toward Neuroconstructivism, where systems are proactive and responses depend on internal context. Integrating language input to "coach" motor sequences also represents a promising frontier for human-robot interaction, potentially allowing robots to learn from natural language instructions.

Summary updated

Abstract

This work has been funded partially by the EU project MirrorBot, grant IST-2001-35282, and NEST-043374 coordinated by SW. CW, JT and AF are supported by the Hertie Foundation, and the EU projects PLICON, grant MEXT-CT-2006-042484, and Daisy, grant FP6-2005015803. Urs Bergmann provided feedback on the manuscript.
BibTeX
@inproceedings{weber2008reinforcement,
  author = {Cornelius Weber and Mark Elshaw and Stefan Wermter and Jochen Triesch and Christopher Willmot},
  title = {{Reinforcement Learning Embedded in Brains and Robots}},
  booktitle = {Openalex Unknown},
  year = {2008},
  doi = {10.5772/5278},
}
Figure