Autonomous reinforcement of behavioral sequences in neural dynamics

Sohrob Kazerounian, Matthew D. Luciw, Yulia Sandamirskaya, Mathis Richter, Jürgen Schmidhuber, Gregor SchönerICDL-EPIROB 2012 · DOI

TL;DR

This paper introduces DN-SARSA(λ), a novel neural-dynamic reinforcement learning algorithm that bridges the gap between continuous, biologically plausible neural representations and the discrete, reward-driven learning of behavioral sequences. It demonstrates how a robot can autonomously learn complex action sequences from delayed rewards, mimicking how biological systems might learn.

What Problem Does This Paper Address?

Traditional reinforcement learning (RL) often relies on discrete state and action spaces, which doesn't align well with how biological systems perceive and act in the continuous, dynamic real world. While computational neuroscience models exist for continuous representations, they often struggle with delayed rewards and general RL problems. This paper tackles the challenge of enabling a neural system to learn sequences of behaviors from potentially delayed and non-specific reward signals, integrating the strengths of Dynamic Field Theory (DFT) for continuous representations with the learning guarantees of RL. Specifically, it addresses how to learn rewarding sequences through delayed reinforcement and how to generate these sequences using real sensors and motors.

Methods

The core of the paper is the DN-SARSA(λ) algorithm, which combines three key components:

  • Dynamic Field Theory (DFT): This mathematical framework models neural populations as continuous fields (Dynamic Neural Fields or DNFs). DNFs represent activation distributions over graded dimensions (like color or space) and evolve in continuous time. They are well-suited for handling continuous time and real-world environments.
  • Standard Reinforcement Learning (RL): The paper leverages the principles of RL, particularly the SARSA(λ) algorithm, which is known for its convergence guarantees. SARSA(λ) uses eligibility traces to update not just the immediate state-action value but also a history of past values.
  • Item and Order Working Memory: To implement the eligibility trace in a biologically plausible way, the authors use a computational neuroscience model of working memory. This model encodes the order of a sequence of events by the relative levels of activity across neurons, which is analogous to how eligibility traces function in RL.

The DN-SARSA(λ) architecture integrates these by using DNFs to represent elementary behaviors (EBs), which are defined by intention and condition-of-satisfaction (CoS) nodes. The learning process involves updating value weights based on TD-errors, which are computed using the eligibility trace mechanism.

Datasets

Not discussed in the paper. The authors use a simulated robotic vehicle (E-Puck) in a Webots environment for testing.

Key Insights and Ideas

  • Bridging the Discretization Gap: The paper demonstrates a successful integration of continuous neural dynamics (DFT) with discrete RL concepts (SARSA(λ)), showing that complex behavioral sequences can be learned without sacrificing biological plausibility.
  • Working Memory as Eligibility Trace: A key insight is the realization that a computational model of Item and Order working memory can effectively serve as the eligibility trace mechanism in RL, providing a neural basis for this crucial RL component.
  • Autonomous Sequence Learning: The system learns a specific rewarding sequence (G → B → Y → R → G) autonomously through exploration and exploitation, highlighting the potential for robots to discover and master complex tasks without explicit programming of the sequence.
  • Handling Delayed Rewards: The use of eligibility traces is shown to be crucial for learning from delayed rewards, a common challenge in RL and a significant limitation in many prior neural models.

Key Results

  • The DN-SARSA(λ) algorithm successfully learned a specific five-item color sequence (G → B → Y → R → G) in a simulated E-Puck robot.
  • The learned behavior stabilized into a consistent, optimal sequence during the exploitation phase after an initial exploration phase.
  • The system demonstrated robustness to perceptual noise and fluctuations in the visual channel when transferred to a real robot.
  • The average TD-error decreased over time, indicating that the neural system learned to predict discounted future reward.

Limitations and Future Work

  • Exploration Strategy: The paper acknowledges that the exploration phase, which involved simple random action selection, is a limitation. Future work could involve more sophisticated exploration methods.
  • Simplified EBs: The proof-of-concept implementation used simplified elementary behaviors. The authors suggest that the DN-SARSA(λ) system could work with more sophisticated DNF behaviors for complex tasks like obstacle avoidance.
  • Integration with Architectures: Future work could involve integrating DN-SARSA(λ) with architectures for organizing elementary behaviors, potentially leading to learning behavioral constraints within such systems.

Summary updated

BibTeX
@inproceedings{kazerounian2012autonomous,
  author = {Sohrob Kazerounian and Matthew D. Luciw and Yulia Sandamirskaya and Mathis Richter and Jürgen Schmidhuber and Gregor Schöner},
  title = {{Autonomous reinforcement of behavioral sequences in neural dynamics}},
  booktitle = {ICDL-EPIROB},
  year = {2012},
  doi = {10.1109/devlrn.2012.6400831},
}
Figure