Crossing the Reality Gap with Evolved Plastic Neurocontrollers

Huanneng Qiu, Matthew Garratt, David Howard, Sreenatha G. Anavatti

TL;DR

This paper tackles the challenge of transferring robot controllers trained in simulation to the real world, a problem exacerbated by the complex dynamics of small drones. They propose a novel approach using Spiking Neural Networks (SNNs) with evolved Hebbian plasticity, allowing the controller to adapt online to discrepancies between the simulated and real-world dynamics, thereby bridging the "reality gap."

What Problem Does This Paper Address?

The core problem is the "reality gap" in evolutionary robotics. Controllers are often trained in simulation because real-world training is time-consuming and potentially damaging. However, simulations are imperfect models of reality. When a controller, perfectly tuned for the simulation, is deployed on a real robot (like a small Unmanned Aerial Vehicle or UAV), its performance often degrades significantly due to unmodeled dynamics, parameter drift, or environmental changes. This is particularly problematic for highly dynamic systems like UAVs, where even small control errors can lead to crashes. Existing methods often rely on highly accurate simulation models, which are difficult to obtain, or on offline system identification, which doesn't account for real-time changes.

Methods

The authors introduce a system called "eSpinn" (Evolving Spiking Neural Networks) that combines neuroevolution with biologically inspired learning mechanisms. Here's a breakdown of their approach:

  • Spiking Neural Networks (SNNs): Unlike traditional Artificial Neural Networks (ANNs) that use continuous values, SNNs communicate using discrete "spikes" over time, mimicking biological neurons. This event-driven nature is well-suited for efficient computation and potentially for neuromorphic hardware. They use the Izhikevich model for their spiking neurons.
  • Neuroevolution (NEAT): They leverage the NeuroEvolution of Augmenting Topologies (NEAT) algorithm. NEAT is known for its ability to evolve both the network structure (topology) and the connection weights simultaneously, starting with simple networks and gradually increasing complexity.
  • Hebbian Plasticity: This is the key innovation. Instead of just evolving fixed weights, they evolve the rules that govern how synaptic weights change over time. Specifically, they use a rate-based Hebbian model inspired by Spiking-Timing Dependent Plasticity (STDP). The crucial aspect is that they evolve parameters (km and ke) that control this plasticity, allowing the network to adapt its connections online based on the activity of pre- and post-synaptic neurons. This means the controller can self-correct during operation.
  • Hierarchical Control and System Identification: For the UAV (a hexacopter), they first create a simplified, identified model of its vertical (heave) dynamics. This identified model is less complex than a full physics simulation but captures essential relationships between thrust, velocity, and acceleration. The neuroevolution process then trains the SNN controller to work with this identified model.

Datasets

The paper does not explicitly mention using pre-existing datasets. Instead, they create their own "dataset" through simulations and experiments:

  • Simulated Hexacopter Model: A Simulink model of a commercial hexacopter is developed, derived from first principles and incorporating non-linear aerodynamics. This serves as the primary environment for training and evaluating controllers.
  • Identified Heave Model: A simplified, linear model of the hexacopter's vertical dynamics is derived from the full simulation model. This identified model is used during the evolutionary training phase.
  • Real Hexacopter Platform: While the paper focuses on simulation-to-simulation transfer (from the identified model to the more complex hexacopter simulation), the ultimate goal is real-world deployment. The authors state they are in the process of applying their developed controllers to a real hexacopter platform.

Key Insights and Ideas

  • Plasticity as a Bridge: The most significant insight is that synaptic plasticity, when evolved, can act as a powerful mechanism to bridge the reality gap. Instead of trying to perfectly model reality, the controller learns to adapt to its imperfections.
  • Evolving the Learning Rule: Evolving the plasticity rule itself, rather than just fixed weights, allows for a more robust and generalizable adaptation strategy. This means the controller doesn't just learn a specific set of corrections but learns how to learn and adapt.
  • Simulation-to-Simulation Transfer: The paper demonstrates successful transfer from a simpler identified model to a more complex simulation model. This is a crucial intermediate step, showing that the plastic controller can handle changes in dynamics, which is a proxy for the reality gap.
  • SNNs for Dynamic Systems: The use of SNNs, with their inherent temporal processing capabilities, is shown to be effective for controlling highly dynamic systems like UAVs.

Key Results

  • Improved Fitness: Plastic controllers consistently achieved higher fitness values than non-plastic controllers when tested on the more complex hexacopter simulation model (Table 2 shows a statistically significant improvement, p < 0.05).
  • Faster Adaptation: The plastic controllers demonstrated faster response times and smaller steady-state errors in height control compared to non-plastic controllers (Figure 9).
  • Online Adaptation: When a sub-optimal network was given the evolved plastic rules, it showed significant performance improvement over multiple runs, gradually reducing steady-state error (Figure 10).
  • Competitive with PID: The plastic SNN controller achieved a lower mean absolute error (0.090 m) in height control compared to a tuned PID controller (0.108 m) designed directly on the hexacopter model (Figure 11).

Limitations and Future Work

  • Simulation-to-Simulation Transfer: While successful, the primary validation is still within simulation. The ultimate test on a real hexacopter is ongoing.
  • Heave Control Focus: The current work focuses on height control. Expanding to encompass all degrees of freedom for the UAV is a stated future direction.
  • Evolving Plasticity Rules: The paper acknowledges that while evolution finds optimal plasticity rules, there's no guarantee that any arbitrary Hebbian rule would lead to desired synaptic changes. The evolutionary process is key to finding effective rules.
  • Real-Time Adaptation: The authors are actively working on deploying this strategy to a real hexacopter platform, which will be the ultimate validation of their approach to real-time, online adaptation.

Summary updated

Abstract

A critical issue in evolutionary robotics is the transfer of controllers learned in simulation to reality. This is especially the case for small Unmanned Aerial Vehicles (UAVs), as the platforms are highly dynamic and susceptible to breakage. Previous approaches often require simulation models with a high level of accuracy, otherwise significant errors may arise when the well-designed controller is being deployed onto the targeted platform. Here we try to overcome the transfer problem from a different perspective, by designing a spiking neurocontroller which uses synaptic plasticity to cross the reality gap via online adaptation. Through a set of experiments we show that the evolved plastic spiking controller can maintain its functionality by self-adapting to model changes that take place after evolutionary training, and consequently exhibit better performance than its non-plastic counterpart.
BibTeX
@inproceedings{qiu2020crossing,
  author = {Huanneng Qiu and Matthew Garratt and David Howard and Sreenatha G. Anavatti},
  title = {{Crossing the Reality Gap with Evolved Plastic Neurocontrollers}},
  booktitle = {GECCO},
  year = {2020},
  url = {https://openreview.net/forum?id=StWZCq-9BA},
}
Figure