Robustness in the long run : Auto-teaching vs. Anticipation in Evolutionary Robotics

Nicolas Godzik, Marc Schoenauer2004

TL;DR

This paper tackles the problem of long-term unreliability in "auto-teaching" neural networks used in evolutionary robotics. They propose a novel "Anticipatory" architecture that significantly improves robustness by having a secondary network predict future sensor states, leading to more stable and adaptive robot behavior over extended periods.

What Problem Does This Paper Address?

Evolutionary Robotics (ER) aims to create robots that can adapt to changing environments. A popular approach, "auto-teaching networks," uses a two-part neural network: an "Agent Model" that's fixed and an "Agent" that learns and adapts its weights during its lifetime to match the model. While effective for short-term adaptation, the paper highlights a critical flaw: this approach can lead to unpredictable and unstable behavior when the robot operates for extended periods beyond its initial training. The core issue is that the learning mechanism is only optimized for a limited "training period," and what happens afterward is not directly accounted for, potentially leading to "opportunistic" solutions that fail in the long run. This paper investigates this long-term failure mode and proposes a more robust alternative.

Methods

The authors introduce the "Action, Anticipation, Adaptation" (AAA) architecture. This builds upon the auto-teaching concept by adding a "Model network." This Model network takes the robot's current sensations and the actuator commands as input and attempts to predict the next state of the robot's sensors. The error between the predicted and actual sensor states is then used via back-propagation to adjust the weights of both the Model network and the original Agent network. This "anticipation" mechanism provides a more grounded error signal, directly related to the robot's interaction with its environment, rather than relying solely on a pre-defined, potentially outdated, Agent Model.

Datasets

The experiments were conducted in a simulated environment using a Khepera robot. The environment involved navigating an arena with walls whose color could change (alternating between dark and white). This setup was designed to mimic the experimental conditions of prior work on auto-teaching networks, allowing for direct comparison. No new datasets were introduced; rather, the existing simulation environment was used to test the robustness of the proposed architecture.

Key Insights and Ideas

  • The "Opportunistic" Trap: Auto-teaching networks can find solutions that work well for the limited training period but are fundamentally unstable for long-term operation. Evolution essentially finds a "loophole" that doesn't generalize.
  • Anticipation as a Stabilizer: By having a network predict future sensor states, the AAA architecture grounds the learning process in the actual consequences of the robot's actions, leading to more stable and predictable behavior.
  • Learning from "True Errors": The AAA architecture uses errors derived from actual environmental feedback (predicted vs. real sensor states) for adaptation, which is more reliable than errors based on a potentially flawed or static "Agent Model."
  • Emergent Adaptability: Surprisingly, the AAA architecture demonstrated an ability to adapt to changing environmental conditions (wall colors) even after extended periods of operation, a capability lacking in the auto-teaching approach.

Key Results

  • Dramatic Improvement in Long-Term Robustness: While auto-teaching networks often started crashing after around 10 epochs, the AAA architecture showed remarkable stability, with many individuals running for thousands of epochs (up to 50,000 in some cases) without crashing.
  • Adaptive Behavior: In experiments where wall colors changed, the AAA architecture showed evidence of adaptation. For instance, after training in a white environment, robots could initially crash in a dark environment but would eventually recover and navigate safely.
  • Comparable Short-Term Performance: The learning curves during the initial evolution phase for the AAA architecture were similar to those of the auto-teaching networks, indicating that the proposed method doesn't sacrifice initial performance for long-term gains.

Limitations and Future Work

  • Understanding Re-adaptation: The paper acknowledges that the exact mechanisms behind the AAA architecture's ability to re-adapt online after long periods of operation are not fully understood and require further investigation.
  • Scaling to More Sensors: The authors suggest exploring how the AAA architecture can be scaled to robots with more complex sensor inputs, such as vision systems, potentially by using information bottleneck principles to compress sensor data.
  • Real-World Validation: The current experiments are in simulation. The ultimate validation would be to test the AAA architecture in real-world environments, though the authors note that the current experimental setup might be difficult to port directly due to result variance.
  • Alternative Error Usage: The paper mentions that the prediction errors could also be directly fed as inputs to the actuator network, a variation not explored in this study but considered for future work.

Summary updated

Abstract

In Evolutionary Robotics, auto-teaching networks, neural networks that modify their own weights during the life-time of the robot, have been shown to be powerful architectures to develop adaptive con-trollers. Unfortunately, when run for a longer period of time than that used during evolution, the long-term behavior of such networks can be-come unpredictable. This paper gives an example of such dangerous be-havior, and proposes an alternative solution based on anticipation: as in auto-teaching networks, a secondary network is evolved, but its out-puts try to predict the next state of the robot sensors. The weights of the action network are adjusted using some back-propagation procedure based on the errors made by the anticipatory network. First results – in simulated environments – show a tremendous increase in robustness of the long-term behavior of the controller. 1
BibTeX
@misc{godzik2004robustness,
  author = {Nicolas Godzik and Marc Schoenauer},
  title = {{Robustness in the long run : Auto-teaching vs. Anticipation in Evolutionary Robotics}},
  year = {2004},
}
Figure