The Effect of Training Schedules on Morphological Robustness and Generalization

Edoardo Barba, Anil Yaman, Giovanni Iacca

Abstract

In the field of robotics, the "reality gap" remains a significant hurdle: controllers optimized in simulation often fail when deployed on physical hardware due to manufacturing variances, wear, or environmental damage. This paper investigates how the schedule and distribution of morphological variations during training influence a controller's robustness to small perturbations and its ability to generalize to entirely unseen body types. Through a series of experiments using neuroevolution on standard continuous control benchmarks, the authors demonstrate that the specific strategy used to present morphological diversity—ranging from curriculum learning to sampling extreme edge cases—is a primary determinant of a controller's ultimate versatility.

Introduction

Developing a "one-size-fits-all" robot controller requires more than simply exposing an agent to random variations. While introducing variability during training is a known technique for improving performance, the specific dynamics of this exposure—the "how" and "when"—remain under-explored. This research addresses the distinction between morphological robustness (the ability to handle minor variations within a known range) and generalization (the ability to adapt to body types outside the training distribution).

By evaluating different training schedules and sampling distributions, the authors aim to identify which strategies best bridge the reality gap and allow evolved neural network controllers to maintain high performance across a wide spectrum of physical configurations.

Methodology

The researchers employed eXponential Natural Evolution Strategy (xNES), a form of neuroevolution, to evolve the weights of artificial neural networks (ANNs) serving as agents' controllers. The study focused on three primary types of training schedules:

  1. Discrete Schedules: These included a Random selection from a fixed set of 36 body types and an Incremental schedule, which functioned as a curriculum by gradually increasing the complexity of the morphological variations.
  2. Continuous Distributions: Body parameters were sampled from various statistical distributions, including Uniform, Gaussian, Cauchy, and Beta. Notably, the Beta distribution was configured to prioritize sampling from the "edges" or extremes of the parameter space.
  3. Multi-Armed Bandit (MAB): An adaptive approach using Thompson Sampling to dynamically prioritize body types that provided the highest training signal (rewarded by improvements in a validation set).

Experimental Setup

The study utilized three benchmarks from OpenAI Gym and MuJoCo: Bipedal Walker, Walker2D, and Ant. Variations were applied to leg length and width. To measure performance, the authors split the morphological space into a Training Set (to test interpolation/robustness) and a Testing Set (to test extrapolation/generalization to parameters outside the training range).

Key Insights and Results

The Power of the Extremes

The Beta distribution emerged as the superior strategy for generalization. By forcing the controller to master "edge cases"—the most extreme body shapes—during the training phase, the resulting AI developed a more flexible control policy capable of handling body types it had never encountered. On the Bipedal Walker test set, the Beta schedule achieved a median score of -35.99, significantly outperforming other distributions that struggled to maintain stability in extrapolated environments.

The Efficacy of Curriculum Learning

The results highlighted that the order of presentation is as critical as the content. In the Walker2D task, the Incremental schedule reached a median training score of -1450.41, compared to -1094.39 for the Random schedule. This suggests that a gradual "climbing" of task complexity allows evolution to find stable, high-performing solutions that random jumping often misses.

Adaptive Teaching via MAB

The Multi-Armed Bandit approach acted as an automated teacher, successfully identifying and prioritizing the most difficult morphological regions to control. For Walker2D, the MAB approach significantly outperformed the Random baseline (p < 0.05), demonstrating that focusing computational resources on regions "prone to improvement" leads to more efficient learning without degrading performance in easier regions.

The Robustness-Generalization Trade-off

A key finding of the study is the inherent trade-off between robustness and generalization. While the Beta distribution was optimal for generalizing to new shapes, the Uniform distribution often provided better overall robustness within the known training range. This suggests that the choice of training schedule should be dictated by the specific requirements of the deployment environment.

Conclusion and Future Work

This research demonstrates that training schedules are a vital component in the evolution of robust robotic controllers. The success of the Beta distribution suggests that exposure to extreme morphological variations is a powerful tool for generalization, while curriculum-based and adaptive schedules like MAB offer superior efficiency and stability.

Future research will focus on "discriminative feature" detection—automatically identifying which morphological parameters (e.g., leg length vs. width) have the greatest impact on performance to prioritize variability in those dimensions. Additionally, the authors plan to explore more complex meta-learning techniques and validate these evolved generalist controllers on physical robotic platforms to further close the reality gap.

Summary updated

BibTeX
@inproceedings{barba2024effect,
  author = {Edoardo Barba and Anil Yaman and Giovanni Iacca},
  title = {{The Effect of Training Schedules on Morphological Robustness and Generalization}},
  booktitle = {GECCO Companion},
  year = {2024},
  url = {https://openreview.net/forum?id=aovUoPSxLJ},
}
Figure