Online Damage Recovery for Physical Robots with Hierarchical Quality-Diversity

Maxime Allard, Simón C. Smith, Konstantinos I. Chatzilygeroudis, Bryan Lim, Antoine Cully

TL;DR

This paper introduces the Hierarchical Trial and Error (HTE) algorithm, which uses a layered approach to robot skill learning to enable faster and more robust adaptation to physical damages. By breaking down complex behaviors into hierarchical sub-skills, HTE significantly reduces the number of actions and failures required for robots to recover from damage and complete tasks, both in simulation and in the real world.

What Problem Does This Paper Address?

Deploying robots in real-world environments presents a significant challenge due to unforeseen situations, particularly physical damages that can drastically alter a robot's performance. While Quality-Diversity (QD) algorithms have shown promise in learning diverse skills for adaptation, scaling these methods to handle a large number of skills and complex tasks leads to increased computational complexity and difficulties in planning. Existing approaches often struggle to efficiently manage and select from a vast repertoire of skills, especially when dealing with novel scenarios like unexpected damage. This paper addresses the need for a more efficient and effective method for robots to learn and adapt to damages online, without prior training for such events.

Methods

The core innovation of this paper is the Hierarchical Trial and Error (HTE) algorithm, which builds upon the Quality-Diversity (QD) framework and introduces a Hierarchical Behavioural Repertoire (HBR).

  • Hierarchical Behavioural Repertoires (HBRs): Instead of a single, flat repertoire of skills, HTE uses a layered structure. Each layer learns a set of behaviors that serve as building blocks for the layer above. For instance, the lowest layer might control individual leg movements, the middle layer combines these to make the robot walk for a short duration, and the top layer chains these walking behaviors for longer durations or more complex tasks. This hierarchical decomposition allows for compositionality and re-usability of skills.
  • Primary and Secondary Behavioural Descriptors: To further enhance diversity and adaptation, HTE introduces two types of descriptors:
  • Primary Descriptors: These define the core task-relevant behaviors (e.g., x, y displacement of the robot).
  • Secondary Descriptors: These add an extra layer of diversity by modulating how a primary behavior is executed (e.g., leg contact with the ground). These are not used during training but are crucial for online adaptation.
  • Trial and Error Adaptation: During deployment, when a damage occurs, HTE uses a trial-and-error process guided by Bayesian Optimization with Gaussian Processes (GPs). This process learns to predict the error associated with different secondary behaviors and selects the most effective ones to compensate for the damage, allowing the robot to adapt its strategy in real-time.
  • Planning: For task execution (like maze navigation), HTE employs Monte Carlo Tree Search (MCTS) to plan actions, leveraging the hierarchical repertoire.

Datasets

The paper does not explicitly mention or introduce new datasets. Instead, it relies on simulated environments (using the RobotDART simulator) and real-world experiments with a physical hexapod robot. The "datasets" are effectively the experimental scenarios created, including various damage configurations in a maze navigation task.

Key Insights and Ideas

  • Hierarchy as a Solution to Complexity: Decomposing complex skills into a hierarchy significantly simplifies the search and storage problem inherent in large QD repertoires. This allows for more efficient exploration and optimization.
  • Secondary Descriptors for Fine-Grained Adaptation: The introduction of secondary behavioral descriptors, used only during adaptation, provides a powerful mechanism to modulate high-level skills without increasing the training complexity of the hierarchical layers. This allows for more nuanced recovery from damage.
  • Trial-and-Error for Online Adaptation: The online trial-and-error selection of secondary behaviors, guided by GPs, is crucial for adapting to unforeseen situations like damage, especially when prior knowledge is limited.
  • Simulation-to-Reality Transfer is Enhanced: While simulation-to-reality gaps are a known challenge, HTE's adaptive capabilities, particularly the online selection of secondary behaviors, help bridge this gap by allowing the robot to fine-tune its actions in the physical world.

Key Results

  • Reduced Actions and Failures: In simulation, HTE required 14% fewer actions than the best baseline and had significantly fewer complete failures (78% less). In the physical world, HTE achieved a 43% reduction in actions compared to the best baseline.
  • Faster Adaptation: HTE demonstrated faster adaptation to damages in both simulation and the physical world compared to baseline methods.
  • Efficient Planning: Despite offering more diversity, HTE's hierarchical structure allowed for planning times comparable to simpler, non-hierarchical methods, making it suitable for real-time deployment.
  • Effective Damage Recovery: HTE successfully adapted to various leg damages in a maze navigation task, both in simulation and on a physical hexapod robot, showcasing its robustness.

Limitations and Future Work

  • Task Expertise for Hierarchy Design: The paper acknowledges that defining the genotypes, behavioral descriptors, and fitness functions for each layer of the hierarchy requires task expertise. This can limit the generality of the approach without careful design.
  • Improvement in Skill Selection: The "Perfect HTE" variant, where secondary behaviors were manually selected based on prior knowledge of the damage, outperformed the automatically adapting HTE. This suggests room for improvement in the online skill selection mechanism.
  • Future Work: The authors suggest exploring more sophisticated skill selection mechanisms and investigating how to further reduce the reliance on task expertise for designing hierarchical structures. They also highlight the potential for applying HTE to a wider range of robots and tasks.

Summary updated

Abstract

In real-world environments, robots need to be resilient to damages and robust to unforeseen scenarios. Quality-Diversity (QD) algorithms have been successfully used to make robots adapt to damages in seconds by leveraging a diverse set of learned skills. A high diversity of skills increases the chances of a robot to succeed at overcoming new situations since there are more potential alternatives to solve a new task.However, finding and storing a large behavioural diversity of multiple skills often leads to an increase in computational complexity. Furthermore, robot planning in a large skill space is an additional challenge that arises with an increased number of skills. Hierarchical structures can help reducing this search and storage complexity by breaking down skills into primitive skills. In this paper, we introduce the Hierarchical Trial and Error algorithm, which uses a hierarchical behavioural repertoire to learn diverse skills and leverages them to make the robot adapt quickly in the physical world. We show that the hierarchical decomposition of skills enables the robot to learn more complex behaviours while keeping the learning of the repertoire tractable. Experiments with a hexapod robot show that our method solves a maze navigation tasks with 20% less actions in simulation, and 43% less actions in the physical world, for the most challenging scenarios than the best baselines while having 78% less complete failures.
BibTeX
@article{allard2023online,
  author = {Maxime Allard and Simón C. Smith and Konstantinos I. Chatzilygeroudis and Bryan Lim and Antoine Cully},
  title = {{Online Damage Recovery for Physical Robots with Hierarchical Quality-Diversity}},
  journal = {ACM Trans. Evol. Learn. Optim.},
  year = {2023},
  url = {https://openreview.net/forum?id=hB5BoYRGr1},
}
Figure