Relevance-guided unsupervised discovery of abilities with quality-diversity algorithms

Luca Grillotti, Antoine Cully

TL;DR

This paper introduces RUDA, a novel Quality-Diversity algorithm that autonomously discovers relevant robot behaviors without requiring explicit task knowledge. RUDA achieves this by dynamically adjusting its search to prioritize behaviors that are useful for a given downstream task, leading to more efficient and effective skill discovery.

What Problem Does This Paper Address?

Robotics research has made significant strides in enabling robots to learn and adapt. However, many learning algorithms still require substantial human engineering, such as defining reward functions or specifying desired behaviors. Quality-Diversity (QD) algorithms offer a way to generate diverse and high-performing solutions, but they typically rely on hand-coded "behavioral descriptors" (BDs) to characterize these behaviors. This reliance on prior knowledge limits their applicability in scenarios where the robot's capabilities or the task are unknown. While some methods like AURORA attempt unsupervised BD discovery, they aim for broad coverage rather than task-specific relevance, potentially wasting valuable repertoire space on irrelevant behaviors. This paper addresses the challenge of discovering robot abilities that are not only diverse but also directly useful for a specific, potentially unknown, downstream task.

Methods

The core contribution is Relevance-guided Unsupervised Discovery of Abilities (RUDA), an extension of the AURORA algorithm. RUDA builds upon AURORA's unsupervised BD discovery by introducing a relevance-guided mechanism:

  • Unsupervised BD Discovery: Like AURORA, RUDA uses dimensionality reduction techniques (specifically, a reconstruction-based auto-encoder) to learn low-dimensional behavioral descriptors from high-dimensional sensory data. This allows the algorithm to discover behaviors without predefined metrics.
  • Task Solver: A crucial new component is the "Task Solver." This module takes the current set of discovered behaviors and identifies a subset that is most effective for solving a specific downstream task. The Task Solver can be implemented in various ways depending on the task (e.g., a path planner for navigation, or selecting individuals with maximum forward displacement).
  • Circular Buffer of Relevant Individuals: The behaviors identified by the Task Solver are stored in a circular buffer. This buffer acts as a memory of useful behaviors, preventing outdated ones from dominating the search.
  • Relevance-Based Distance Metric: RUDA modifies the distance metric used within the QD algorithm. This new metric is "distorted" to favor individuals that are closer to the relevant behaviors stored in the buffer. This effectively biases the search towards regions of the BD space that are more likely to contain useful solutions for the downstream task.

Datasets

The paper uses a simulated hexapod robot environment. The experiments involve three specific downstream tasks: Navigation, Moving Forward at high velocity, and performing Half-rolls. The sensory data collected from the robot includes joint positions, torso positions, and orientations.

Key Insights and Ideas

  • Task-Relevance is Key: The most significant insight is that guiding the search towards task-relevant behaviors, even in an unsupervised setting, leads to more effective skill discovery than simply aiming for broad behavioral coverage.
  • Unsupervised BDs Can Be Task-Adapted: RUDA demonstrates that unsupervised learning of behavioral descriptors can be effectively combined with task-specific guidance, bridging the gap between general exploration and targeted skill acquisition.
  • Balancing Specialization and Diversity: While RUDA specializes the repertoire for a given task, it still maintains a degree of diversity within that specialized space, allowing for robust performance. The paper notes that specialization inherently leads to some loss of broad diversity, which is an expected trade-off.
  • Flexibility in Task Solver Design: The Task Solver can be adapted to various tasks, highlighting the algorithm's potential for broad applicability.

Key Results

The experimental results on the simulated hexapod robot show that RUDA consistently outperforms other variants, including AURORA and hand-coded BD approaches, in terms of task performance (Task Score) and relevance to the downstream task (Container Score).

  • Moving Forward & Half-roll Tasks: RUDA and its variant R-MeS (which also uses relevance but with a hand-coded high-dimensional BD) showed significantly higher container scores compared to non-relevance-based methods. RUDA, with its automatically learned BD, further improved upon R-MeS, suggesting the benefit of unsupervised dimensionality reduction.
  • Navigation Task: RUDA achieved significantly better task scores than AURORA, demonstrating its ability to find more efficient solutions. While hand-coded BDs performed best here, RUDA's performance was still superior to unsupervised methods without relevance guidance.
  • Container Diversity: While RUDA's specialization reduces overall diversity compared to purely exploratory methods like AURORA and MeS, it still exhibits better diversity within the specialized regions of the BD space, particularly when considering minimum fitness thresholds.

Limitations and Future Work

  • Loss of Broad Diversity: The paper acknowledges that specializing for a downstream task inherently reduces the overall behavioral diversity compared to algorithms that aim for broad coverage.
  • Future Work: The authors suggest exploring RUDA on more complex multi-task problems and investigating whether other methods for distorting the BD space will exhibit similar trade-offs between specialization and diversity. They also propose applying the approach to more complex robotic problems.

Summary updated

Abstract

Quality-Diversity algorithms provide efficient mechanisms to generate large collections of diverse and high-performing solutions, which have shown to be instrumental for solving downstream tasks. However, most of those algorithms rely on a behavioural descriptor to characterise the diversity that is hand-coded, hence requiring prior knowledge about the considered tasks. In this work, we introduce Relevance-guided Unsupervised Discovery of Abilities; a Quality-Diversity algorithm that autonomously finds a behavioural characterisation tailored to the task at hand. In particular, our method introduces a custom diversity metric that leads to higher densities of solutions near the areas of interest in the learnt behavioural descriptor space. We evaluate our approach on a simulated robotic environment, where the robot has to autonomously discover its abilities based on its full sensory data. We evaluated the algorithms on three tasks: navigation to random targets, moving forward with a high velocity, and performing half-rolls. The experimental results show that our method manages to discover collections of solutions that are not only diverse, but also well-adapted to the considered downstream task.
BibTeX
@inproceedings{grillotti2022relevance,
  author = {Luca Grillotti and Antoine Cully},
  title = {{Relevance-guided unsupervised discovery of abilities with quality-diversity algorithms}},
  booktitle = {GECCO},
  year = {2022},
  url = {https://openreview.net/forum?id=Kc31RqZOav},
}
Figure