Unsupervised Behaviour Discovery with Quality-Diversity Optimisation

Luca Grillotti, Antoine Cully

TL;DR

This paper introduces AURORA, a novel Quality-Diversity (QD) algorithm that automatically learns behavioral descriptors from raw sensory data, eliminating the need for manual feature engineering. AURORA demonstrates comparable performance to traditional QD methods while discovering more diverse behaviors, particularly when sensory data contains richer information than hand-coded descriptors.

What Problem Does This Paper Address?

Robots are increasingly deployed in complex and unpredictable environments, requiring them to exhibit versatility and resilience. A key challenge in achieving this is enabling robots to autonomously discover a wide range of behaviors without explicit human programming for every possible scenario. Quality-Diversity (QD) algorithms are a promising approach, aiming to generate a collection of diverse and high-performing solutions (a "behavioral repertoire"). However, a significant bottleneck in existing QD methods is the reliance on hand-coded "behavioral descriptors" (BDs). These BDs, which characterize a behavior's novelty, require prior knowledge of the task and can inadvertently constrain the diversity of discovered behaviors. This paper addresses the problem of automatically learning these crucial BDs from raw, high-dimensional sensory data, thereby removing the need for manual feature engineering and potentially unlocking richer behavioral discovery.

Methods

The core innovation of this paper is the AURORA (Autonomous Robots Realising their Abilities) algorithm. AURORA operates in an alternating fashion between two phases:

  1. Quality-Diversity (QD) Phase: This phase is similar to standard QD algorithms. Individuals (controllers) are selected, mutated, and evaluated. Crucially, instead of returning a hand-coded BD, the evaluation returns raw sensory data collected by the robot.
  2. Encoder Update Phase: This phase refines the "encoder," a dimensionality reduction technique (like a Convolutional Auto-Encoder or MLP) that maps the high-dimensional sensory data into a low-dimensional, unsupervised BD. The encoder is trained using the sensory data from all individuals currently in the QD container. This process is iterative and online, meaning the encoder is continuously updated and never reset.

This approach allows AURORA to learn BDs that capture features present in the raw sensory data, which may be more informative than hand-coded BDs. The paper also introduces AURORA-CSC (Container Size Control), a variant designed to manage the container size effectively, especially in high-dimensional BD spaces, by adaptively adjusting the novelty threshold.

Datasets

The paper evaluates AURORA on three simulated robotic tasks:

  • Maze Task: A simulated circular robot navigating a maze. Sensory data is a 64x64 RGB image of the maze. The hand-coded BD is the robot's final (x,y) position.
  • Hexapod Task: A hexapod robot learning to reach diverse positions. Sensory data is a 64x64 RGB top-down image of the environment. The hand-coded BD is the robot's final torso position.
  • Air-Hockey Task: A robotic arm learning to push a puck to various locations. Sensory data is the trajectory of the puck (a sequence of (x,y) positions). The hand-coded BD is the final puck position.

These tasks are chosen to represent varying levels of difficulty and discrepancy between the rich sensory data and the simpler hand-coded BDs.

Key Insights and Ideas

  • Unsupervised BDs are Viable: The paper demonstrates that QD algorithms can effectively learn and utilize automatically generated, unsupervised behavioral descriptors, achieving performance comparable to methods relying on hand-coded features.
  • Richer Sensory Data Unlocks Deeper Diversity: When the raw sensory data contains more information than the hand-coded BD (e.g., full puck trajectories vs. just the final position), AURORA can discover behaviors that are diverse across more features, going beyond what the hand-coded BD captures.
  • Incremental Learning is Key: The online, incremental training of the encoder is crucial for AURORA to adapt to the evolving BD space and maintain meaningful descriptors throughout the optimization process.
  • Container Size Management is Critical: The introduction of AURORA-CSC highlights the importance of robust methods for managing the container size in QD, especially when dealing with potentially unstable or high-dimensional BD spaces.

Key Results

  • Comparable Performance: AURORA-CSC variants achieved coverage in the hand-coded BD space comparable to or exceeding other QD algorithms, including those using hand-coded BDs.
  • Enhanced Diversity: In tasks where sensory data was richer than hand-coded BDs (e.g., Air-Hockey), AURORA demonstrated significantly higher coverage and diversity scores, indicating it explored a broader range of behaviors. For instance, in the Air-Hockey task, AURORA achieved significantly higher coverage (p < 3 × 10−6).
  • Robustness to Dimensionality: AURORA-CSC proved more robust to the dimensionality of the learned BD space compared to the Volume Adaptive Threshold (VAT) variant, maintaining stable container size and coverage.
  • Selector Impact: The choice of selector (uniform, novelty-based, surprise-based) influenced AURORA's exploration efficiency, with novelty-based and surprise-based selectors often leading to higher coverage in the initial stages.

Limitations and Future Work

  • Encoder Resetting: The paper notes that in low-dimensional latent spaces (e.g., 2D), incremental encoder training can lead to "entanglement" where distinct behaviors in the latent space map to similar hand-coded BDs. Retraining the encoder from scratch improved coherence, suggesting future work could explore more sophisticated encoder update strategies.
  • Real-World and Complex Tasks: The current evaluation is limited to simulated locomotion tasks. The authors suggest extending AURORA to real-world scenarios, grasping tasks, and tasks with partial observability or non-resetting environments.
  • Advanced Encoding: For tasks involving temporal data (e.g., full trajectories), the authors propose exploring more advanced encoding techniques like LSTM-based Auto-Encoders.
  • Container Size Control Refinement: While CSC is effective, the authors suggest exploring more advanced control techniques (e.g., PID controllers) for managing container size and adapting to potential sudden changes in the BD space.

Summary updated

Abstract

Quality-Diversity algorithms refer to a class of evolutionary algorithms designed to find a collection of diverse and high-performing solutions to a given problem. In robotics, such algorithms can be used for generating a collection of controllers covering most of the possible behaviours of a robot. To do so, these algorithms associate a behavioural descriptor to each of these behaviours. Each behavioural descriptor is used for estimating the novelty of one behaviour compared to the others. In most existing algorithms, the behavioural descriptor needs to be hand-coded, thus requiring prior knowledge about the task to solve. In this paper, we introduce: Autonomous Robots Realising their Abilities, an algorithm that uses a dimensionality reduction technique to automatically learn behavioural descriptors based on raw sensory data. The performance of this algorithm is assessed on three robotic tasks in simulation. The experimental results show that it performs similarly to traditional hand-coded approaches without the requirement to provide any hand-coded behavioural descriptor. In the collection of diverse and high-performing solutions, it also manages to find behaviours that are novel with respect to more features than its hand-coded baselines. Finally, we introduce a variant of the algorithm which is robust to the dimensionality of the behavioural descriptor space.
BibTeX
@article{grillotti2022unsupervised,
  author = {Luca Grillotti and Antoine Cully},
  title = {{Unsupervised Behaviour Discovery with Quality-Diversity Optimisation}},
  journal = {IEEE Trans. Evol. Comput.},
  year = {2022},
  url = {https://openreview.net/forum?id=IFiDcMLSxA},
}
Figure