Abstract
While standard approaches to optimisation focus on producing a single high-performing solution, Quality-Diversity (QD) algorithms allow large diverse collections of such solutions to be found. If QD has proven promising across a large variety of domains, it still struggles when faced with uncertain domains, where quantification of performance and diversity are non-deterministic. Previous work in Uncertain Quality-Diversity (UQD) has proposed methods and metrics designed for such uncertain domains. In this paper, we propose a first set of benchmark tasks to analyse and estimate the performance of UQD algorithms. We identify the key uncertainty properties to easily define UQD benchmark tasks: the uncertainty location, the type of distribution and its parameters. By varying the nature of those key UQD components, we introduce a set of 8 easy-to-implement and lightweight tasks, split into 3 main categories. All our tasks build on the Redundant Arm: a common QD environment that is lightweight and easily replicable. Each one of these tasks highlights one specific limitation that arises when considering UQD domains. With this first benchmark, we hope to facilitate later advances in UQD.
Introduction
Quality-Diversity (QD) algorithms, such as MAP-Elites, aim to discover a diverse collection of high-performing solutions. In robotics, for instance, a QD algorithm might identify numerous distinct joint configurations that allow a robot arm to reach a target point. However, standard QD algorithms typically assume a deterministic environment. In real-world scenarios, evaluations are often "uncertain" due to sensor noise or physical variability, meaning the same solution can yield different fitness scores or behavioral descriptors upon re-evaluation.
Without accounting for this stochasticity, standard QD algorithms frequently succumb to the "lucky outlier" trap, retaining solutions that performed well once by chance but are mediocre on average. This paper addresses the lack of standardized benchmarks to test how UQD algorithms handle these "illusory" results and prioritize reproducible solutions.
The UQD Framework and Benchmark Tasks
The proposed framework defines UQD tasks based on three fundamental properties:
- Uncertainty Location: Where noise is introduced—on the fitness score, the behavioral descriptor, or the "phenotype" parameters (e.g., joint angles).
- Distribution Type: The nature of the noise (e.g., Gaussian vs. Bimodal).
- Parameters: The scale or variance of the noise.
Based on these properties, eight benchmark tasks were developed, categorized into three primary goals: Performance Estimation, Reproducibility Maximization, and Realistic Combination (applying noise directly to the robot's phenotype).
The Redundant Arm Benchmark
The benchmarks utilize the Redundant Arm environment, a staple in QD research. It features a simulated planar robotic arm with multiple joints, where the goal is to reach as many points as possible in a 2D workspace. The "descriptor" is typically the (x, y) position of the arm's tip.
The authors have integrated these eight noisy variants into the QDax library, providing a standardized platform for UQD research.
Key Insights: The Challenge of Reproducibility
A critical finding is that standard MAP-Elites often creates an "Illusory Archive"—a map populated by solutions that appear high-performing but fail upon re-evaluation. While simple sampling and averaging can mitigate Gaussian noise, it can be detrimental in bimodal noise scenarios where rare, poor outcomes might be missed.
Furthermore, the paper emphasizes reproducibility as a key objective. In many applications, a reliable solution is preferable to a "jittery" one, even if their average performance is identical. Standard algorithms often fail to distinguish between these cases.
Results and Discussion
Evaluation of standard MAP-Elites and its sampling-based variant (MAP-Elites-sampling) revealed several key points:
- Descriptor Noise: Noise applied to the behavioral descriptor is particularly challenging, often leading to solutions being placed in incorrect niches.
- Bimodal Failure: MAP-Elites-sampling performed worse than the standard version in "Bimodal Fitness" tasks, as it was more easily misled by the complex noise distribution.
- Performance Gap: There is a significant quantitative gap between "Illusory" performance and "Corrected" performance (true performance upon re-evaluation), highlighting the need for specialized UQD algorithms.
Conclusion and Future Work
While the Redundant Arm provides a lightweight and accessible benchmark, its low dimensionality is a limitation. It may not fully capture the complexities of high-dimensional Deep RL tasks or dynamic noise environments. Future work should extend these uncertainty definitions to more complex simulators and focus on developing algorithms that explicitly maximize the "Reproducibility-Score" introduced in this study.
Summary updated