QDax: A Library for Quality-Diversity and Population-based Algorithms with Hardware Acceleration

Félix Chalumeau, Bryan Lim, Raphaël Boige, Maxime Allard, Luca Grillotti, Manon Flageat, Valentin Macé, Arthur Flajolet, Thomas Pierrot, Antoine Cully

Introduction

In traditional optimization, the objective is typically to identify a single global optimum. However, Quality-Diversity (QD) optimization shifts this focus toward discovering a diverse set of high-performing solutions. For instance, in robotics, a QD algorithm might generate a variety of gaits—ranging from fast and stable to energy-efficient—rather than just one.

Despite its potential, QD optimization is computationally intensive, often requiring millions of evaluations in complex simulations. Previous libraries forced researchers to choose between the performance of C++ and the ease of use of Python. QDax addresses this by leveraging JAX, a high-performance numerical computing library, to provide hardware-accelerated, modular, and user-friendly QD implementations.

Modular Architecture and Workflow

QDax unifies diverse research areas by decomposing algorithms into two primary building blocks:

  • Containers: These define how the population of solutions is organized and stored (e.g., Grids or Centroidal Voronoi Tessellations).
  • Emitters: These define the mechanisms for generating new solutions (e.g., Mutation, Evolution Strategies, or Policy Gradients).
The QDax framework: (A) Core components, (B) Quality-Diversity workflow, and (C) examples of modular algorithm construction.

This modularity allows researchers to easily experiment with hybrid approaches, such as using Reinforcement Learning (RL) policy gradients as emitters within a QD container. The library includes 16 meticulously crafted implementations, ranging from classic QD methods to unsupervised skill discovery.

Summary of algorithms implemented in QDax and their corresponding modular components.

Performance and Key Results

The primary advantage of QDax is its ability to run both the optimization algorithm and the environment simulation (e.g., Brax) on the same GPU or TPU. This eliminates the communication overhead typically associated with CPU-based evolutionary algorithms.

Key performance metrics include:

  • Massive Speedups: Runtimes are up to 10x faster than previous implementations, turning multi-day experiments into minutes.
  • High Throughput: On a single GPU, MAP-Elites achieved over 1.7 billion training steps in two hours on the "Ant Omni" task.
  • Efficiency: The PGA-MAP-Elites implementation performs $10^8$ steps per hour on one GPU, a significant improvement over original implementations that required dozens of CPUs and much longer runtimes.

Benchmarks and Versatility

QDax is evaluated across a variety of domains, including native JAX physics simulations like Brax (Ant, HalfCheetah, etc.) and specific QD-focused tasks like Ant-Trap. It also supports standard mathematical benchmarks and combinatorial optimization via the Jumanji suite, demonstrating versatility beyond robotics.

Limitations and Future Outlook

While QDax lowers the barrier for large-scale QD research, it requires familiarity with JAX’s functional programming paradigm. Furthermore, the maximum performance gains are realized when the environment itself is written in JAX to avoid CPU-GPU data transfer bottlenecks. Future work aims to apply this modular, high-performance framework to complex industrial decision-making problems requiring diverse solution sets.

Summary updated

BibTeX
@article{chalumeau2024qdax,
  author = {Félix Chalumeau and Bryan Lim and Raphaël Boige and Maxime Allard and Luca Grillotti and Manon Flageat and Valentin Macé and Arthur Flajolet and Thomas Pierrot and Antoine Cully},
  title = {{QDax: A Library for Quality-Diversity and Population-based Algorithms with Hardware Acceleration}},
  journal = {J. Mach. Learn. Res.},
  year = {2024},
  url = {https://openreview.net/forum?id=XPy8sdxG0V},
}
Figure