Abstract
The synergies between Quality-Diversity (QD) and Deep Reinforcement Learning (RL) have led to powerful hybrid QD-RL algorithms that have shown tremendous potential, and brings the best of both fields. However, only a single deep RL algorithm (TD3) has been used in prior hybrid methods despite notable progress made by other RL algorithms. Additionally, there are fundamental differences in the optimization procedures between QD and RL which would benefit from a more principled approach. We propose Generalized Actor-Critic QD-RL, a unified modular framework for actor-critic deep RL methods in the QD-RL setting. This framework provides a path to study insights from Deep RL in the QD-RL setting, which is an important and efficient way to make progress in QD-RL. We introduce two new algorithms, PGA-ME (SAC) and PGA-ME (DroQ) which apply recent advancements in Deep RL to the QD-RL setting, and solves the humanoid environment which was not possible using existing QD-RL algorithms. However, we also find that not all insights from Deep RL can be effectively translated to QD-RL. Critically, this work also demonstrates that the actor-critic models in QD-RL are generally insufficiently trained and performance gains can be achieved without any additional environment evaluations.
Introduction
Quality-Diversity (QD) algorithms, such as MAP-Elites, aim to discover a diverse archive of high-performing solutions rather than a single global optimum. Conversely, Deep Reinforcement Learning (RL) excels at fine-tuning individual policies through gradient-based optimization. Hybrid QD-RL algorithms attempt to combine these strengths, yet they have historically been limited to a narrow set of RL techniques, primarily TD3.
Despite significant advancements in the broader Deep RL field—such as Soft Actor-Critic (SAC) and DroQ—these innovations have not been systematically integrated into QD settings. Furthermore, fundamental differences in data handling and training schedules between QD and RL have lacked a formal reconciliation. This work addresses this "integration gap" by introducing a unified framework to translate RL advancements into diversity-seeking agents.
The Generalized Actor-Critic QD-RL Framework
To bridge the gap between these fields, we propose the Generalized Actor-Critic QD-RL framework. This modular architecture allows researchers to "plug and play" various off-policy Deep RL algorithms into a QD optimization loop.
The framework decomposes the hybrid system into several interchangeable components:
- Actor & Critic Trainers: These define the core RL logic. For instance, one can switch from the deterministic updates of TD3 to the maximum entropy approach of SAC.
- Data Organizer: This component manages the replay buffer. In QD-RL, this is particularly challenging as data is aggregated from a vast population of diverse policies rather than a single agent.
- Update Loops ($G$): This parameter controls the training intensity, specifically how many gradient updates are performed per generation of evolution.
Leveraging this framework, we introduce two novel algorithms: PGA-ME (SAC) and PGA-ME (DroQ).
Experimental Setup
The framework was evaluated using the Brax simulator, a high-performance physics engine, across three continuous control benchmarks:
- HalfCheetah: A standard 2D locomotion task.
- Ant: A 3D quadruped robot task requiring coordinated limb movement.
- Humanoid: A highly complex bipedal walking task characterized by high dimensionality and instability, making it notoriously difficult for traditional QD algorithms.
Key Insights and Results
The Impact of Training Intensity
A critical finding of this research is that actor-critic models in QD-RL have historically been significantly under-trained. While standard RL updates models every few environment steps, QD-RL hybrids often performed only 150 updates after over 100,000 new environment steps. We discovered that increasing the number of update loops ($G$) by orders of magnitude (e.g., from 150 to 25,000) leads to massive performance gains in both quality and diversity without requiring any additional environment simulations.
Solving the Humanoid Task
By integrating SAC—which encourages exploration through maximum entropy—our framework successfully solved the Humanoid task. Previous TD3-based QD-RL methods typically failed in this high-dimensional space due to insufficient exploration. PGA-ME (SAC) achieved fitness scores previously unreachable in the QD setting.
Translation of RL Techniques
Interestingly, not all Deep RL advancements translate directly to QD-RL. For example, regularization techniques like Dropout (used in DroQ) are essential in standard RL to prevent overfitting to a single agent's data. However, in QD-RL, the population-based approach naturally provides data diversity, and we found that some standard RL regularization methods can actually hinder performance in this context.
Discussion and Future Work
While increasing the training intensity ($G$) yields significant performance improvements without extra simulation data, it does increase computational overhead in terms of GPU time. Training with very high $G$ values can be time-intensive.
Future research will address the challenges posed by the stochastic nature of policies in algorithms like SAC, which can complicate reproducibility in QD archives. Additionally, further optimization of the Data Organizer is needed to better handle the distribution shifts inherent in population-based learning. This modular framework provides a principled path for future explorations at the intersection of Quality-Diversity and Reinforcement Learning.
Summary updated