Abstract: Imitation learning (IL) has shown great potential in various applications, such as robot control. However, traditional IL methods are usually designed to learn only one specific type of behavior since demonstrations typically correspond to a single expert. In this work, we introduce the first generic framework for Quality Diversity Imitation Learning (QD-IL), which enables the agent to learn a broad range of skills from limited demonstrations. Our framework integrates the principles of quality diversity with adversarial imitation learning (AIL) methods, and can potentially improve any inverse reinforcement learning (IRL) method. Empirically, our framework significantly improves the QD performance of GAIL and VAIL on the challenging continuous control tasks derived from Mujoco environments. Moreover, our method even achieves 2x expert performance in the most challenging Humanoid environment.
Introduction
Standard Imitation Learning (IL) typically functions as a "copycat," attempting to mimic an expert's exact actions. While effective for specific tasks, this approach often fails when a robot encounters situations not covered by the expert data—for example, if a robot only knows one way to walk and its leg is damaged, it cannot adapt.
Quality Diversity (QD) optimization aims to solve this by finding an archive of diverse, high-performing solutions. However, combining QD with Inverse Reinforcement Learning (IRL)—where the agent must infer the reward function from demonstrations—is notoriously difficult. Traditional IRL methods tend to overfit to specific expert patterns, getting stuck in local optima and failing to explore the full range of possible useful behaviors.
Methodology: QD-IRL and Extrinsic Behavioral Curiosity
The authors introduce QD-IRL, a framework that integrates QD optimization with adversarial imitation methods such as GAIL, VAIL, and DiffAIL. The system utilizes Proximal Policy Gradient Arborescence (PPGA) as its underlying QD algorithm.
The core innovation is Extrinsic Behavioral Curiosity (EBC). The system maintains a behavioral archive (a grid) based on specific traits, such as foot-contact frequency. If an agent's trajectory ends in a previously empty archive cell, it receives an episode-level extrinsic reward bonus. Unlike previous curiosity models that reward novel states at every step—which can lead to erratic movement—EBC rewards the outcome of the entire episode, encouraging the discovery of functional, novel behaviors.
Experimental Setup
The framework was evaluated on three standard MuJoCo robot locomotion tasks: HalfCheetah, Walker2d, and Humanoid. To simulate a limited-data scenario, agents were provided with only four diverse expert demonstrations per environment, selected from a pool of 500 high-performance policies to ensure they represented distinct behavioral strategies.
Key Results and Insights
EBC significantly improved the QD-Score of various IRL methods, with gains of up to 185% in the Humanoid environment and 71% in Walker2d. In many tests, EBC-enhanced models reached nearly 100% coverage of the defined behavior space, whereas standard methods often explored less than half.
Several key insights emerged from the study:
- Breaking Overfitting: EBC acts as a nudge, forcing the optimizer to explore empty regions of the behavior space rather than strictly adhering to the expert's path.
- Surpassing the Expert: Agents often achieved higher QD-Scores than the original experts. This suggests that the IRL-learned reward function can be smoother and easier to optimize than the environment's true sparse reward.
- Robustness: The diverse library of learned skills allowed for few-shot adaptation. For instance, if a motor in a Humanoid's knee failed, the agent could quickly switch to a different walking style from its archive that did not rely on that joint.
Conclusion
While the framework faces challenges such as dynamic reward instability and a dependency on Markov Decision Process (MDP) structures, QD-IRL represents a major advancement in imitation learning. By rewarding behavioral curiosity at the episode level, robots can move beyond simple mimicry to discover a robust repertoire of functional skills from minimal human guidance.
Summary updated