TL;DR
Standard imitation learning methods replicate only a narrow set of expert behaviors, struggling when faced with novel situations or dynamic environments. This paper introduces Quality Diversity Imitation Learning (QD-IL) with a framework called MConbo-IRL, which combines adversarial imitation learning with quality-diversity algorithms to discover a wide archive of high-performing, diverse skills from as few as 1 to 4 expert demonstrations.
What Problem Does This Paper Address?
Traditional Inverse Reinforcement Learning (IRL) and imitation learning (IL) aim to mimic a single expert policy. This leads to two major shortcomings:
- Unbalanced exploration and exploitation: Policy space contains numerous local optima, trapping agents in small regions of behavioral space.
- Localized reward functions: Because IL algorithms are trained against a narrow demonstration set, they assign high rewards only to behaviors that tightly mimic the demonstrated trajectories. When faced with stochastic environments, robot damage, or unseen conditions, these policies cannot adapt.
Quality Diversity (QD) algorithms (such as MAP-Elites) solve this by populating an archive of diverse behaviors defined across behavioral measure dimensions (e.g., how long each leg touches the ground during locomotion). However, QD methods typically require access to an environment's true reward function. This paper tackles how to train a diverse archive of high-performing policies without access to the true reward, learning purely from a tiny set of demonstrations.
Methods
The authors propose MConbo-IRL (Measure-Conditioned and Bonus-Driven IRL), built on top of the state-of-the-art QD reinforcement learning algorithm PPGA (Proximal Policy Gradient Arborescence):
- Measure Bonus ($r_{\text{diversity}}$): An episodic exploration bonus $r_{\text{diversity}} = p + q \cdot \mathbb{I}(m_i \in \mathcal{A}_e)$ awarded when an episode lands in an unoccupied cell $\mathcal{A}_e$ of the behavior archive $\mathcal{A}$. This prevents the evolutionary search policy from stalling in local optima and dynamically shifts attention toward under-explored behaviors.
- Measure Conditioning: Rather than feeding only state-action pairs $(s, a)$ to the discriminator, the authors condition the discriminator on a Markovian Measure Proxy $\delta(s)$—a single-step abstraction of the behavioral measure (e.g., ground contact at time $t$). This allows the discriminator to reward behaviors based on high-level task features rather than memorizing low-level trajectory mechanics, facilitating generalization across behavior space.
- Algorithm Flexibility: The authors instantiate this with two standard adversarial IL backbones: MConbo-GAIL and MConbo-VAIL (which uses a Variational Information Bottleneck), as well as an observation-only variant (QD-IFO) that requires no expert action labels.
Datasets
The authors evaluate their approach on three standard continuous-control robot locomotion benchmarks from MuJoCo (simulated via Brax):
- Halfcheetah (17-dim observation, 6-dim action)
- Walker2d (18-dim observation, 6-dim action)
- Humanoid (227-dim observation, 17-dim action)
Demonstrations: Expert demonstrations were generated using an archive previously trained with PPGA using the true reward. From the top 500 elite policies, 4 diverse demonstrations were selected per environment to mimic realistic, sparse demonstration regimes. Scalability experiments also evaluated regimes with 1, 2, and 10 demonstrations.
Key Insights and Ideas
- Measure abstraction prevents discriminator overfitting: Low-level state representations cause adversarial discriminators to overfit quickly on tiny demonstration sets. By conditioning the discriminator on $\delta(s)$, the model abstracts out irrelevant joint angles/velocities and learns generalizable behavioral principles.
- Surpassing the expert via Quality Diversity: Standard IL is fundamentally bounded by expert demonstration quality. By forcing behavioral exploration via the measure bonus, MConbo-GAIL discovered regions of the behavior space that the original expert (trained via true-reward PPGA) completely missed—enabling it to dramatically outperform the expert archive.
- Feasibility of Imitation From Observation (IFO): Removing action sequences from the demonstrations entirely (MConbo-GAIL-Obs) caused virtually no drop in QD performance, demonstrating that behavioral measures provide enough structural signal to infer diverse policies from state observations alone.
Key Results
Experiments evaluated archives across four metrics using the ground-truth environment reward: QD-Score (total archive performance), Coverage (% of archive cells filled), Best Reward, and Average Reward:
- Humanoid Environment:
- True Reward Expert: QD-Score: $5.71 \times 10^6$ | Coverage: $49.96\%$ | Avg Reward: $4,570$
- Generic GAIL: QD-Score: $1.86 \times 10^6$ | Coverage: $82.36\%$ | Avg Reward: $924$
- MConbo-GAIL: QD-Score: $\mathbf{8.47 \times 10^6}$ (~$1.5\times$ the expert, $>4.5\times$ GAIL) | Coverage: $\mathbf{93.47\%}$ | Avg Reward: $3,618$
- MConbo-VAIL: QD-Score: $\mathbf{8.91 \times 10^6}$ | Coverage: $\mathbf{91.52\%}$ | Avg Reward: $3,899$
- Walker2d: MConbo-GAIL achieved a QD-Score of $4.12 \times 10^6$ and $91.69\%$ coverage, surpassing the expert ($3.64 \times 10^6$ QD-Score, $77.04\%$ coverage) and baseline GAIL ($2.48 \times 10^6$ QD-Score).
- Extreme Few-Shot Robustness: On Walker2d, reducing demonstrations down to a single demonstration still yielded nearly $100\%$ behavior space coverage and soundly outperformed GAIL.
Limitations and Future Work
- Dynamic Reward Shift in MAP-Elites: MAP-Elites assumes an objective function with static evaluation. Because adversarial discriminators change throughout training, elites stored early in training were evaluated on a different reward scale than later candidates.
- Tuning Hyperparameters: The balance between staying alive ($p$) and pursuing diversity bonuses ($q$) was kept fixed ($p=q=0.5$), but hyperparameter sweeps show that tuning them could yield further gains.
- Future Directions: Integrating MConbo into additional IRL backbones, exploring context-conditioned meta-policies for rapid online behavioral adaptation, and deeper exploration of Quality-Diversity Imitation From Observation (QD-IFO).
Summary updated