Introduction
Imitation Learning (IL) has emerged as a powerful paradigm for training agents to perform complex tasks by mimicking expert behavior. However, a significant bottleneck in standard IL frameworks, such as Generative Adversarial Imitation Learning (GAIL) and Adversarial Inverse Reinforcement Learning (AIRL), is the assumption that all provided demonstrations originate from a perfect expert. In practical scenarios, collecting high-quality expert data is often prohibitively expensive or time-consuming. Consequently, practitioners frequently rely on "diverse-quality" datasets—mixtures of expert demonstrations and amateurish, noisy trajectories sourced from crowdsourcing or sub-optimal automated agents.
When confronted with such heterogeneous data, traditional IL methods tend to fail. By assuming uniform expertise, these algorithms attempt to match the average distribution of the entire dataset, effectively "averaging out" the expert's successes with the amateur's mistakes. This leads to mediocre policies that lack the precision of the expert. To address this, we present VILD (Variational Imitation Learning with Diverse-quality demonstrations), a framework designed to robustly learn from mixed-quality data without requiring explicit labels for demonstrator expertise.
Methodology
VILD reformulates the imitation learning problem by treating the expertise level of each demonstrator as a latent variable. The core idea is to model the observed actions as noisy versions of an underlying optimal policy, where the variance of the noise is determined by the demonstrator's specific expertise.
The framework incorporates several key components:
- Probabilistic Latent Modeling: By introducing a latent variable $k$ representing the demonstrator, VILD can mathematically distinguish between high-variance (amateur) and low-variance (expert) behaviors.
- Variational Inference: To handle high-dimensional state and action spaces typical in robotics, the authors employ a variational approach. This transforms the complex posterior estimation into a tractable optimization problem compatible with modern Reinforcement Learning (RL) algorithms like TRPO and SAC.
- Reward Learning and Importance Sampling: Unlike simple behavior cloning, VILD learns a robust reward function that captures the underlying intent of the task. Furthermore, an Importance Sampling (IS) mechanism is introduced to prioritize learning from trajectories that the model identifies as more "expert-like," significantly enhancing data efficiency.
Experimental Evaluation
The performance of VILD was benchmarked against state-of-the-art baselines (GAIL, AIRL, InfoGAIL, VAIL, and MaxEnt-IRL) using four continuous-control tasks from the MuJoCo physics engine: HalfCheetah, Ant, Walker2d, and Humanoid. To simulate realistic diverse-quality conditions, datasets were generated using ten distinct demonstrators with varying noise profiles, including standard Gaussian noise and Time-Signal-Dependent (TSD) noise, which better reflects human motor errors.
Key Results
VILD consistently outperformed all baseline methods across the evaluated benchmarks. The ability to filter out amateur noise allowed VILD to achieve near-expert performance even when the majority of the data was sub-optimal.
As shown in the results, VILD with Importance Sampling (red line) demonstrates superior learning speed and final performance. For instance, in the HalfCheetah task with Gaussian noise, VILD reached a cumulative reward of approximately 4559, whereas GAIL struggled to exceed 551. The model also successfully learned to rank demonstrators by their latent noise levels, effectively identifying the true experts within the mixed dataset.
Conclusion
VILD provides a robust solution for imitation learning in the presence of diverse-quality data. By explicitly modeling latent expertise, it avoids the pitfalls of behavioral averaging that plague traditional IL methods. While the current implementation relies on a Gaussian noise assumption, its strong performance on TSD noise suggests a degree of robustness to model misspecification. Future work will focus on exploring more flexible noise models and extending the framework to real-world robotic systems where data quality is inherently varied.
Summary updated