Unlabeled Imperfect Demonstrations in Adversarial Imitation Learning

Yunke Wang, Bo Du, Chang Xu

Abstract

Adversarial Imitation Learning (AIL) typically assumes access to perfect expert demonstrations. However, real-world datasets often contain a mixture of optimal and suboptimal behaviors. We present Unlabeled Imperfect Demonstrations (UID), a framework that leverages Positive-Unlabeled (PU) learning to distinguish high-quality demonstrations within a noisy pool. By treating the agent's own trajectories as the positive class and the expert pool as unlabeled data, UID creates a self-paced learning curriculum that focuses on optimal behaviors. Experiments on MuJoCo and RoboSuite demonstrate that UID significantly outperforms existing methods in high-noise regimes and handles real-world human data effectively.

Introduction

Adversarial Imitation Learning (AIL), exemplified by frameworks like GAIL (Generative Adversarial Imitation Learning), has become a standard approach for teaching agents complex behaviors. These methods train a discriminator to distinguish between an agent's behavior and an expert's behavior, with the agent learning to "fool" the discriminator.

A critical limitation of standard AIL is the assumption of expert optimality. In practice, collecting purely optimal data is often prohibitively expensive or impossible. Most real-world datasets are "imperfect," containing a few high-quality examples mixed with suboptimal or failing attempts. Standard AIL applied to such data often results in the agent mimicking the average, mediocre behavior. Previous solutions attempted to assign "confidence scores" to demonstrations, but these often require manual labeling or fragile heuristics.

Methodology: The UID Framework

The authors propose UID, which treats the entire set of expert demonstrations as unlabeled data rather than a ground-truth positive set.

Positive-Unlabeled (PU) Learning

UID shifts the classification paradigm. In traditional AIL, the expert is "positive" and the agent is "negative." UID utilizes PU learning, where one has a set of "Positive" examples and a large pool of "Unlabeled" data containing both positives and negatives.

  1. The Agent as Anchor: The agent's current trajectories are treated as the "Positive" class (representing the current, often suboptimal, behavior).
  2. The Expert Pool as Unlabeled: The expert demonstrations are treated as the "Unlabeled" pool.
  3. Mixture Modeling: The expert distribution is modeled as a mixture of "matched" data (similar to the current agent) and "residual" data (the high-quality parts the agent has yet to master).

Self-Paced Learning Curriculum

By employing a PU-based risk estimator for the discriminator, UID creates a natural curriculum. Early in training, the agent's poor performance causes the discriminator to label similar "bad" parts of the expert data as "matched." As the agent improves, it matches higher-quality segments of the expert data, forcing the discriminator to focus on the remaining, more optimal demonstrations. This mechanism allows the agent to eventually recover the true expert distribution, a property the authors support with a theoretical proof of Nash Equilibrium.

Experimental Evaluation

The framework was evaluated on two distinct platforms:

  • MuJoCo: Physics-based locomotion tasks (Ant-v2, HalfCheetah-v2, Walker2d-v2) with synthetic noise.
  • RoboSuite: A "Nut Assembly" task using real-world human demonstrations from the RoboTurk dataset, which are naturally imperfect.
Visualizations of the HalfCheetah and Walker2d locomotion tasks from the MuJoCo suite.

Results and Discussion

UID demonstrates significant robustness to noise. In the Ant-v2 task, as the ratio of optimal data decreased from 50% to 16.7%, UID's performance remained stable, while competing methods like WGAIL and BCND collapsed.

Performance of UID across varying ratios of optimal demonstrations (left) and sensitivity to the alpha hyperparameter (right).

As shown in Figure 1 (left), UID (red line) maintains a lower average distance to the expert even as the ratio of optimal demonstrations becomes highly skewed (e.g., 1:5). Furthermore, Figure 1 (right) illustrates the influence of the $\alpha$ hyperparameter (the mixing proportion). While UID is relatively robust, performance does vary with $\alpha$, suggesting that estimating the true ratio of optimal data remains a key challenge for fully autonomous deployment.

On the RoboSuite task, UID achieved the highest average reward, outperforming algorithms that rely on manual confidence scores or heuristics. This confirms UID's ability to handle the unpredictable noise inherent in human-generated data.

Conclusion

UID provides a robust and flexible framework for imitation learning from imperfect, unlabeled data. By reframing the problem through PU learning and using the agent as a moving anchor, it avoids the need for manual labeling or complex heuristics. Future work may focus on automating the estimation of the mixing parameter $\alpha$ to further enhance the framework's utility in diverse real-world scenarios.

Summary updated

BibTeX
@inproceedings{wang2023unlabeled,
  author = {Yunke Wang and Bo Du and Chang Xu},
  title = {{Unlabeled Imperfect Demonstrations in Adversarial Imitation Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i8.26222},
}
Figure