Adversarial Imitation Learning from Incomplete Demonstrations

Mingfei Sun, Xiaojuan Ma

TL;DR

This paper introduces AGAIL, a novel algorithm for imitation learning that can learn effective policies even when expert demonstrations are incomplete (i.e., missing action information). By separating state and action information and using a "guide" network to leverage available actions, AGAIL achieves performance comparable to state-of-the-art methods that require complete demonstrations.

What Problem Does This Paper Address?

Imitation learning aims to teach an agent a behavior policy by observing expert demonstrations. Traditionally, these demonstrations are assumed to be complete, meaning every state visited is paired with the exact action taken. However, in real-world scenarios, obtaining complete state-action trajectories is often difficult or impossible. For instance, a robot might observe a human's body movements (states) but not the precise forces or torques applied (actions). Existing imitation learning methods struggle with such incomplete data, either failing to learn or requiring extensive additional interaction. This paper addresses the critical gap of learning effectively from demonstrations where action information is partially or entirely missing.

Methods

The proposed algorithm, Action-Guided Adversarial Imitation Learning (AGAIL), builds upon the Generative Adversarial Imitation Learning (GAIL) framework. AGAIL introduces a novel "guide" network to complement the existing generator (policy) and discriminator.

  • Generator: Learns the agent's policy, aiming to mimic expert behavior.
  • Discriminator: Distinguishes between state-action pairs from expert demonstrations and those generated by the policy. This provides a reward signal to the generator.
  • Guide: This is the key innovation. It leverages available action information from demonstrations. It works by maximizing the mutual information between demonstrated actions and the policy's generated actions for the same states. This provides an auxiliary reward to the generator, guiding it when action data is present.

AGAIL separates demonstrations into state trajectories and action trajectories. The state trajectories are used for standard adversarial imitation (similar to GAIL), while the available action trajectories are used by the guide network. The algorithm combines these two components to train the policy.

Datasets

The paper evaluates AGAIL on four standard benchmark environments: CartPole, Hopper, Walker2d, and Humanoid. Demonstrations were collected by running a TRPO-trained policy and then artificially masking out actions to create varying levels of incompleteness (0%, 25%, 50%, 75%). The rewards in these environments are primarily state-dependent.

Key Insights and Ideas

  • Action information is valuable, even when partial: The paper demonstrates that even incomplete action sequences can significantly improve imitation learning performance, especially in complex tasks like Humanoid.
  • Separating state and action is key: By treating state and action information distinctly and using a guide network for actions, AGAIL effectively handles incomplete demonstrations.
  • Demonstration quality matters: The performance of AGAIL and other methods can be sensitive to the quality of the demonstrations. In some cases (e.g., Hopper), using more complete demonstrations did not necessarily lead to better results, suggesting that suboptimal or noisy demonstrations can be detrimental.
  • Robustness to incompleteness: AGAIL shows consistent performance across different levels of action incompleteness, highlighting its robustness.

Key Results

  • AGAIL achieves performance comparable to TRPO (which uses true rewards) and GAIL (which requires complete demonstrations) across various tasks.
  • AGAIL consistently outperforms state-only GAIL (which ignores actions entirely) in Walker2d and Humanoid environments.
  • In the Humanoid task, AGAIL with incomplete demonstrations achieved the highest returns, outperforming GAIL and TRPO.
  • The performance of AGAIL was robust to varying degrees of action incompleteness, often achieving similar or better results than GAIL even with significant action masking.

Limitations and Future Work

  • Demonstration quality sensitivity: The paper notes that the quality of demonstrations can impact performance, and in some environments (like Hopper), using more complete demonstrations did not improve results, suggesting that demonstrations sampled from unstable training checkpoints can be problematic.
  • Potential for policy collapse: While AGAIL aims to mitigate this, the paper acknowledges that forceful requirements for policy actions to share similar distributions can still lead to policy collapse, especially when dealing with noisy or suboptimal demonstrations.
  • Future work: The paper implicitly suggests exploring more sophisticated ways to handle demonstration quality and potentially developing methods that can actively identify and filter out low-quality demonstrations. Further investigation into the theoretical underpinnings of why more actions don't always help could also be a fruitful direction.

Summary updated

Abstract

Imitation learning targets deriving a mapping from states to actions, a.k.a. policy, from expert demonstrations. Existing methods for imitation learning typically require any actions in the demonstrations to be fully available, which is hard to ensure in real applications. Though algorithms for learning with unobservable actions have been proposed, they focus solely on state information and overlook the fact that the action sequence could still be partially available and provide useful information for policy deriving. In this paper, we propose a novel algorithm called Action-Guided Adversarial Imitation Learning (AGAIL) that learns a policy from demonstrations with incomplete action sequences, i.e., incomplete demonstrations. The core idea of AGAIL is to separate demonstrations into state and action trajectories, and train a policy with state trajectories while using actions as auxiliary information to guide the training whenever applicable. Built upon the Generative Adversarial Imitation Learning, AGAIL has three components: a generator, a discriminator, and a guide. The generator learns a policy with rewards provided by the discriminator, which tries to distinguish state distributions between demonstrations and samples generated by the policy. The guide provides additional rewards to the generator when demonstrated actions for specific states are available. We compare AGAIL to other methods on benchmark tasks and show that AGAIL consistently delivers comparable performance to the state-of-the-art methods even when the action sequence in demonstrations is only partially available.
BibTeX
@inproceedings{sun2019adversarial,
  author = {Mingfei Sun and Xiaojuan Ma},
  title = {{Adversarial Imitation Learning from Incomplete Demonstrations}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/487},
}
Figure