Situated GAIL: Multitask imitation using task-conditioned adversarial inverse reinforcement learning

Kyoichiro Kobayashi, Takato Horii, Ryo Iwaki, Yukie Nagai, Minoru Asada

TL;DR

This paper introduces Situated GAIL (S-GAIL), a novel framework that extends Generative Adversarial Imitation Learning (GAIL) to enable robots to learn multiple tasks with different reward functions and policies within a single model. By incorporating a "task variable" into both the generator and discriminator, S-GAIL effectively differentiates between tasks while allowing for shared knowledge, leading to more efficient and robust multi-task imitation learning.

What Problem Does This Paper Address?

Traditional imitation learning methods, like GAIL, are excellent at teaching robots to perform a single task by observing an expert. However, they struggle when an expert demonstrates multiple distinct tasks, each with its own underlying goal (reward function) and corresponding way of achieving it (policy). Existing extensions like InfoGAIL can handle multiple policies but still assume a single reward function. This limitation means that to teach a robot multiple tasks, one would typically need to train separate models for each task, which is inefficient and doesn't leverage potential shared knowledge about the environment. The paper addresses this by proposing a unified framework that can learn multiple tasks simultaneously, each with its own reward function and policy, within a single learning architecture.

Methods

The core innovation of S-GAIL lies in two key modifications to the GAIL framework:

  1. Task Variable Integration: A "task variable" (denoted as c) is introduced and fed into both the generator (which learns the policy) and the discriminator (which learns to distinguish expert from generated behavior and implicitly estimates the reward function). This variable acts as a conditional input, allowing the model to learn task-specific reward functions and policies.
  2. AIRL Discriminator Structure: The paper adopts the discriminator structure proposed in Adversarial Inverse Reinforcement Learning (AIRL). This structure is designed to explicitly recover reward functions, rather than just learning policies. By combining this with the task variable, S-GAIL can learn distinct reward functions for each task.

Additionally, S-GAIL incorporates an entropy-regularized term with an adjustable coefficient in the generator's objective function. This coefficient is scheduled to be high initially (focusing on mimicking the expert precisely) and then decreased, allowing for more exploration and learning of robust policies. This helps balance imitation accuracy with policy exploration.

Datasets

The paper uses two experimental setups:

  • Grid World: A custom 11x11 grid world maze where an agent must navigate to one of two distinct target locations. Expert trajectories were generated for each task.
  • Robot Arm Simulator (Reacher-v2): A continuous control environment from OpenAI Gym, where a robot arm must reach one of two target objects. Expert demonstrations were collected for each target.

Key Insights and Ideas

  • Task Conditioning is Key: Explicitly conditioning both the generator and discriminator on a task variable is crucial for learning distinct reward functions and policies for multiple tasks within a single model.
  • Shared Knowledge Benefits Learning: By using a single framework for multiple tasks, S-GAIL can leverage shared environmental dynamics and potentially learn faster and more robustly than training separate models.
  • AIRL's Discriminator is Essential for Reward Recovery: The specific structure of the AIRL discriminator is vital for enabling the model to actually estimate reward functions, which is a significant improvement over standard GAIL.
  • Scheduled Entropy Regularization Balances Imitation and Exploration: Dynamically adjusting the entropy regularization coefficient allows the model to first accurately imitate the expert and then explore to find more robust policies, leading to faster convergence and better performance.

Key Results

  • Superior Performance in Grid World: S-GAIL significantly outperformed InfoGAIL and its variants in a grid world navigation task, achieving higher performance in reaching target locations.
  • Clearer Value Functions: S-GAIL learned distinct value functions for each task, whereas InfoGAIL's value functions were confused between tasks, indicating S-GAIL's ability to differentiate reward structures.
  • Effective Multi-Task Learning in Continuous Space: In the robot arm simulation, S-GAIL successfully learned to reach two different targets, demonstrating its scalability to continuous control problems.
  • Faster Convergence with Scheduled Entropy: Using a scheduled entropy regularization coefficient (β) led to faster convergence and improved performance compared to fixed coefficients.
  • Reduced Demonstrations Needed: S-GAIL achieved comparable or better performance with half the number of expert demonstrations compared to separate AIRL models for each task.

Limitations and Future Work

  • Task Variable Assumption: The current framework assumes that the task variable c is provided alongside the expert demonstrations. The paper suggests future work to automatically estimate the number and type of tasks.
  • Discrete Task Variables: The current task variable is discrete. Future work aims to introduce continuous variables to modulate reward function shapes.
  • Real-World Robot Teleoperation: The authors plan to validate S-GAIL on a real robot learning from human teleoperation, using task switching signals and language instructions.

Summary updated

Abstract

Generative adversarial imitation learning (GAIL) has attracted increasing attention in the field of robot learning. It enables robots to learn a policy to achieve a task demonstrated by an expert while simultaneously estimating the reward function behind the expert's behaviors. However, this framework is limited to learning a single task with a single reward function. This study proposes an extended framework called situated GAIL (S-GAIL), in which a task variable is introduced to both the discriminator and generator of the GAIL framework. The task variable has the roles of discriminating different contexts and making the framework learn different reward functions and policies for multiple tasks. To achieve the early convergence of learning and robustness during reward estimation, we introduce a term to adjust the entropy regularization coefficient in the generator's objective function. Our experiments using two setups (navigation in a discrete grid world and arm reaching in a continuous space) demonstrate that the proposed framework can acquire multiple reward functions and policies more effectively than existing frameworks. The task variable enables our framework to differentiate contexts while sharing common knowledge among multiple tasks.
BibTeX
@misc{kobayashi2019situated,
  author = {Kyoichiro Kobayashi and Takato Horii and Ryo Iwaki and Yukie Nagai and Minoru Asada},
  title = {{Situated GAIL: Multitask imitation using task-conditioned adversarial inverse reinforcement learning}},
  year = {2019},
  eprint = {1911.00238},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.00238},
}
Figure