Goal-conditioned Imitation Learning

Yiming Ding, Carlos Florensa, Pieter Abbeel, Mariano Phielipp

Reinforcement Learning (RL) has a notorious "reward problem." In robotics, defining a reward function that is both easy to compute and efficient to optimize is incredibly difficult. While Goal-conditioned RL—where an agent learns to reach any given state—offers a path toward self-supervision, it usually relies on sparse rewards. The agent only receives a positive signal if it perfectly matches the goal, making learning slow and difficult.

To solve this, researchers often use Hindsight Experience Replay (HER). HER "relabels" failed attempts as successful ones by pretending a visited state was the intended goal all along. However, HER essentially performs a breadth-first search and struggles with "bottlenecks" in the state space. For example, an agent might need to pass through a narrow doorway or successfully grasp an object before the rest of the task can even be explored.

The paper introduces goalGAIL, a novel framework that combines the exploration power of HER with the efficiency of Generative Adversarial Imitation Learning (GAIL). By introducing a "hindsight" relabeling technique for expert demonstrations, goalGAIL allows robots to learn complex, multi-goal tasks using as few as 20 demonstrations—even if those demonstrations are noisy or lack action data.

Core Innovations

The authors propose three primary methodological improvements to overcome the limitations of standard imitation learning and HER:

1. Expert Relabeling Standard Imitation Learning treats an expert trajectory as a path to a single, specific goal. The authors realize that any state visited by an expert is a valid goal in itself. By relabeling expert transitions with future states from the same trajectory, they effectively perform data augmentation. This extracts significantly more "goal-reaching" information from a tiny number of demonstrations.

2. goalGAIL and Hybrid Rewards goalGAIL is a goal-conditioned version of GAIL. It uses a discriminator network to distinguish between the agent's behavior and the expert's behavior for a specific goal. The agent is trained to "fool" this discriminator. Crucially, the agent receives a hybrid reward: a combination of the sparse environment reward (did it reach the goal?) and the GAIL reward (does it look like the expert?). To ensure the agent can eventually outperform a suboptimal expert, the weight of the GAIL reward is gradually annealed over time. This allows the sparse environment reward to take over once the agent has discovered how to reach the goal.

3. State-Only Learning The authors demonstrate that the discriminator can function using only state transitions $(s, s')$ rather than state-action pairs $(s, a)$. This is vital for real-world robotics, where we might have video demonstrations of a task but lack the specific motor torques (actions) used to achieve it.

Evaluated Environments

The researchers evaluated goalGAIL on four continuous control environments simulated in MuJoCo:

  • Continuous Four Rooms: A point-mass must navigate through narrow openings to reach goals in different rooms, serving as a classic bottleneck problem.
Continuous Four Rooms Trajectories and Coverage
  • Pointmass Block Pusher: A point-mass must navigate to a block and push it to a target location.
  • Fetch Pick & Place: A 7-DOF robot arm must pick up a block and move it to a 3D target.
  • Fetch Stack Two: A more complex task where the robot must stack two blocks at a specific location.
Fetch Robot Environment

Key Results and Insights

The combination of HER and GAIL yields several significant advantages over baseline methods:

  • Faster Convergence and Higher Coverage: In all tested environments, goalGAIL reached high "coverage" (the percentage of possible goals the agent can successfully reach) significantly faster than HER alone.
  • Outperforming Baselines: goalGAIL consistently outperformed Behavioral Cloning combined with HER (BC + HER). In the complex "Fetch Stack Two" task, goalGAIL achieved nearly 80% coverage, whereas other methods struggled to reach 20-40% in the same timeframe.
  • Robustness to Noise: Unlike Behavioral Cloning, which tries to copy expert actions exactly and fails if the expert is noisy, goalGAIL only cares if the agent is moving in the right direction. When tested with suboptimal experts (e.g., experts taking random actions 40% of the time), goalGAIL's performance remained stable, while BC methods collapsed.
  • High Data Efficiency: The system works with as few as 20 demonstrations. This exceptionally low number makes the approach highly feasible for real-world robotic applications where collecting expert data is expensive.

Limitations and Future Work

While the results are promising, the authors note a few limitations. First, the experiments were conducted entirely in simulation (MuJoCo); transitioning these methods to physical robots is the next logical step. Second, the current experiments relied on low-dimensional state inputs like coordinates and joint angles. The authors suggest goalGAIL could be a solution for high-dimensional vision-based tasks where Behavioral Cloning typically fails in low-data regimes, but this remains to be fully demonstrated. Finally, balancing the GAIL reward and the sparse indicator reward can be tricky, as it affects the interpretation of the Q-function, suggesting room for further theoretical refinement.

Summary updated

Abstract

Designing rewards for Reinforcement Learning (RL) is challenging because it needs to convey the desired task, be efficient to optimize, and be easy to compute. The latter is particularly problematic when applying RL to robotics, where detecting whether the desired configuration is reached might require considerable supervision and instrumentation. Furthermore, we are often interested in being able to reach a wide range of configurations, hence setting up a different reward every time might be unpractical. Methods like Hindsight Experience Replay (HER) have recently shown promise to learn policies able to reach many goals, without the need of a reward. Unfortunately, without tricks like resetting to points along the trajectory, HER might require many samples to discover how to reach certain areas of the state-space. In this work we propose a novel algorithm goalGAIL, which incorporates demonstrations to drastically speed up the convergence to a policy able to reach any goal, surpassing the performance of an agent trained with other Imitation Learning algorithms. Furthermore, we show our method can also be used when the available expert trajectories do not contain the actions or when the expert is suboptimal, which makes it applicable when only kinesthetic, third person or noisy demonstration is available.
BibTeX
@inproceedings{ding2019goal,
  author = {Yiming Ding and Carlos Florensa and Pieter Abbeel and Mariano Phielipp},
  title = {{Goal-conditioned Imitation Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=dSeT9dZnoQ},
}
Figure