A Dual Approach to Imitation Learning from Observations with Offline Datasets

Harshit Sikchi, Caleb Chuck, Amy Zhang, Scott Niekum

TL;DR

This paper introduces DILO, a novel imitation learning algorithm that learns from expert demonstrations without requiring expert actions. DILO overcomes limitations of prior methods by directly learning a multi-step utility function, enabling it to scale to high-dimensional observations and achieve improved performance on real-world robotic tasks.

What Problem Does This Paper Address?

Imitation Learning (IL) aims to train agents by observing expert behavior, which is often more practical than manually designing reward functions. However, a significant challenge arises when expert demonstrations are only available as observation trajectories (e.g., videos) without explicit action labels. This "learning from observations" (LfO) setting is further complicated by the need to leverage existing offline datasets of the agent's own interactions, which may be suboptimal. Existing LfO methods often rely on intermediate models (like inverse dynamics models or discriminators) that can introduce compounding errors, especially in offline settings with limited data. This paper addresses the need for an efficient, principled, and robust LfO algorithm that can learn from action-free expert observations and arbitrary quality offline agent data, without relying on error-prone intermediate models.

Methods

The core of DILO lies in reframing the LfO problem as a distribution matching objective between the agent's and expert's joint state-next state visitations. By leveraging the principle of duality, DILO derives an action-free objective that bypasses the need for learning intermediate one-step models. Specifically, it formulates the problem as a constrained optimization problem and then uses Lagrangian duality to obtain an unconstrained dual objective. This dual objective learns a multi-step utility function that quantifies how transitioning to a particular next state impacts the agent's divergence from the expert's visitation distribution. For practical implementation, DILO uses the Pearson Chi-square divergence and a value-weighted regression approach to extract the policy.

Datasets

The paper evaluates DILO on a variety of datasets:

  • D4RL: Used for offline imitation learning benchmarks, including locomotion and manipulation tasks. These datasets consist of suboptimal agent trajectories mixed with expert trajectories.
  • Robomimic: Used for experiments with image observations, including locomotion and manipulation tasks. These datasets include suboptimal "Machine Generated" (MG) and "Multi-human" (MH) data, with "Proficient-human" (PH) data serving as expert observations.
  • Real-world Robot Manipulation: Datasets were collected from a UR5e robotic arm performing tasks like safe object manipulation, stationary puck striking, and dynamic puck hitting. These datasets include expert trajectories and suboptimal agent trajectories.

Key Insights and Ideas

  • Next-state encodes action information: The paper leverages the insight that the next state in a trajectory implicitly contains information about the missing expert actions.
  • Duality as a powerful tool: Applying Lagrangian duality to the distribution matching objective allows for the derivation of an action-free, unconstrained objective, simplifying the learning process.
  • Avoiding compounding errors: By directly learning a multi-step utility function, DILO sidesteps the error accumulation issues inherent in methods that rely on learned one-step models (like discriminators or IDMs).
  • Leveraging suboptimal data: DILO is designed to effectively utilize offline datasets of arbitrary quality, which is crucial for practical applications.
  • Action-free learning is feasible and effective: The success of DILO demonstrates that learning from observation-only expert data is achievable and can lead to competitive or superior performance compared to methods that require expert actions.

Key Results

  • Improved performance across benchmarks: DILO consistently outperformed baselines like BCO and SMODICE on various D4RL locomotion and manipulation tasks, especially in settings with few expert observations or high-dimensional image observations.
  • Scalability to image observations: DILO demonstrated impressive performance on image-based imitation tasks from Robomimic, without requiring extensive hyperparameter tuning.
  • Real-world robotic success: On real-robot manipulation tasks (air hockey), DILO achieved significantly higher success rates compared to baselines, particularly in challenging dynamic scenarios. For instance, in the dynamic puck-hitting task, DILO achieved a 9/10 success rate compared to 1/10 for SMODICE and 8/10 for BCO.
  • Robustness to dataset quality: DILO showed improved performance on real-world tasks compared to prior methods, which were more sensitive to the quality of the suboptimal dataset.

Limitations and Future Work

  • Distribution matching in observation space: The current method relies on matching visitation distributions in the observation space. The authors hope that advancements in universal representations will enable matching in more meaningful semantic spaces.
  • Expert optimality assumption: The method assumes expert optimality, but real-world experts exhibit biases. Extending the method to handle these biases is a future direction.
  • Extrapolation control: The conservatism parameter ($\lambda$) in DILO helps regulate extrapolation. However, in some settings, values can diverge, and adaptively selecting $\lambda$ to maximize extrapolation while avoiding divergence is an area for future investigation.
  • Failure modes: While DILO generally performs well, it can exhibit conservative behavior, leading to slower motions or "dead zone" behavior in low-dimensional settings. In challenging tasks with narrow data regimes, DILO's conservatism can lead to less aggressive but safer actions, which might not always be optimal for tasks requiring precise timing and force.

Summary updated

Abstract

Demonstrations are an effective alternative to task specification for learning agents in settings where designing a reward function is difficult. However, demonstrating expert behavior in the action space of the agent becomes unwieldy when robots have complex, unintuitive morphologies. We consider the practical setting where an agent has a dataset of prior interactions with the environment and is provided with observation-only expert demonstrations. Typical learning from observations approaches have required either learning an inverse dynamics model or a discriminator as intermediate steps of training. Errors in these intermediate one-step models compound during downstream policy learning or deployment. We overcome these limitations by directly learning a multi-step utility function that quantifies how each action impacts the agent's divergence from the expert's visitation distribution. Using the principle of duality, we derive DILO (Dual Imitation Learning from Observations), an algorithm that can leverage arbitrary suboptimal data to learn imitating policies without requiring expert actions. DILO reduces the learning from observations problem to that of simply learning an actor and a critic, bearing similar complexity to vanilla offline RL. This allows DILO to gracefully scale to high dimensional observations, and demonstrate improved performance across the board.
BibTeX
@inproceedings{sikchi2024dual,
  author = {Harshit Sikchi and Caleb Chuck and Amy Zhang and Scott Niekum},
  title = {{A Dual Approach to Imitation Learning from Observations with Offline Datasets}},
  booktitle = {Conference on Robot Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=uHdVI3QMr6},
}
Figure