Offline Imitation from Observation via Primal Wasserstein State Occupancy Matching

Kai Yan, Alexander G. Schwing, Yu-Xiong Wang

Abstract

In real-world scenarios, arbitrary interactions with the environment can often be costly, and actions of expert demonstrations are not always available. To reduce the need for both, offline Learning from Observations (LfO) is extensively studied: the agent learns to solve a task given only expert states and task-agnostic non-expert state-action pairs. The state-of-the-art DIstribution Correction Estimation (DICE) methods, as exemplified by SMODICE, minimize the state occupancy divergence between the learner's and empirical expert policies. However, such methods are limited to either $f$-divergences (KL and $chi^2$) or Wasserstein distance with Rubinstein duality, the latter of which constrains the underlying distance metric crucial to the performance of Wasserstein-based solutions. To enable more flexible distance metrics, we propose Primal Wasserstein DICE (PW-DICE). It minimizes the primal Wasserstein distance between the learner and expert state occupancies and leverages a contrastively learned distance metric. Theoretically, our framework is a generalization of SMODICE, and is the first work that unifies $f$-divergence and Wasserstein minimization. Empirically, we find that PW-DICE improves upon several state-of-the-art methods. The code is available at https://github.com/KaiYan289/PW-DICE.

Introduction

Developing autonomous agents often faces two significant hurdles: the high cost of real-world environment interaction and the scarcity of complete expert data. Offline Reinforcement Learning (RL) addresses the former by utilizing static datasets, yet it relies on reward labels that are frequently difficult to engineer. Imitation Learning (IL) bypasses reward design by mimicking experts but typically requires access to expert actions.

The most realistic and challenging setting is Offline Learning from Observation (LfO), where the agent has access to a task-agnostic dataset of interactions and a small set of expert state-only trajectories. Current state-of-the-art methods, such as DICE (Distribution Correction Estimation), attempt to match the learner's state occupancy to the expert's. However, these are often restricted to $f$-divergences that ignore environmental geometry or use dual-form Wasserstein distances that mandate simple Euclidean metrics. PW-DICE addresses this by enabling flexible, learned distance metrics through a primal Wasserstein formulation.

Methodology

PW-DICE introduces a novel framework for state occupancy matching by shifting from the dual form of the Wasserstein distance to its primal form. This shift is critical as it removes the Lipschitz constraint typically associated with dual Wasserstein methods, allowing for the integration of arbitrary distance metrics.

Overview of the PW-DICE framework, illustrating the problem setting, Wasserstein optimization, and policy extraction.

The core components of the methodology include:

  1. Primal Optimization: The problem is formulated as a single-level convex optimization, avoiding the complexities of dual-form constraints.
  2. Flexible Distance Metrics: PW-DICE utilizes a hybrid metric $c(s, s')$ that combines a discriminator-based reward to identify expert-like states with a contrastively learned embedding (via InfoNCE loss). This embedding measures "reachability," defining proximity based on the agent's ability to transition between states in the task-agnostic data.
  3. Policy Retrieval: After determining the optimal matching between learner and expert distributions, the final policy is extracted using weighted behavior cloning on the task-agnostic dataset.

Results and Discussion

The authors evaluated PW-DICE across a diverse suite of benchmarks, including tabular MDPs, D4RL MuJoCo locomotion tasks (Hopper, HalfCheetah, Ant, Walker2d), and complex navigation environments like Antmaze.

Impact of distance metrics on performance across various tasks.

Key Insights

  • The Metric is the Message: A central finding is that the underlying distance metric $c(s, s')$ is a primary driver of performance. As demonstrated in comparative analyses, standard Euclidean distance is often insufficient for capturing the true similarity between complex robot states compared to more flexible metrics like cosine similarity in learned embedding spaces.
  • Unified Theory: PW-DICE is a theoretical generalization of previous methods like SMODICE. Under specific hyperparameter configurations, SMODICE emerges as a special case, providing a unified framework for $f$-divergence and Wasserstein-based imitation.
  • Robustness and Transfer: By prioritizing reachability over raw coordinates, PW-DICE demonstrates superior robustness to distorted state representations and successfully handles embodiment transfer (mismatched dynamics), such as learning from an expert with different physical proportions.

Performance

PW-DICE consistently outperformed or matched seven major baselines, including SMODICE and LobsDICE, across nearly all MuJoCo tasks. It maintained high performance in scenarios where traditional $f$-divergence methods failed due to state representation distortions.

Conclusion

While PW-DICE sets a new state-of-the-art in offline LfO, it faces challenges such as potential numerical instability from logsumexp terms and a slight computational overhead from training the contrastive metric. Future work will focus on refining the optimization objective to enhance stability while preserving the flexibility of the primal Wasserstein approach.

Summary updated

Abstract

In real-world scenarios, arbitrary interactions with the environment can often be costly, and actions of expert demonstrations are not always available. To reduce the need for both, offline Learning from Observations (LfO) is extensively studied: the agent learns to solve a task given only expert states and task-agnostic non-expert state-action pairs. The state-of-the-art DIstribution Correction Estimation (DICE) methods, as exemplified by SMODICE, minimize the state occupancy divergence between the learner's and empirical expert policies. However, such methods are limited to either $f$-divergences (KL and $chi^2$) or Wasserstein distance with Rubinstein duality, the latter of which constrains the underlying distance metric crucial to the performance of Wasserstein-based solutions. To enable more flexible distance metrics, we propose Primal Wasserstein DICE (PW-DICE). It minimizes the primal Wasserstein distance between the learner and expert state occupancies and leverages a contrastively learned distance metric. Theoretically, our framework is a generalization of SMODICE, and is the first work that unifies $f$-divergence and Wasserstein minimization. Empirically, we find that PW-DICE improves upon several state-of-the-art methods. The code is available at https://github.com/KaiYan289/PW-DICE.
BibTeX
@inproceedings{yan2024offline,
  author = {Kai Yan and Alexander G. Schwing and Yu-Xiong Wang},
  title = {{Offline Imitation from Observation via Primal Wasserstein State Occupancy Matching}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=4Zr7T6UrBS},
}
Figure