TL;DR
PW-DICE is a novel framework for Offline Learning from Observation (LfO) that allows agents to mimic experts using only state trajectories (no expert actions required). By optimizing the primal form of the Wasserstein distance, the method enables the use of flexible, contrastively learned distance metrics, effectively unifying $f$-divergence and Wasserstein-based imitation while setting new state-of-the-art performance records.
What Problem Does This Paper Address?
In the quest for autonomous agents, two major bottlenecks exist: interacting with the real world is expensive, and expert data is scarce. Offline Reinforcement Learning solves the first by learning from static datasets, but it requires reward labels—which are notoriously hard to design. Imitation Learning (IL) removes the need for rewards by following an expert, but it usually requires knowing exactly which actions the expert took.
The most challenging (and realistic) scenario is Offline Learning from Observation (LfO). Here, we have a task-agnostic dataset of "random" interactions and a small set of expert "videos" (states only). Current state-of-the-art methods, known as DICE (Distribution Correction Estimation), try to match the learner's state occupancy to the expert's. However, they are often limited to $f$-divergences (like KL divergence) that ignore the underlying geometry of the environment, or they use the "dual" form of the Wasserstein distance, which forces them to use simple Euclidean distance metrics. This paper asks: Can we use more flexible, learned distance metrics to improve imitation?
Methods
The authors introduce PW-DICE (Primal Wasserstein DICE). The core innovation is shifting from the dual form of the Wasserstein distance to its primal form.
- Primal Optimization: By working with the primal form, the researchers avoid the "Lipschitz constraint" that usually ties Wasserstein methods to Euclidean distance. They formulate the problem as a single-level convex optimization.
- Flexible Distance Metrics: Because the framework is no longer tied to Euclidean distance, the authors propose a hybrid metric $c(s_i, s_j)$. This metric combines:
- A discriminator-based reward (identifying "expert-like" states).
- A contrastively learned embedding (using InfoNCE loss) that measures "reachability." Essentially, states are considered "close" if the agent can easily travel between them in the task-agnostic dataset.
- Policy Retrieval: Once the optimal "matching" between the learner and expert distributions is found, the final policy is extracted via weighted behavior cloning on the task-agnostic data.
Datasets
The authors evaluated PW-DICE across a rigorous suite of benchmarks:
- Tabular MDPs: For theoretical validation and comparison against exact solvers.
- D4RL MuJoCo Benchmarks: Standard locomotion tasks including Hopper, HalfCheetah, Ant, and Walker2d.
- Complex Environments: More difficult manipulation and navigation tasks like Antmaze and Franka Kitchen.
- Distorted/Mismatched Settings: They tested robustness by "distorting" state representations (scrambling coordinates) and using "mismatched dynamics" (e.g., an expert robot with an amputated leg).
Key Insights and Ideas
- The Metric is the Message: One of the paper's biggest "aha" moments is demonstrating that the underlying distance metric $c(s, s')$ is a massive performance lever. Standard Euclidean distance is often a poor proxy for how "similar" two robot states actually are.
- A Unified Theory: The authors prove that PW-DICE is a generalization of previous SOTA methods like SMODICE. By choosing specific hyperparameters, SMODICE emerges as a special case of the PW-DICE framework. This provides a unified lens for viewing $f$-divergence and Wasserstein-based imitation.
- Reachability over Coordinates: By using contrastive learning to define distance based on reachability, the agent learns a "topological map" of the environment. This makes the agent much more robust to changes in how states are represented (e.g., changing units from inches to meters or scrambling sensor indices).
Key Results
- New State-of-the-Art: PW-DICE outperformed or matched seven major baselines (including SMODICE and LobsDICE) across almost all MuJoCo tasks.
- Superior Robustness: In experiments where state representations were randomly distorted, PW-DICE maintained high performance while traditional $f$-divergence methods collapsed.
- Embodiment Transfer: PW-DICE successfully learned from experts with different physical bodies (mismatched dynamics), a common hurdle in real-world robotics where a human might provide the "expert" demonstration for a robot arm.
Limitations and Future Work
The authors identify a few areas for improvement:
- LogSumExp Instability: The mathematical formulation results in a
logsumexpterm. While common in DICE methods, this can sometimes lead to numerical instability when training with small minibatches due to biased gradient estimates. - Computational Overhead: Training the contrastive distance metric adds an extra step to the pipeline (about 20-40 minutes in their experiments), though they note this is not a bottleneck compared to the overall training time.
- Future Directions: The next step is developing a more robust formulation that retains the benefits of the primal Wasserstein approach while avoiding the potential instabilities of the current dual-objective optimization.
Summary updated