MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations

Anqi Li, Byron Boots, Ching-An Cheng

Abstract

We study a new paradigm for sequential decision making called offline policy learning from observations (PLfO). Offline PLfO aims to learn policies using datasets with substandard qualities: 1) only a subset of trajectories is labeled with rewards, 2) labeled trajectories may not contain actions, 3) labeled trajectories may not be of high quality, and 4) the data may not have full coverage. Such imperfection is common in real-world learning scenarios, and offline PLfO encompasses many existing offline learning setups, including offline imitation learning (IL), offline IL from observations (ILfO), and offline reinforcement learning (RL). In this work, we present a generic approach to offline PLfO, called Modality-agnostic Adversarial Hypothesis Adaptation for Learning from Observations (MAHALO). Built upon the pessimism concept in offline RL, MAHALO optimizes the policy using a performance lower bound that accounts for uncertainty due to the dataset's insufficient coverage. We implement this idea by adversarially training data-consistent critic and reward functions, which forces the learned policy to be robust to data deficiency. We show that MAHALO consistently outperforms or matches specialized algorithms across a variety of offline PLfO tasks in theory and experiments.

Introduction

In the ideal world of Reinforcement Learning (RL), agents learn from perfectly labeled datasets where every state transition is accompanied by a known action and a corresponding reward. However, real-world robotic and industrial data is rarely this clean. Practical datasets are often "substandard," consisting of a mix of expert videos (missing actions), logs of random exploration (missing rewards), or low-quality demonstrations with significant gaps in state coverage.

To address this, researchers from the University of Washington and Microsoft Research have introduced MAHALO (Modality-agnostic Adversarial Hypothesis Adaptation for Learning from Observations). MAHALO is a unified framework designed to bridge the gap between disparate offline learning paradigms, allowing agents to learn robust policies from messy, incomplete data.

The Challenge: Offline Policy Learning from Observations (PLfO)

Most existing offline learning algorithms are specialists. Offline RL requires reward labels for every transition. Imitation Learning (IL) requires expert demonstrations that include specific actions. IL from Observations (ILfO) removes the action requirement but assumes the data is provided by an expert.

The authors identify a critical gap: real-world data is often a hybrid of these scenarios. They define the Offline PLfO paradigm to handle four specific substandard qualities:

  1. Partial Reward Labeling: Only a subset of trajectories contains rewards.
  2. Missing Actions: Labeled trajectories (like human videos) may lack action data.
  3. Mixed Quality: Labeled data may come from non-experts.
  4. Insufficient Coverage: The data does not cover all possible states or transitions.

The primary challenge in PLfO is the "mismatch" between data sources. For instance, the data describing how the environment works (dynamics) might not overlap perfectly with the data describing what is desirable (rewards).

The MAHALO Framework

MAHALO addresses these challenges through the lens of pessimism. In offline learning, if an agent encounters a state-action pair not well-represented in the dataset, it should assume the worst-case outcome to avoid dangerous or suboptimal behavior.

MAHALO implements this principle through a Stackelberg game (a leader-follower optimization):

  • The Actor (Leader): Seeks a policy that maximizes a performance lower bound.
  • The Critic and Reward Function (Followers): These components are trained adversarially. They attempt to find a reward and value function that are consistent with the available data but yield the lowest possible performance for the actor's current policy.

By forcing the actor to succeed even when the pessimistic critic is actively searching for flaws, the resulting policy becomes highly robust to missing data and distribution shifts. Technically, MAHALO extends the Adversarial Training for Actor-Critic (ATAC) algorithm by adding a crucial adversarial reward learning component, allowing it to function even when rewards are absent from the majority of the dataset.

Experimental Evaluation

The researchers evaluated MAHALO on two major benchmarks: D4RL (locomotion tasks like Hopper and HalfCheetah) and Meta-World (robotic manipulation tasks like reaching and pushing). To test the limits of the framework, they constructed five distinct scenarios:

  • ILfO: Expert states only + mixed quality dynamics.
  • IL: Expert states/actions + mixed quality dynamics.
  • RLfO: Expert states/rewards + mixed quality dynamics.
  • RL-expert: Expert states/actions/rewards + mixed quality dynamics.
  • RL-sample: Mixed quality dynamics with only 1% of transitions labeled with rewards.

Key Results and Insights

MAHALO demonstrated remarkable versatility across all tested scenarios, often outperforming algorithms specifically designed for those individual tasks.

  • Superiority over Specialists: In the ILfO scenario on the D4RL Hopper task, MAHALO achieved a score of 104.66, significantly outperforming SMODICE (a state-of-the-art ILfO baseline), which scored 67.72.
  • Robustness to Sparse Rewards: In the RL-sample scenario, where only 1% of the data was labeled with rewards, MAHALO achieved performance nearly identical to an "Oracle" agent that had access to 100% of the reward labels.
  • Meta-World Success: In complex manipulation tasks, MAHALO was frequently the only algorithm to maintain a 100% success rate across all scenarios, whereas baselines like Behavioral Cloning from Observations (BCO) failed when reward and dynamics data were mismatched.

Conclusion and Future Work

MAHALO proves that a unified approach to offline learning is not only possible but often superior to specialized methods. By treating expert demonstrations as high-reward data and employing adversarial pessimism, the framework automatically adapts to the hidden structure of the data, whether it includes actions, rewards, or neither.

While highly effective, the authors note that adversarial training is computationally intensive and can be sensitive to hyperparameter selection. Future research may focus on extending MAHALO to handle "embodiment mismatch," where the agent learns from observations of a different physical entity, such as a robot learning from a video of a human hand.

Summary updated

Abstract

We study a new paradigm for sequential decision making, called offline policy learning from observations (PLfO). Offline PLfO aims to learn policies using datasets with substandard qualities: 1) only a subset of trajectories is labeled with rewards, 2) labeled trajectories may not contain actions, 3) labeled trajectories may not be of high quality, and 4) the data may not have full coverage. Such imperfection is common in real-world learning scenarios, and offline PLfO encompasses many existing offline learning setups, including offline imitation learning (IL), offline IL from observations (ILfO), and offline reinforcement learning (RL). In this work, we present a generic approach to offline PLfO, called $\textbf{M}$odality-agnostic $\textbf{A}$dversarial $\textbf{H}$ypothesis $\textbf{A}$daptation for $\textbf{L}$earning from $\textbf{O}$bservations (MAHALO). Built upon the pessimism concept in offline RL, MAHALO optimizes the policy using a performance lower bound that accounts for uncertainty due to the dataset's insufficient coverage. We implement this idea by adversarially training data-consistent critic and reward functions, which forces the learned policy to be robust to data deficiency. We show that MAHALO consistently outperforms or matches specialized algorithms across a variety of offline PLfO tasks in theory and experiments. Our code is available at https://github.com/AnqiLi/mahalo.
BibTeX
@inproceedings{li2023mahalo,
  author = {Anqi Li and Byron Boots and Ching-An Cheng},
  title = {{MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=BZYKMfm2CO},
}
Figure