MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations

Anqi Li, Byron Boots, Ching-An Cheng

TL;DR

MAHALO introduces a unified framework called Offline Policy Learning from Observations (PLfO) that allows agents to learn from "messy" real-world datasets where rewards, actions, or expert-level quality may be missing. By using adversarial training to enforce a "pessimistic" performance lower bound, MAHALO creates a single, robust algorithm that matches or outperforms specialized methods across both Reinforcement Learning and Imitation Learning tasks.

What Problem Does This Paper Address?

In an ideal world, offline Reinforcement Learning (RL) datasets are perfectly labeled with states, actions, and rewards. In reality, data is often "substandard." You might have a large dynamics dataset of a robot wandering around (actions and states, but no rewards), a few videos of an expert performing a task (states, but no actions or rewards), and some scattered trajectories with sparse reward labels.

Existing methods are highly specialized: Offline RL requires rewards for every transition; Imitation Learning (IL) requires expert actions; and IL from Observations (ILfO) requires expert trajectories but no rewards. There is no single "Swiss Army knife" algorithm that can ingest all these different data modalities simultaneously.

Comparison of offline learning paradigms and their data requirements.

To bridge these gaps, the authors propose Offline Policy Learning from Observations (PLfO) as a general paradigm. As shown in the table above, PLfO is uniquely capable of utilizing a mix of expert and non-expert data, with or without action and reward labels, encompassing all the specialized setups.

Methods: The MAHALO Algorithm

The authors introduce MAHALO (Modality-agnostic Adversarial Hypothesis Adaptation for Learning from Observations). The core idea is built on pessimism—a fundamental concept in offline RL that prevents an agent from overestimating the value of states it hasn't seen much of in the training data.

MAHALO is structured as a Stackelberg game (a leader-follower game):

  1. The Actor (Leader): Tries to find a policy that maximizes a performance lower bound.
  2. The Critic and Reward Function (Followers): These are trained adversarially. They are forced to be consistent with the available data (e.g., matching known rewards or maintaining Bellman consistency), but they are also incentivized to find the "worst-case" scenario for the actor's current policy.

By training the reward function and critic together in this adversarial way, the model becomes robust to "data deficiency." If the actor tries to go into a region of the state space not covered by the data, the adversarial reward function will penalize it, effectively keeping the agent within the safe support of the training data.

Datasets and Scenarios

The researchers evaluated MAHALO on two major benchmarks, modified to simulate five different "substandard" data scenarios:

  • D4RL: Locomotion tasks (Hopper, Walker2d, HalfCheetah).
  • Meta-World: Robotic manipulation tasks (Reach, Push, Plate-slide, etc.).

They created five specific scenarios to test the algorithm's flexibility:

  1. ILfO: Expert states only + mixed quality dynamics data.
  2. IL: Expert states/actions + mixed quality dynamics data.
  3. RLfO: Expert states/rewards + mixed quality dynamics data.
  4. RL-expert: Expert states/actions/rewards + mixed quality dynamics data.
  5. RL-sample: Mixed quality dynamics data where only a small fraction of transitions are labeled with rewards.

Key Insights

  • Unification via PLfO: The paper successfully argues that most offline learning problems are just special cases of PLfO. By solving the most general version, you get a "one-size-fits-all" algorithm.
  • Adversarial Reward Learning: Unlike standard RL which treats rewards as fixed ground truth, MAHALO treats the reward function as a learnable hypothesis. This allows it to "fill in the blanks" for unlabeled data in a way that remains safe and pessimistic.
  • Robustness to Distribution Shift: By optimizing for a performance lower bound, the agent naturally avoids "hallucinating" high rewards in regions where it lacks data, a common failure mode in offline learning.
  • Modality Agnosticism: MAHALO doesn't care if the expert data has actions or not. It uses the dynamics data to learn how to achieve the states seen in the expert observations.

Key Results

MAHALO demonstrated impressive versatility across all tested scenarios, consistently matching or beating specialized baselines.

Performance comparison of MAHALO against baselines across various D4RL tasks and scenarios.
  • Superiority over Specialized Baselines: In D4RL tasks, MAHALO consistently outperformed specialized algorithms like SMODICE (for ILfO) and BCO. For example, in the Hopper ILfO task, MAHALO scored 104.66, while the next best baseline (RP) scored 97.48, and SMODICE scored 67.72.
  • Meta-World Success: In robotic manipulation, MAHALO achieved a 100% success rate in several tasks across multiple scenarios, whereas baselines often struggled to generalize from the limited reward data.
  • Matching the "Oracle": In many cases, MAHALO performed nearly as well as an "Oracle" algorithm that had access to full reward and dynamics information for all data, proving it can effectively leverage unlabeled transitions.

Limitations and Future Work

  • Dynamics Mismatch: MAHALO assumes the dynamics data and the target environment share the same embodiment. It is not yet designed to handle "domain adaptation" (e.g., learning from a human video to control a robotic arm with different physics).
  • Hyperparameter Tuning: While the "relative pessimism" version of MAHALO is more stable than "absolute pessimism," it still introduces hyperparameters that control the strength of the adversarial penalties, which may require tuning for specific tasks.
  • Computational Complexity: Adversarial training (min-max optimization) is notoriously more difficult to stabilize and computationally heavier than standard supervised learning or traditional RL.

Summary updated

Abstract

We study a new paradigm for sequential decision making, called offline policy learning from observations (PLfO). Offline PLfO aims to learn policies using datasets with substandard qualities: 1) only a subset of trajectories is labeled with rewards, 2) labeled trajectories may not contain actions, 3) labeled trajectories may not be of high quality, and 4) the data may not have full coverage. Such imperfection is common in real-world learning scenarios, and offline PLfO encompasses many existing offline learning setups, including offline imitation learning (IL), offline IL from observations (ILfO), and offline reinforcement learning (RL). In this work, we present a generic approach to offline PLfO, called Modality-agnostic Adversarial Hypothesis Adaptation for Learning from Observations (MAHALO). Built upon the pessimism concept in offline RL, MAHALO optimizes the policy using a performance lower bound that accounts for uncertainty due to the dataset’s insufficient coverage. We implement this idea by adversarially training data-consistent critic and reward functions, which forces the learned policy to be robust to data deficiency. We show that MAHALO consistently outperforms or matches specialized algorithms across a variety of offline PLfO tasks in theory and experiments. Our code is available at https://github.com/AnqiLi/mahalo.
BibTeX
@inproceedings{li2023mahalo,
  author = {Anqi Li and Byron Boots and Ching-An Cheng},
  title = {{MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=BZYKMfm2CO},
}
Figure