SubIQ: Inverse Soft-Q Learning for Offline Imitation with Suboptimal Demonstrations

Huy Hoang, Tien Mai, Pradeep Varakantham

Abstract

We focus on offline imitation learning (IL), which aims to mimic an expert's behavior using demonstrations without any interaction with the environment. One of the main challenges in offline IL is the limited support of expert demonstrations, which typically cover only a small fraction of the state-action space. While it may not be feasible to obtain numerous expert demonstrations, it is often possible to gather a larger set of sub-optimal demonstrations. In this paper, we propose an offline IL approach that leverages the larger set of sub-optimal demonstrations while effectively mimicking expert trajectories. Our approach, called SPRINQL, transforms the offline IL problem into a convex optimization over the space of Q functions. Through comprehensive experimental evaluations, we demonstrate that the SPRINQL algorithm achieves state-of-the-art (SOTA) performance on offline IL benchmarks.

Introduction

Imitation Learning (IL) typically requires a large amount of expert data to teach an agent how to perform a task. However, in real-world applications like surgery or specialized robotics, expert demonstrations are rare and expensive. Conversely, "sub-optimal" data from novices is often abundant but contains errors that are dangerous to mimic directly.

Existing offline IL methods struggle with this trade-off, often overfitting to a tiny set of expert data or inadvertently learning bad habits from larger sub-optimal datasets. The core challenge addressed by this work is how to utilize "noisy" sub-optimal data to understand environment dynamics without copying poor performance.

Methodology: The SPRINQL Framework

The proposed algorithm, SPRINQL (Sub-oPtimal demonstrations driven Reward regularized INverse soft Q Learning), is built on four key pillars:

  1. Inverse Soft-Q Learning Base: Unlike Behavior Cloning, which just maps states to actions, SPRINQL captures the sequential nature of tasks by learning a Q-function (value function).
  2. Convex Optimization: A major technical contribution is transforming the imitation objective into a convex optimization problem. This avoids the training instability and "catastrophic forgetting" common in adversarial (GAN-based) imitation methods.
  3. Reward Regularization & Ranking: The algorithm automatically infers a "reference reward" by looking at the relative quality of different datasets (e.g., "this set of trajectories is better than that one"). It uses a Bradley-Terry preference model to ensure the learned rewards align with these expertise levels.
  4. Conservative Q-Learning (CQL): To handle the "offline" nature of the data, it incorporates a conservative term that penalizes the agent for being too confident about actions it hasn't seen in the dataset, preventing the policy from drifting into unknown, low-value states.

Experimental Evaluation

The researchers evaluated SPRINQL on two major suites of continuous control benchmarks:

  • MuJoCo: Standard locomotion tasks including HalfCheetah, Ant, and Humanoid.
  • Panda-gym: Robotic arm manipulation tasks such as Push, Pick-and-Place (PnP), and Slide.

For each task, the datasets were structured with varying levels of optimality: one high-quality expert trajectory and thousands of sub-optimal transitions generated by adding noise to expert actions.

Results

SPRINQL significantly outperformed all baselines, including specialized methods like DWBC and DemoDICE.

Table 1: Performance comparison of SPRINQL against baseline methods on MuJoCo and Panda-gym benchmarks.

As shown in Table 1:

  • Average Performance: SPRINQL achieved an average score of 67.7, while the next best baseline (DWBC) trailed far behind at 14.6.
  • Complex Tasks: In the Humanoid task—one of the most difficult locomotion benchmarks—SPRINQL scored 82.9, whereas most other methods failed to break 10.
  • Robotics: In robotic manipulation (Slide), SPRINQL achieved 37.7, while almost all other baselines scored near 0, essentially failing to learn the task at all.

Discussion and Conclusion

SPRINQL demonstrates that sub-optimal data is a feature rather than a bug; it provides the necessary "support" for the model to understand environment dynamics, helping it stay on the expert path. By proving the objective is convex in the space of Q-functions, the authors provide a more reliable training recipe than the "cat-and-mouse" game of adversarial imitation learning.

While SPRINQL is a major step forward, future work includes addressing the current assumption of known expertise rankings and investigating the theoretical impact of varying expert-to-non-expert data ratios on Q-learning convergence.

Summary updated

Abstract

We focus on offline imitation learning (IL), which aims to mimic an expert's behavior using demonstrations without any interaction with the environment. One of the main challenges in offline IL is the limited support of expert demonstrations, which typically cover only a small fraction of the state-action space. While it may not be feasible to obtain numerous expert demonstrations, it is often possible to gather a larger set of sub-optimal demonstrations. For example, in treatment optimization problems, there are varying levels of doctor treatments available for different chronic conditions. These range from treatment specialists and experienced general practitioners to less experienced general practitioners. Similarly, when robots are trained to imitate humans in routine tasks, they might learn from individuals with different levels of expertise and efficiency. In this paper, we propose an offline IL approach that leverages the larger set of sub-optimal demonstrations while effectively mimicking expert trajectories. Existing offline IL methods based on behavior cloning or distribution matching often face issues such as overfitting to the limited set of expert demonstrations or inadvertently imitating sub-optimal trajectories from the larger dataset. Our approach, which is based on inverse soft-Q learning, learns from both expert and sub-optimal demonstrations. It assigns higher importance (through learned weights) to aligning with expert demonstrations and lower importance to aligning with sub-optimal ones. A key contribution of our approach, called SPRINQL, is transforming the offline IL problem into a convex optimization over the space of Q functions. Through comprehensive experimental evaluations, we demonstrate that the SPRINQL algorithm achieves state-of-the-art (SOTA) performance on offline IL benchmarks. Code is available at https://github.com/hmhuy0/SPRINQL.
BibTeX
@inproceedings{hoang2024subiq,
  author = {Huy Hoang and Tien Mai and Pradeep Varakantham},
  title = {{SubIQ: Inverse Soft-Q Learning for Offline Imitation with Suboptimal Demonstrations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=uDD44NROOt},
}
Figure