SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning

Huy Hoang, Tien Mai, Pradeep Varakantham

Abstract

We focus on offline imitation learning (IL), which aims to mimic an expert's behavior using demonstrations without any interaction with the environment. One of the main challenges in offline IL is the limited support of expert demonstrations, which typically cover only a small fraction of the state-action space. While it may not be feasible to obtain numerous expert demonstrations, it is often possible to gather a larger set of sub-optimal demonstrations. In this paper, we propose an offline IL approach that leverages the larger set of sub-optimal demonstrations while effectively mimicking expert trajectories. Our approach, called SPRINQL, transforms the offline IL problem into a convex optimization over the space of Q functions. Through comprehensive experimental evaluations, we demonstrate that the SPRINQL algorithm achieves state-of-the-art (SOTA) performance on offline IL benchmarks.

Introduction

In real-world applications like robotic surgery or autonomous driving, "perfect" expert data is often scarce. While we may only have a handful of expert demonstrations, we frequently possess vast amounts of "sub-optimal" data from trainees or less experienced practitioners. Standard Offline Imitation Learning (IL) methods struggle with this mix: Behavior Cloning (BC) tends to overfit to small expert sets or average out when trained on noisy data, while distribution matching methods often require massive expert datasets to remain stable.

SPRINQL (Sub-oPtimal demonstrations driven Reward regularized INverse soft Q Learning) addresses this "scarcity of experts" problem. It effectively learns from a small set of expert data combined with a much larger pool of sub-optimal demonstrations by transforming the imitation task into a stable, convex optimization problem.

Methodology

SPRINQL is built upon three primary pillars designed to handle imperfect data robustly:

  1. Inverse Soft-Q Learning (IQ-Learn) Foundation: Unlike unstable adversarial training (GAN-style) methods, SPRINQL builds on IQ-Learn, which learns a Q-function directly from demonstrations, providing a more stable training signal.
  2. Reward Regularization with Reference Rewards: The algorithm utilizes a "reference reward" $\bar{r}(s, a)$ that assigns higher values to state-action pairs found in higher-ranked demonstrations. This ensures the learned reward function aligns with a known expertise hierarchy.
  3. Convex Optimization: A key theoretical contribution is the derivation of a concave lower bound for the objective function. This transformation ensures that the training process is mathematically well-behaved and converges reliably.
  4. Conservatism (CQL): To prevent the agent from becoming overconfident about actions not present in the dataset, SPRINQL incorporates a conservative term that lower-bounds the Q-values, a technique borrowed from Conservative Q-Learning.

Experimental Setup

The researchers evaluated SPRINQL on two major suites of environments:

  • MuJoCo: Continuous control tasks including HalfCheetah, Ant, Humanoid, Walker2d, and Hopper.
  • Panda-gym: Robotic arm manipulation tasks like Reach, Push, Pick-and-Place (PnP), and Slide.

For each task, datasets were generated with multiple levels of expertise by adding varying amounts of noise to an expert policy, mimicking real-world scenarios with a few expert trajectories and many noisy, sub-optimal ones.

Key Results

SPRINQL significantly outperformed all baselines, including specialized methods like DemoDICE and DWBC. On average across MuJoCo tasks, SPRINQL achieved a normalized score of 78.3, while the next best baseline (W-BC) reached only 31.9. In Panda-gym, SPRINQL averaged 68.2, compared to 44.7 for the next best competitor (DWBC).

Performance comparison of SPRINQL against baselines on MuJoCo and Panda-gym benchmarks.

Beyond raw performance, SPRINQL demonstrated several key advantages:

  • Robustness to Data Size: Performance remains stable even when the amount of sub-optimal data is reduced, whereas other methods see their performance collapse.
  • Reward Recovery: SPRINQL can recover a reward function that correlates highly with the ground-truth reward, making the model more interpretable than "black box" policy-only methods.
  • Ablation Success: Experiments showed that both Distribution Matching and Reward Regularization are essential; removing either component leads to a massive drop in performance.

Conclusion and Future Work

SPRINQL provides a robust framework for offline IL by effectively utilizing ranked sub-optimal data. While it achieves SOTA results, some limitations remain for future exploration. The method currently relies on demonstrations being pre-classified into expertise levels. Future work could investigate how to automate this ranking or explore the theoretical limits of the expert-to-non-expert data ratio. Additionally, the performance in extremely low-sample regimes (e.g., only 1 or 2 expert trajectories) remains an open area for further refinement.

Summary updated

Abstract

We focus on offline imitation learning (IL), which aims to mimic an expert's behavior using demonstrations without any interaction with the environment. One of the main challenges in offline IL is the limited support of expert demonstrations, which typically cover only a small fraction of the state-action space. While it may not be feasible to obtain numerous expert demonstrations, it is often possible to gather a larger set of sub-optimal demonstrations. For example, in treatment optimization problems, there are varying levels of doctor treatments available for different chronic conditions. These range from treatment specialists and experienced general practitioners to less experienced general practitioners. Similarly, when robots are trained to imitate humans in routine tasks, they might learn from individuals with different levels of expertise and efficiency. In this paper, we propose an offline IL approach that leverages the larger set of sub-optimal demonstrations while effectively mimicking expert trajectories. Existing offline IL methods based on behavior cloning or distribution matching often face issues such as overfitting to the limited set of expert demonstrations or inadvertently imitating sub-optimal trajectories from the larger dataset. Our approach, which is based on inverse soft-Q learning, learns from both expert and sub-optimal demonstrations. It assigns higher importance (through learned weights) to aligning with expert demonstrations and lower importance to aligning with sub-optimal ones. A key contribution of our approach, called SPRINQL, is transforming the offline IL problem into a convex optimization over the space of Q functions. Through comprehensive experimental evaluations, we demonstrate that the SPRINQL algorithm achieves state-of-the-art (SOTA) performance on offline IL benchmarks. Code is available at https://github.com/hmhuy0/SPRINQL.
BibTeX
@inproceedings{hoang2024sprinql,
  author = {Huy Hoang and Tien Mai and Pradeep Varakantham},
  title = {{SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=uDD44NROOt},
}
Figure