SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning

Huy Hoang, Tien Mai, Pradeep Varakantham

TL;DR

SPRINQL is a novel offline imitation learning (IL) algorithm that solves the "scarcity of experts" problem by effectively learning from a small set of expert data combined with a much larger pool of sub-optimal demonstrations. By transforming the imitation task into a stable, convex optimization problem over Q-functions and using a "reference reward" to weight data by expertise, SPRINQL achieves state-of-the-art performance across standard robotic control benchmarks.

What Problem Does This Paper Address?

In the real world, "perfect" data is hard to come by. If you want to train a robot to perform surgery or assemble a car using Offline Imitation Learning (learning from a fixed dataset without trial-and-error in a simulator), you usually only have a handful of expert demonstrations. However, you likely have mountains of "okay" or "sub-optimal" data from trainees or less experienced practitioners.

The challenge is that existing IL methods struggle with this mix. Simple Behavior Cloning (BC) might overfit to the tiny expert set or, if trained on everything, "average out" and become mediocre. More advanced Distribution Matching methods often require massive amounts of expert data to be stable. The authors tackle the question: How can we use a large amount of mediocre data to actually help us reach expert-level performance?

Methods

The authors introduce SPRINQL (Sub-oPtimal demonstrations driven Reward regularized INverse soft Q Learning). Its architecture rests on three main pillars:

  1. Inverse Soft-Q Learning (IQ-Learn) Foundation: Instead of the unstable adversarial training (GAN-style) used in many IL methods, SPRINQL builds on IQ-Learn, which learns a Q-function directly from demonstrations.
  2. Reward Regularization with Reference Rewards: The authors define a "reference reward" $\bar{r}(s, a)$ that assigns higher values to state-action pairs found in higher-ranked demonstrations. This acts as a guide, ensuring the learned reward function stays close to a known expertise hierarchy.
  3. Convex Optimization: A major theoretical contribution is proving that their complex objective function can be simplified into a convex optimization problem. This makes the training process much more stable and mathematically well-behaved compared to prior methods.
  4. Conservatism (CQL): To prevent the agent from becoming "overconfident" about actions not seen in the dataset (a common failure in offline learning), they incorporate a conservative term that lower-bounds the Q-values.

Datasets

The researchers evaluated SPRINQL on two major suites of environments:

  • MuJoCo: Standard continuous control tasks including HalfCheetah, Ant, Humanoid, Walker2d, and Hopper.
  • Panda-gym: Robotic arm manipulation tasks like Reach, Push, Pick-and-Place (PnP), and Slide.

Data Setup: For each task, they generated datasets with multiple levels of expertise ($N=3$ or $N=2$) by adding varying amounts of noise to an expert policy. This mimics a real-world scenario where you have a few expert trajectories and many noisy, sub-optimal ones.

Key Insights and Ideas

  • Ranking is a Signal: You don't need to know the exact reward of a sub-optimal demonstration; you just need to know that Dataset A is generally better than Dataset B. SPRINQL uses this ranking to automatically learn weights ($w_i$) for different data sources.
  • The Tractable Lower Bound: The authors found that the original "ideal" objective was not convex. They derived a new objective, $\hat{\Gamma}(Q)$, which is a lower bound of the original. Crucially, they proved this new version is concave, allowing for efficient and guaranteed optimization.
  • Reward Recovery: Unlike many imitation methods that only produce a policy (a "black box" of what to do), SPRINQL can recover a reward function that correlates highly with the ground-truth reward. This makes the model more interpretable and useful for Inverse Reinforcement Learning (IRL).

Key Results

SPRINQL significantly outperformed all baselines, including specialized methods like DemoDICE and DWBC.

  • MuJoCo Performance: On average across five tasks, SPRINQL achieved a normalized score of 78.3, while the next best baseline (W-BC) only reached 31.9.
  • Panda-gym Performance: SPRINQL averaged 68.2, compared to 44.7 for the next best (DWBC).
  • Robustness to Data Size: SPRINQL remains stable even when the amount of sub-optimal data is reduced, whereas other methods like BC and DemoDICE see their performance collapse.
  • Ablation Success: The authors showed that removing either the "Distribution Matching" or the "Reward Regularization" components led to a massive drop in performance, proving that both are essential for learning from imperfect data.

Limitations and Future Work

  • Expertise Identification: The method still relies on demonstrations being pre-classified into expertise levels. While the authors provide a way to automate weight learning, the initial "ranking" of datasets is assumed to be known.
  • Theoretical Gap: The authors note a lack of theoretical investigation into exactly how the ratio of expert to non-expert data affects the Q-learning convergence.
  • Low-Sample Regime: There is still room to explore how the reward regularizer interacts with distribution matching when expert samples are extremely sparse (e.g., only 1 or 2 trajectories).

Summary updated

Abstract

We focus on offline imitation learning (IL), which aims to mimic an expert's behavior using demonstrations without any interaction with the environment. One of the main challenges in offline IL is the limited support of expert demonstrations, which typically cover only a small fraction of the state-action space. While it may not be feasible to obtain numerous expert demonstrations, it is often possible to gather a larger set of sub-optimal demonstrations. For example, in treatment optimization problems, there are varying levels of doctor treatments available for different chronic conditions. These range from treatment specialists and experienced general practitioners to less experienced general practitioners. Similarly, when robots are trained to imitate humans in routine tasks, they might learn from individuals with different levels of expertise and efficiency. In this paper, we propose an offline IL approach that leverages the larger set of sub-optimal demonstrations while effectively mimicking expert trajectories. Existing offline IL methods based on behavior cloning or distribution matching often face issues such as overfitting to the limited set of expert demonstrations or inadvertently imitating sub-optimal trajectories from the larger dataset. Our approach, which is based on inverse soft-Q learning, learns from both expert and sub-optimal demonstrations. It assigns higher importance (through learned weights) to aligning with expert demonstrations and lower importance to aligning with sub-optimal ones. A key contribution of our approach, called SPRINQL, is transforming the offline IL problem into a convex optimization over the space of Q functions. Through comprehensive experimental evaluations, we demonstrate that the SPRINQL algorithm achieves state-of-the-art (SOTA) performance on offline IL benchmarks. Code is available at https://github.com/hmhuy0/SPRINQL .
BibTeX
@inproceedings{hoang2024sprinql,
  author = {Huy Hoang and Tien Mai and Pradeep Varakantham},
  title = {{SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=uDD44NROOt},
}
Figure