Abstract
We focus on offline imitation learning (IL), which aims to mimic an expert's behavior using demonstrations without any interaction with the environment. One of the main challenges in offline IL is the limited support of expert demonstrations, which typically cover only a small fraction of the state-action space. While it may not be feasible to obtain numerous expert demonstrations, it is often possible to gather a larger set of sub-optimal demonstrations. In this paper, we propose an offline IL approach that leverages the larger set of sub-optimal demonstrations while effectively mimicking expert trajectories. Our approach, called SPRINQL, transforms the offline IL problem into a convex optimization over the space of Q functions. Through comprehensive experimental evaluations, we demonstrate that the SPRINQL algorithm achieves state-of-the-art (SOTA) performance on offline IL benchmarks.
Introduction
In real-world applications like robotic surgery or autonomous driving, "perfect" expert data is often scarce. While we may only have a handful of expert demonstrations, we frequently possess vast amounts of "sub-optimal" data from trainees or less experienced practitioners. Standard Offline Imitation Learning (IL) methods struggle with this mix: Behavior Cloning (BC) tends to overfit to small expert sets or average out when trained on noisy data, while distribution matching methods often require massive expert datasets to remain stable.
SPRINQL (Sub-oPtimal demonstrations driven Reward regularized INverse soft Q Learning) addresses this "scarcity of experts" problem. It effectively learns from a small set of expert data combined with a much larger pool of sub-optimal demonstrations by transforming the imitation task into a stable, convex optimization problem.
Methodology
SPRINQL is built upon three primary pillars designed to handle imperfect data robustly:
- Inverse Soft-Q Learning (IQ-Learn) Foundation: Unlike unstable adversarial training (GAN-style) methods, SPRINQL builds on IQ-Learn, which learns a Q-function directly from demonstrations, providing a more stable training signal.
- Reward Regularization with Reference Rewards: The algorithm utilizes a "reference reward" $\bar{r}(s, a)$ that assigns higher values to state-action pairs found in higher-ranked demonstrations. This ensures the learned reward function aligns with a known expertise hierarchy.
- Convex Optimization: A key theoretical contribution is the derivation of a concave lower bound for the objective function. This transformation ensures that the training process is mathematically well-behaved and converges reliably.
- Conservatism (CQL): To prevent the agent from becoming overconfident about actions not present in the dataset, SPRINQL incorporates a conservative term that lower-bounds the Q-values, a technique borrowed from Conservative Q-Learning.
Experimental Setup
The researchers evaluated SPRINQL on two major suites of environments:
- MuJoCo: Continuous control tasks including HalfCheetah, Ant, Humanoid, Walker2d, and Hopper.
- Panda-gym: Robotic arm manipulation tasks like Reach, Push, Pick-and-Place (PnP), and Slide.
For each task, datasets were generated with multiple levels of expertise by adding varying amounts of noise to an expert policy, mimicking real-world scenarios with a few expert trajectories and many noisy, sub-optimal ones.
Key Results
SPRINQL significantly outperformed all baselines, including specialized methods like DemoDICE and DWBC. On average across MuJoCo tasks, SPRINQL achieved a normalized score of 78.3, while the next best baseline (W-BC) reached only 31.9. In Panda-gym, SPRINQL averaged 68.2, compared to 44.7 for the next best competitor (DWBC).
Beyond raw performance, SPRINQL demonstrated several key advantages:
- Robustness to Data Size: Performance remains stable even when the amount of sub-optimal data is reduced, whereas other methods see their performance collapse.
- Reward Recovery: SPRINQL can recover a reward function that correlates highly with the ground-truth reward, making the model more interpretable than "black box" policy-only methods.
- Ablation Success: Experiments showed that both Distribution Matching and Reward Regularization are essential; removing either component leads to a massive drop in performance.
Conclusion and Future Work
SPRINQL provides a robust framework for offline IL by effectively utilizing ranked sub-optimal data. While it achieves SOTA results, some limitations remain for future exploration. The method currently relies on demonstrations being pre-classified into expertise levels. Future work could investigate how to automate this ranking or explore the theoretical limits of the expert-to-non-expert data ratio. Additionally, the performance in extremely low-sample regimes (e.g., only 1 or 2 expert trajectories) remains an open area for further refinement.
Summary updated