SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning

Huy Hoang, Tien Anh Mai, Pradeep Varakantham

Imitation Learning (IL) allows machines to learn by observing demonstrations. However, acquiring a comprehensive dataset of "perfect" expert behavior is often prohibitively expensive. For instance, training a surgical robot might yield only a few hours of data from a world-class surgeon, but thousands of hours from residents and medical students.

Standard algorithms typically discard this "non-expert" data or struggle to learn effectively from it. A new approach from researchers at Singapore Management University, titled SPRINQL, introduces a framework to leverage this abundant "sub-optimal" data, significantly accelerating and improving the learning process.

TL;DR

SPRINQL (Sub-optimal Demonstrations driven Reward regularized INverse soft Q Learning) is a novel offline imitation learning algorithm that utilizes a small set of expert data alongside a larger pool of ranked, sub-optimal demonstrations. By framing the problem as a stable, convex optimization over Q-functions and employing a learned "reference reward," it achieves state-of-the-art performance across standard continuous control and robotics benchmarks.

The Challenge of Limited Expert Data

Offline Imitation Learning suffers from a "data support" problem. Expert demonstrations usually cover only a narrow slice of the state-action space. When an agent deviates into unfamiliar territory, it lacks the knowledge to recover—a phenomenon known as distributional shift.

While expert data is scarce, sub-optimal data (from novices or intermediate policies) is often plentiful. The core challenge is utilizing this sub-optimal data to understand environment dynamics without inadvertently mimicking poor decisions. Existing methods often overfit to the limited expert set or fail to differentiate between high- and low-quality supplementary data.

The SPRINQL Approach

The authors introduce SPRINQL, an architecture built on four foundational pillars:

  1. Inverse Soft-Q Learning: Building on the "IQ-Learn" framework, SPRINQL avoids the instability of adversarial (GAN-style) training. Instead of alternating between training a discriminator and a policy, it solves a single optimization problem directly in the space of Q-functions.
  2. Preference-Based Reward Regularization: Assuming the data is ranked (e.g., "Expert," "Intermediate," "Novice"), the authors use a Bradley-Terry model to learn a reference reward function. This reference reward acts as a guide, indicating which state-action pairs are generally superior.
  3. Convex Lower-Bound Objective: The original mathematical objective for this problem is notoriously difficult to solve. The authors derive a new objective function that is a concave lower bound of the original. This ensures the optimization is "well-behaved" and converges to a unique, stable solution.
  4. Conservative Q-Learning (CQL): To handle the offline nature of the data, SPRINQL incorporates a penalty for actions absent from the dataset, preventing the model from becoming overconfident about unseen states.

Experimental Setup

The researchers evaluated SPRINQL on two major suites of environments:

  • MuJoCo: Classic continuous control tasks including HalfCheetah, Ant, and Humanoid.
  • Panda-gym: Robotic arm manipulation tasks including Push, PickAndPlace (PnP), and Slide.

To simulate real-world data availability, sub-optimal datasets were generated by taking expert policies and injecting varying levels of noise into their actions, creating a ranked hierarchy of expertise.

Key Results

SPRINQL significantly outperforms existing state-of-the-art (SOTA) baselines like DemoDICE, DWBC, and TRAIL.

Performance comparison of SPRINQL against baselines on MuJoCo and Panda-gym environments.

As detailed in the table above, SPRINQL achieves an average normalized score of 67.7 across the evaluated MuJoCo and Panda-gym tasks. This vastly exceeds the performance of the next best baseline, DWBC, which only reached an average of 14.6. The model demonstrated high efficiency, learning effectively even when expert data was extremely limited, provided there was sufficient sub-optimal data to map the environment.

Core Innovations

  • Sub-optimal data as a map: SPRINQL proves that "bad" data is not just a distraction. It helps the model understand the physics and dynamics of the world in areas the expert never visits, while the expert data is reserved for defining the ultimate goal.
  • Convexity ensures stability: Many imitation learning algorithms struggle with training instability. By proving their objective is concave/convex, the authors ensure SPRINQL is significantly more reliable and easier to tune than adversarial methods.
  • Reward Recovery: Unlike most IL algorithms that only produce a policy, SPRINQL can recover a reward function that correlates highly with the ground truth, making it highly valuable for Inverse Reinforcement Learning (IRL) applications.

Limitations and Future Work

The authors identify a few areas for future exploration:

  • Ranking Requirement: SPRINQL currently requires datasets to be pre-ranked by expertise level. In real-world scenarios, the exact quality of a specific demonstration might be ambiguous.
  • Data Scaling: There is currently no theoretical "rule of thumb" for the optimal ratio of expert to sub-optimal data.
  • Theoretical Depth: While the convex properties are proven, the exact interaction between the reward regularizer and distribution matching at very low data volumes warrants further study.

Code is available at: https://github.com/hmhuy0/SPRINQL

Summary updated

Abstract

We focus on offline imitation learning (IL), which aims to mimic an expert's behavior using demonstrations without any interaction with the environment. One of the main challenges in offline IL is the limited support of expert demonstrations, which typically cover only a small fraction of the state-action space. While it may not be feasible to obtain numerous expert demonstrations, it is often possible to gather a larger set of sub-optimal demonstrations. For example, in treatment optimization problems, there are varying levels of doctor treatments available for different chronic conditions. These range from treatment specialists and experienced general practitioners to less experienced general practitioners. Similarly, when robots are trained to imitate humans in routine tasks, they might learn from individuals with different levels of expertise and efficiency. In this paper, we propose an offline IL approach that leverages the larger set of sub-optimal demonstrations while effectively mimicking expert trajectories. Existing offline IL methods based on behavior cloning or distribution matching often face issues such as overfitting to the limited set of expert demonstrations or inadvertently imitating sub-optimal trajectories from the larger dataset. Our approach, which is based on inverse soft-Q learning, learns from both expert and sub-optimal demonstrations. It assigns higher importance (through learned weights) to aligning with expert demonstrations and lower importance to aligning with sub-optimal ones. A key contribution of our approach, called SPRINQL, is transforming the offline IL problem into a convex optimization over the space of Q functions. Through comprehensive experimental evaluations, we demonstrate that the SPRINQL algorithm achieves state-of-the-art (SOTA) performance on offline IL benchmarks. Code is available at https://github.com/hmhuy0/SPRINQL .
BibTeX
@inproceedings{hoang2024sprinql,
  author = {Huy Hoang and Tien Anh Mai and Pradeep Varakantham},
  title = {{SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=uDD44NROOt},
}
Figure