Imitation Learning (IL) allows machines to learn by observing demonstrations. However, acquiring a comprehensive dataset of "perfect" expert behavior is often prohibitively expensive. For instance, training a surgical robot might yield only a few hours of data from a world-class surgeon, but thousands of hours from residents and medical students.
Standard algorithms typically discard this "non-expert" data or struggle to learn effectively from it. A new approach from researchers at Singapore Management University, titled SPRINQL, introduces a framework to leverage this abundant "sub-optimal" data, significantly accelerating and improving the learning process.
TL;DR
SPRINQL (Sub-optimal Demonstrations driven Reward regularized INverse soft Q Learning) is a novel offline imitation learning algorithm that utilizes a small set of expert data alongside a larger pool of ranked, sub-optimal demonstrations. By framing the problem as a stable, convex optimization over Q-functions and employing a learned "reference reward," it achieves state-of-the-art performance across standard continuous control and robotics benchmarks.
The Challenge of Limited Expert Data
Offline Imitation Learning suffers from a "data support" problem. Expert demonstrations usually cover only a narrow slice of the state-action space. When an agent deviates into unfamiliar territory, it lacks the knowledge to recover—a phenomenon known as distributional shift.
While expert data is scarce, sub-optimal data (from novices or intermediate policies) is often plentiful. The core challenge is utilizing this sub-optimal data to understand environment dynamics without inadvertently mimicking poor decisions. Existing methods often overfit to the limited expert set or fail to differentiate between high- and low-quality supplementary data.
The SPRINQL Approach
The authors introduce SPRINQL, an architecture built on four foundational pillars:
- Inverse Soft-Q Learning: Building on the "IQ-Learn" framework, SPRINQL avoids the instability of adversarial (GAN-style) training. Instead of alternating between training a discriminator and a policy, it solves a single optimization problem directly in the space of Q-functions.
- Preference-Based Reward Regularization: Assuming the data is ranked (e.g., "Expert," "Intermediate," "Novice"), the authors use a Bradley-Terry model to learn a reference reward function. This reference reward acts as a guide, indicating which state-action pairs are generally superior.
- Convex Lower-Bound Objective: The original mathematical objective for this problem is notoriously difficult to solve. The authors derive a new objective function that is a concave lower bound of the original. This ensures the optimization is "well-behaved" and converges to a unique, stable solution.
- Conservative Q-Learning (CQL): To handle the offline nature of the data, SPRINQL incorporates a penalty for actions absent from the dataset, preventing the model from becoming overconfident about unseen states.
Experimental Setup
The researchers evaluated SPRINQL on two major suites of environments:
- MuJoCo: Classic continuous control tasks including HalfCheetah, Ant, and Humanoid.
- Panda-gym: Robotic arm manipulation tasks including Push, PickAndPlace (PnP), and Slide.
To simulate real-world data availability, sub-optimal datasets were generated by taking expert policies and injecting varying levels of noise into their actions, creating a ranked hierarchy of expertise.
Key Results
SPRINQL significantly outperforms existing state-of-the-art (SOTA) baselines like DemoDICE, DWBC, and TRAIL.
As detailed in the table above, SPRINQL achieves an average normalized score of 67.7 across the evaluated MuJoCo and Panda-gym tasks. This vastly exceeds the performance of the next best baseline, DWBC, which only reached an average of 14.6. The model demonstrated high efficiency, learning effectively even when expert data was extremely limited, provided there was sufficient sub-optimal data to map the environment.
Core Innovations
- Sub-optimal data as a map: SPRINQL proves that "bad" data is not just a distraction. It helps the model understand the physics and dynamics of the world in areas the expert never visits, while the expert data is reserved for defining the ultimate goal.
- Convexity ensures stability: Many imitation learning algorithms struggle with training instability. By proving their objective is concave/convex, the authors ensure SPRINQL is significantly more reliable and easier to tune than adversarial methods.
- Reward Recovery: Unlike most IL algorithms that only produce a policy, SPRINQL can recover a reward function that correlates highly with the ground truth, making it highly valuable for Inverse Reinforcement Learning (IRL) applications.
Limitations and Future Work
The authors identify a few areas for future exploration:
- Ranking Requirement: SPRINQL currently requires datasets to be pre-ranked by expertise level. In real-world scenarios, the exact quality of a specific demonstration might be ambiguous.
- Data Scaling: There is currently no theoretical "rule of thumb" for the optimal ratio of expert to sub-optimal data.
- Theoretical Depth: While the convex properties are proven, the exact interaction between the reward regularizer and distribution matching at very low data volumes warrants further study.
Code is available at: https://github.com/hmhuy0/SPRINQL
Summary updated