Curricular Subgoals for Inverse Reinforcement Learning

Shunyu Liu, Yunpeng Qing, Shuqi Xu, Hongyan Wu, Jiangtao Zhang, Jingyuan Cong, Tianhao Chen, Yunfu Liu, Mingli Song

TL;DR

Inverse Reinforcement Learning (IRL) typically attempts to learn a single, global reward function to mimic expert behavior. However, this approach often fails in long, complex tasks due to compounding errors. This paper introduces Curricular Subgoal-based IRL (CSIRL), a framework that dynamically decomposes tasks into manageable subgoals based on agent uncertainty. By learning local reward functions for these subgoals, CSIRL creates a natural curriculum that significantly outperforms state-of-the-art methods on complex robotics and autonomous driving benchmarks.

The Challenge: Global Rewards in Long-Horizon Tasks

In traditional IRL, an agent deduces an underlying reward function from expert demonstrations and uses it to learn a policy. Most existing methods attempt to learn a global reward function that evaluates an entire trajectory at once.

While effective for simple tasks, this approach struggles in complex, multi-stage environments. A minor error early in a long sequence can propagate, causing the agent's trajectory to diverge significantly from the expert's. Furthermore, global rewards face a "credit assignment" problem: in a multi-step task like pick-and-place, an agent cannot effectively learn to place an object before it has mastered picking it up. Existing methods often fail to assign appropriate credit during these intermediate stages, leading to suboptimal policies.

Comparison of global reward structures versus the proposed local, subgoal-based approach.

Methods: A Divide-and-Conquer Framework

To address these limitations, the authors propose CSIRL, which replaces the global reward with a modular, divide-and-conquer strategy. Built upon the Soft Actor-Critic (SAC) algorithm, CSIRL introduces two primary innovations:

1. Curricular Subgoal Selection via Uncertainty

Rather than using hardcoded subgoals, CSIRL autonomously identifies milestones by measuring the agent's uncertainty. The system maintains a "mixing critic pool"—an ensemble of Q-networks. By evaluating an expert trajectory through this pool, the variance in critic predictions serves as a proxy for uncertainty. CSIRL scans the trajectory and sets a subgoal where the agent's uncertainty spikes, effectively defining the current "exploration boundary" of the agent's competence.

2. Intrinsic Reward Generation via Meta-Learning

Once a subgoal is established, the agent requires a local reward to reach it. CSIRL combines a standard "matching reward" (proximity to the subgoal) with an intrinsic reward generator. This generator is trained using a meta-imitation objective, backpropagating gradients through the SAC policy evaluation and improvement steps. This process optimizes the intrinsic reward to ensure the updated policy closely matches expert actions.

The CSIRL architecture, showing the interaction process, subgoal selection, and reward generation.

Evaluation Benchmarks

The authors evaluated CSIRL across two challenging domains:

  • D4RL Benchmark: Includes continuous control tasks such as AntUMaze (navigating a U-shaped maze with a multi-jointed ant) and Relocate (a complex robotic hand task requiring picking and placing a ball).
  • Autonomous Driving (highway-env): A partially observable simulator featuring four scenarios of increasing difficulty: Highway-fast, Merge, Intersection, and Roundabout.

Key Insights

  • Uncertainty as a Curriculum: Utilizing Q-network variance to identify the "edge" of agent competence allows the system to automatically generate a curriculum of appropriate difficulty without human intervention.
  • Differentiable RL Updates: CSIRL bridges the gap between rewards and actions by unrolling RL training steps (Bellman updates) and using the chain rule to connect reward parameters directly to a behavioral cloning loss.
  • Local vs. Global Efficiency: By disentangling long tasks into local segments, the agent avoids noise from future stages it is not yet ready to handle, mitigating the error propagation issues inherent in global IRL.

Results and Impact

  • Superior Performance: In the difficult Roundabout driving task, where baselines like IQ-Learn and ValueDICE achieved success rates below 13%, CSIRL reached a 74% success rate.
  • Robotic Dexterity: In the D4RL Relocate task, CSIRL was the only method among those tested to achieve a positive cumulative reward.
  • Sample Efficiency: CSIRL maintained a significant performance lead even when trained on only a single expert trajectory.
  • Interpretability: The explicit selection of subgoals allows researchers to visually monitor the agent's learning progress through different task stages.

Limitations and Future Directions

CSIRL currently assumes that expert demonstrations are optimal. If the expert data is noisy or suboptimal, uncertainty-based selection may identify unstable subgoals. Additionally, the framework is specifically designed for sequential, multi-stage tasks and may be less effective for repetitive, short-term skills. Future work will focus on extending the framework to handle suboptimal expert datasets.

Summary updated

Abstract

Inverse Reinforcement Learning (IRL) aims to reconstruct the reward function from expert demonstrations to facilitate policy learning, and has demonstrated its remarkable success in imitation learning. To promote expert-like behavior, existing IRL methods mainly focus on learning global reward functions to minimize the trajectory difference between the imitator and the expert. However, these global designs are still limited by the redundant noise and error propagation problems, leading to the unsuitable reward assignment and thus downgrading the agent capability in complex multi-stage tasks. In this paper, we propose a novel Curricular Subgoal-based Inverse Reinforcement Learning (CSIRL) framework, that explicitly disentangles one task with several local subgoals to guide agent imitation. Specifically, CSIRL firstly introduces decision uncertainty of the trained agent over expert trajectories to dynamically select specific states as subgoals, which directly determines the exploration boundary of different task stages. To further acquire local reward functions for each stage, we customize a meta-imitation objective based on these curricular subgoals to train an intrinsic reward generator. Experiments on the D4RL and autonomous driving benchmarks demonstrate that the proposed methods yields results superior to the state-of-the-art counterparts, as well as better interpretability. Our code is publicly available at https://github.com/Plankson/CSIRL.
BibTeX
@article{liu2025curricular,
  author = {Shunyu Liu and Yunpeng Qing and Shuqi Xu and Hongyan Wu and Jiangtao Zhang and Jingyuan Cong and Tianhao Chen and Yunfu Liu and Mingli Song},
  title = {{Curricular Subgoals for Inverse Reinforcement Learning}},
  journal = {IEEE Trans. Intell. Transp. Syst.},
  year = {2025},
  doi = {10.1109/tits.2025.3532519},
}
Figure