TL;DR
Inverse Reinforcement Learning (IRL) typically attempts to learn a single, global reward function to mimic expert behavior. However, this approach often fails in long, complex tasks due to compounding errors. This paper introduces Curricular Subgoal-based IRL (CSIRL), a framework that dynamically decomposes tasks into manageable subgoals based on agent uncertainty. By learning local reward functions for these subgoals, CSIRL creates a natural curriculum that significantly outperforms state-of-the-art methods on complex robotics and autonomous driving benchmarks.
The Challenge: Global Rewards in Long-Horizon Tasks
In traditional IRL, an agent deduces an underlying reward function from expert demonstrations and uses it to learn a policy. Most existing methods attempt to learn a global reward function that evaluates an entire trajectory at once.
While effective for simple tasks, this approach struggles in complex, multi-stage environments. A minor error early in a long sequence can propagate, causing the agent's trajectory to diverge significantly from the expert's. Furthermore, global rewards face a "credit assignment" problem: in a multi-step task like pick-and-place, an agent cannot effectively learn to place an object before it has mastered picking it up. Existing methods often fail to assign appropriate credit during these intermediate stages, leading to suboptimal policies.
Methods: A Divide-and-Conquer Framework
To address these limitations, the authors propose CSIRL, which replaces the global reward with a modular, divide-and-conquer strategy. Built upon the Soft Actor-Critic (SAC) algorithm, CSIRL introduces two primary innovations:
1. Curricular Subgoal Selection via Uncertainty
Rather than using hardcoded subgoals, CSIRL autonomously identifies milestones by measuring the agent's uncertainty. The system maintains a "mixing critic pool"—an ensemble of Q-networks. By evaluating an expert trajectory through this pool, the variance in critic predictions serves as a proxy for uncertainty. CSIRL scans the trajectory and sets a subgoal where the agent's uncertainty spikes, effectively defining the current "exploration boundary" of the agent's competence.
2. Intrinsic Reward Generation via Meta-Learning
Once a subgoal is established, the agent requires a local reward to reach it. CSIRL combines a standard "matching reward" (proximity to the subgoal) with an intrinsic reward generator. This generator is trained using a meta-imitation objective, backpropagating gradients through the SAC policy evaluation and improvement steps. This process optimizes the intrinsic reward to ensure the updated policy closely matches expert actions.
Evaluation Benchmarks
The authors evaluated CSIRL across two challenging domains:
- D4RL Benchmark: Includes continuous control tasks such as AntUMaze (navigating a U-shaped maze with a multi-jointed ant) and Relocate (a complex robotic hand task requiring picking and placing a ball).
- Autonomous Driving (highway-env): A partially observable simulator featuring four scenarios of increasing difficulty: Highway-fast, Merge, Intersection, and Roundabout.
Key Insights
- Uncertainty as a Curriculum: Utilizing Q-network variance to identify the "edge" of agent competence allows the system to automatically generate a curriculum of appropriate difficulty without human intervention.
- Differentiable RL Updates: CSIRL bridges the gap between rewards and actions by unrolling RL training steps (Bellman updates) and using the chain rule to connect reward parameters directly to a behavioral cloning loss.
- Local vs. Global Efficiency: By disentangling long tasks into local segments, the agent avoids noise from future stages it is not yet ready to handle, mitigating the error propagation issues inherent in global IRL.
Results and Impact
- Superior Performance: In the difficult Roundabout driving task, where baselines like IQ-Learn and ValueDICE achieved success rates below 13%, CSIRL reached a 74% success rate.
- Robotic Dexterity: In the D4RL Relocate task, CSIRL was the only method among those tested to achieve a positive cumulative reward.
- Sample Efficiency: CSIRL maintained a significant performance lead even when trained on only a single expert trajectory.
- Interpretability: The explicit selection of subgoals allows researchers to visually monitor the agent's learning progress through different task stages.
Limitations and Future Directions
CSIRL currently assumes that expert demonstrations are optimal. If the expert data is noisy or suboptimal, uncertainty-based selection may identify unstable subgoals. Additionally, the framework is specifically designed for sequential, multi-stage tasks and may be less effective for repetitive, short-term skills. Future work will focus on extending the framework to handle suboptimal expert datasets.
Summary updated