Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching

Arnav Kumar Jain, Harley Wiltzer, Jesse Farebrother, Irina Rish, Glen Berseth, Sanjiban Choudhury

TL;DR

This paper introduces Successor Feature Matching (SFM), a novel non-adversarial approach to Inverse Reinforcement Learning (IRL) that learns expert behavior from state-only demonstrations without requiring explicit reward functions. SFM achieves state-of-the-art performance, even outperforming methods that use expert action labels, by directly optimizing successor features.

What Problem Does This Paper Address?

Inverse Reinforcement Learning (IRL) aims to infer a reward function from expert demonstrations to enable an agent to replicate expert behavior. Traditional IRL methods often frame this as an adversarial game, where an agent and an adversary compete. This adversarial approach is computationally expensive, difficult to stabilize, and often requires knowledge of expert actions, limiting its applicability to scenarios where actions are not available (e.g., video demonstrations). This paper addresses the challenge of developing a non-adversarial IRL method that can learn effectively from state-only demonstrations, offering a more stable and broadly applicable solution.

Methods

The core of SFM lies in its use of Successor Features (SFs). SFs linearly factorize the action-value function into a product of SFs and a reward vector. SFM leverages this by:

  • Direct Policy Search: Instead of learning an explicit reward function, SFM directly optimizes the policy to match the expert's successor features. This is achieved through a policy gradient descent on the gap between the learner's and expert's SFs.
  • State-Only Learning: SFM can learn from state-only demonstrations by using action-independent base features. This significantly broadens its applicability to scenarios where expert actions are not recorded.
  • Adaptive Feature Learning: SFM learns the base features adaptively from data using unsupervised RL techniques, rather than requiring them to be pre-specified.
  • Non-Adversarial Approach: By avoiding the adversarial game, SFM offers improved stability and computational efficiency compared to traditional IRL methods.

Datasets

The experiments were conducted on 10 environments from the DeepMind Control (DMC) suite.

Key Insights and Ideas

  • Successor Features as a Bridge: SFs provide a powerful mechanism to bridge the gap between expert and agent behavior without explicitly modeling the reward function.
  • Direct Policy Optimization is Sufficient: The paper demonstrates that directly optimizing the policy to match successor features is a viable and effective alternative to adversarial reward learning in IRL.
  • State-Only Imitation is Achievable and Effective: SFM's ability to learn from state-only demonstrations is a significant advancement, opening up new possibilities for imitation learning from diverse data sources.
  • Robustness to Policy Optimizers: SFM shows remarkable robustness to the choice of policy optimizer, performing well even with weaker optimizers like TD3, unlike adversarial baselines.

Key Results

  • SFM significantly outperforms baselines like Behavior Cloning (BC) and IQ-Learn, even when those baselines have access to expert action labels.
  • On average, SFM achieves a higher mean normalized return across DMC tasks, with improvements of up to 16% compared to competitors.
  • SFM demonstrates strong performance with as few as a single expert demonstration.
  • The Forward Dynamics Model (FDM) was found to be the most effective base feature function for SFM across the DMC tasks.

Limitations and Future Work

  • Exploration Problem: SFM, like other IRL methods, does not theoretically alleviate the exploration problem. The authors suggest combining SFM with resets or existing exploration algorithms as future work.
  • Checkpointing Policies: The paper notes that it's unclear how to effectively checkpoint policies in IRL tasks due to the absence of average returns, and proposes using the negative MSE between agent and demonstration SFs as a proxy.
  • Leveraging Pre-trained Features: The authors suggest that leveraging pre-trained features for more complex tasks could be a future research direction to speed up learning.

Summary updated

Abstract

In inverse reinforcement learning (IRL), an agent seeks to replicate expert demonstrations through interactions with the environment. Traditionally, IRL is treated as an adversarial game, where an adversary searches over reward models, and a learner optimizes the reward through repeated RL procedures. This game-solving approach is both computationally expensive and difficult to stabilize. In this work, we propose a novel approach to IRL by _direct policy search_: by exploiting a linear factorization of the return as the inner product of successor features and a reward vector, we design an IRL algorithm by policy gradient descent on the gap between the learner and expert features. Our non-adversarial method does not require learning an explicit reward function and can be solved seamlessly with existing RL algorithms. Remarkably, our approach works in state-only settings without expert action labels, a setting which behavior cloning (BC) cannot solve. Empirical results demonstrate that our method learns from as few as a single expert demonstration and achieves improved performance on various control tasks.
BibTeX
@inproceedings{jain2025non,
  author = {Arnav Kumar Jain and Harley Wiltzer and Jesse Farebrother and Irina Rish and Glen Berseth and Sanjiban Choudhury},
  title = {{Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=LvRQgsvd5V},
}
Figure