Introduction
Inverse Reinforcement Learning (IRL) aims to recover the underlying reward function of an agent based on observed behavior. A persistent challenge in IRL is the dependence on feature representation. Traditional methods often assume a linear relationship between rewards and a set of handcrafted features. While non-parametric approaches like Gaussian Process IRL (GPIRL) introduce non-linearity, they still rely on the quality of the initial feature set. Conversely, deep learning models can learn features automatically but typically require large datasets, which are often unavailable in expert demonstration scenarios.
This paper presents DGP-IRL, a framework that utilizes Deep Gaussian Processes (DGPs) to simultaneously learn complex reward functions and the necessary feature representations from limited expert data. By stacking GP layers, the model can automatically "unwarp" raw state features into abstract representations, allowing it to outperform state-of-the-art methods even with very limited data.
Methodology: Deep Gaussian Process IRL
DGP-IRL integrates representation learning directly into the IRL process by stacking multiple GP layers. This hierarchical structure allows the model to perform a non-linear "unwarping" of raw state features ($X$) into a latent space ($D$), which is then mapped to the reward signal ($r$).
The model operates within a Maximum Entropy IRL framework, assuming that the expert's actions are stochastically optimal relative to the hidden reward. To handle the mathematical intractability of Deep GPs, the authors employ a variational inference approach using "inducing points"—a sparse set of representative data points. This Bayesian treatment provides a robust mechanism against overfitting, a common pitfall when training deep models on small demonstration sets. Unlike two-stage approaches, DGP-IRL optimizes feature extraction and reward mapping simultaneously.
Experimental Evaluation
The authors evaluated DGP-IRL against several baselines, including GPIRL, MaxEnt IRL, and Maximum Margin Planning (MMP), across three environments: Object World, Highway Driving, and a new Binary World benchmark. Binary World is particularly challenging because states that appear similar in raw feature space may have opposing rewards, necessitating sophisticated representation learning.
Quantitative Performance and Data Efficiency
A key metric for evaluation is the Expected Value Difference (EVD), which quantifies the performance gap between the learned policy and the true optimal policy. DGP-IRL demonstrates superior data efficiency, achieving low EVD with as few as 4 to 16 samples, whereas competing methods require significantly more data to reach comparable performance.
Qualitative Fidelity and Transferability
Qualitatively, DGP-IRL produces reward maps that closely mirror the ground truth, capturing sharp transitions that other methods tend to smooth over. This fidelity is a direct result of the model's ability to learn abstract representations that "untangle" the input space.
Furthermore, the learned latent representations prove highly transferable. When moved to new environment layouts without additional demonstrations, DGP-IRL maintains high performance. In the Binary World transfer task, DGP-IRL remained robust while previous state-of-the-art methods saw significant performance degradation.
Discussion and Limitations
While DGP-IRL offers significant advantages in representation learning and data efficiency, it comes with increased computational costs compared to linear models. The current implementation for task transfer uses point estimates for latent variables to maintain tractability; the authors suggest that a full Bayesian treatment for transfer could further enhance robustness at the cost of higher complexity. Future work may explore selective attention mechanisms and cross-task knowledge sharing to further leverage the hierarchical nature of Deep GPs.
Summary updated