Truly Batch Apprenticeship Learning with Deep Successor Features

Donghun Lee, Srivatsan Srinivasan, Finale Doshi-Velez

Abstract

Applying Reinforcement Learning (RL) to high-stakes, real-world domains often requires learning from expert data without environment interaction. This paper introduces a "truly batch" Inverse Reinforcement Learning (IRL) framework that recovers expert reward functions using only a fixed set of demonstrations. The approach combines a novel Deep Successor Feature Network (DSFN) for off-policy estimation with a Transition Regularized Imitation Learning (TRIL) warm-start. Experimental results across classical control and real-world sepsis management tasks demonstrate state-of-the-art performance and clinical interpretability.

Introduction

Standard Inverse Reinforcement Learning (IRL) typically requires on-policy rollouts to compare a proposed policy's feature expectations with those of an expert. However, in critical domains like healthcare or finance, simulators are often unavailable or unreliable, leaving only "batch" logs of expert behavior. Existing batch IRL methods frequently rely on linear models with limited representational power or require additional non-expert data. This work addresses these challenges by enabling robust, deep IRL using only expert demonstrations in a purely offline setting.

Methodology

The proposed framework consists of two primary components designed to enable batch max-margin IRL:

Deep Successor Feature Network (DSFN)

DSFN serves as an off-policy estimator for feature expectations—the discounted sum of features a policy is expected to encounter. By observing that feature expectations follow a Bellman-like equation, the authors adapt Deep Q-Network (DQN) style Temporal Difference (TD) learning to estimate these features directly from batch data. This eliminates the requirement for a simulator during the IRL process.

Transition Regularized Imitation Learning (TRIL)

Off-policy estimation can be biased if the evaluated policy deviates significantly from the data distribution. TRIL provides a "warm-start" by learning an initial near-expert policy via supervised learning. It utilizes a two-channel architecture: one channel predicts the expert's action, while the other predicts the next state (dynamics). This "transition regularization" prevents overfitting to action labels and forces the model to learn a robust feature representation ($\phi$) of the environment's physics.

Overview of the TRIL and DSFN framework, showing the warm-start process and the iterative IRL loop.

Experimental Evaluation

The model was evaluated on classical control benchmarks (MountainCar-v0, CartPole-v0, Acrobot-v1) and a real-world clinical task involving sepsis management using the MIMIC-III database.

Classical Control Results

In classical control tasks, the TRIL+DSFN approach demonstrated high data efficiency, outperforming baselines even with a limited number of trajectories. The method shows robustness across different regimes, from single trajectories to large-scale datasets.

Performance comparison of DSFN against baselines across three classical control tasks as a function of the number of expert trajectories.

Sepsis Management

On the MIMIC-III dataset, which included 17,898 patients with 46 physiological features, DSFN achieved a 79% Top-1 action matching accuracy. This significantly outperformed traditional batch baselines such as LSTD-$\mu$ (39%) and SCIRL (36%). Furthermore, the learned reward function aligned with clinical intuition, penalizing inaction and rewarding vasopressor administration in patients showing signs of septic shock.

Discussion and Conclusion

The core innovation of this work lies in treating the estimation of a policy's "intent" as a value-estimation problem via Successor Features. By using dynamics as a regularizer in TRIL, the authors ensure that learned features capture environmental constraints, mitigating issues like covariate shift. While the current implementation focuses on discrete action spaces and uses dynamics primarily for regularization, future work may extend these methods to continuous actions and more robust model-based simulation.

Summary updated

Abstract

We introduce a novel apprenticeship learning algorithm to learn an expert's underlying reward structure in off-policy model-free \emph{batch} settings. Unlike existing methods that require a dynamics model or additional data acquisition for on-policy evaluation, our algorithm requires only the batch data of observed expert behavior. Such settings are common in real-world tasks---health care, finance or industrial processes ---where accurate simulators do not exist or data acquisition is costly. To address challenges in batch settings, we introduce Deep Successor Feature Networks(DSFN) that estimate feature expectations in an off-policy setting and a transition-regularized imitation network that produces a near-expert initial policy and an efficient feature representation. Our algorithm achieves superior results in batch settings on both control benchmarks and a vital clinical task of sepsis management in the Intensive Care Unit.
BibTeX
@inproceedings{lee2019truly,
  author = {Donghun Lee and Srivatsan Srinivasan and Finale Doshi-Velez},
  title = {{Truly Batch Apprenticeship Learning with Deep Successor Features}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/819},
}
Figure