Truly Batch Apprenticeship Learning with Deep Successor Features

Donghun Lee, Srivatsan Srinivasan, Finale Doshi-Velez

Abstract

We introduce a novel apprenticeship learning algorithm to learn an expert's underlying reward structure in off-policy model-free batch settings. Unlike existing methods that require a dynamics model or additional data acquisition for on-policy evaluation, our algorithm requires only the batch data of observed expert behavior. Such settings are common in real-world tasks—health care, finance or industrial processes—where accurate simulators do not exist or data acquisition is costly. To address challenges in batch settings, we introduce Deep Successor Feature Networks (DSFN) that estimate feature expectations in an off-policy setting and a transition-regularized imitation network that produces a near-expert initial policy and an efficient feature representation. Our algorithm achieves superior results in batch settings on both control benchmarks and a vital clinical task of sepsis management in the Intensive Care Unit.

Introduction

In Reinforcement Learning (RL), defining an accurate reward function is often the most challenging part of system design. Apprenticeship Learning (AL) addresses this by inferring the underlying goals of an expert through observation. However, traditional Inverse Reinforcement Learning (IRL) algorithms typically operate in "on-policy" environments, requiring a simulator or active data collection to evaluate candidate policies.

In high-stakes domains like healthcare, finance, or industrial control, simulators are often unavailable, and testing unverified policies on real systems is unethical or prohibitively expensive. We are instead restricted to "truly batch" settings: static logs of historical expert behavior. Estimating the performance of a new policy using only static data is mathematically fragile; if a policy explores states outside the expert's distribution (the "support" problem), value estimates become unreliable, leading to failure in the IRL process.

Methodology: DSFN and TRIL

To overcome the limitations of batch IRL, we propose a framework combining Deep Successor Feature Networks (DSFN) and Transition-Regularized Imitation Learning (TRIL).

The proposed framework: TRIL provides a warm-start for the IRL + DSFN loop, ensuring the policy stays near the data support.

Deep Successor Feature Networks (DSFN)

Successor features represent the expected future occupancy of state features under a specific policy. We treat the estimation of these features as a value-estimation problem, analogous to Deep Q-Networks. By utilizing neural networks and experience replay, DSFN enables the off-policy estimation of feature expectations for any candidate policy, which is critical for evaluating policies without a simulator.

Transition-Regularized Imitation Learning (TRIL)

Standard IRL often begins with a random policy, which is problematic in batch settings because random behavior deviates significantly from the expert data. TRIL provides a "warm-start" by training a dual-channel neural network to simultaneously predict the expert's action and the subsequent state. This multi-task learning forces the network to learn a feature representation ($\phi$) that captures both behavioral patterns and environment dynamics, providing a "near-expert" starting point for the IRL loop.

Experimental Evaluation

We evaluated our approach on standard OpenAI Gym benchmarks (MountainCar-v0, CartPole-v0, Acrobot-v1) and a real-world clinical dataset (MIMIC-III) for sepsis management.

Sample Efficiency in Control Tasks

In the control benchmarks, our DSFN-based approach demonstrated significant sample efficiency. As shown in the figure below, DSFN achieved near-expert performance with an order of magnitude less data compared to traditional batch IRL baselines like LSTD-$\mu$.

Performance comparison on OpenAI Gym benchmarks. DSFN (red) consistently outperforms baselines in sample efficiency across different tasks.

Sepsis Management in the ICU

On the MIMIC-III dataset, which includes records for nearly 18,000 patients, the goal was to recover the rewards guiding clinicians in administering vasopressors. DSFN achieved 79% Top-1 action matching with clinicians, significantly outperforming baselines like SCIRL (36%) and LSTD-$\mu$ (39%).

Beyond accuracy, the recovered reward functions provided clinical interpretability. The model learned to penalize inaction during periods of dangerously low blood pressure or high heart rate, and correctly identified that high-dose vasopressors are rewarded in these critical scenarios. This aligns with medical logic and demonstrates the utility of IRL in extracting actionable knowledge from clinical logs.

Conclusion and Future Work

Our framework addresses the core challenges of batch apprenticeship learning by leveraging successor features for off-policy evaluation and transition regularization for robust initialization. While the current implementation uses discretized actions for clinical tasks, future work will explore continuous action spaces. Additionally, while TRIL provides a strong starting point, the algorithm remains sensitive to the quality of the initial batch data. Integrating these components into other IRL frameworks, such as Maximum Entropy IRL, could further enhance robustness in diverse real-world applications.

Summary updated

Abstract

We introduce a novel apprenticeship learning algorithm to learn an expert's underlying reward structure in off-policy model-free \emph{batch} settings. Unlike existing methods that require a dynamics model or additional data acquisition for on-policy evaluation, our algorithm requires only the batch data of observed expert behavior. Such settings are common in real-world tasks---health care, finance or industrial processes ---where accurate simulators do not exist or data acquisition is costly. To address challenges in batch settings, we introduce Deep Successor Feature Networks(DSFN) that estimate feature expectations in an off-policy setting and a transition-regularized imitation network that produces a near-expert initial policy and an efficient feature representation. Our algorithm achieves superior results in batch settings on both control benchmarks and a vital clinical task of sepsis management in the Intensive Care Unit.
BibTeX
@inproceedings{lee2019truly,
  author = {Donghun Lee and Srivatsan Srinivasan and Finale Doshi-Velez},
  title = {{Truly Batch Apprenticeship Learning with Deep Successor Features}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/819},
}
Figure