Abstract
Applying Reinforcement Learning (RL) to high-stakes, real-world domains often requires learning from expert data without environment interaction. This paper introduces a "truly batch" Inverse Reinforcement Learning (IRL) framework that recovers expert reward functions using only a fixed set of demonstrations. The approach combines a novel Deep Successor Feature Network (DSFN) for off-policy estimation with a Transition Regularized Imitation Learning (TRIL) warm-start. Experimental results across classical control and real-world sepsis management tasks demonstrate state-of-the-art performance and clinical interpretability.
Introduction
Standard Inverse Reinforcement Learning (IRL) typically requires on-policy rollouts to compare a proposed policy's feature expectations with those of an expert. However, in critical domains like healthcare or finance, simulators are often unavailable or unreliable, leaving only "batch" logs of expert behavior. Existing batch IRL methods frequently rely on linear models with limited representational power or require additional non-expert data. This work addresses these challenges by enabling robust, deep IRL using only expert demonstrations in a purely offline setting.
Methodology
The proposed framework consists of two primary components designed to enable batch max-margin IRL:
Deep Successor Feature Network (DSFN)
DSFN serves as an off-policy estimator for feature expectations—the discounted sum of features a policy is expected to encounter. By observing that feature expectations follow a Bellman-like equation, the authors adapt Deep Q-Network (DQN) style Temporal Difference (TD) learning to estimate these features directly from batch data. This eliminates the requirement for a simulator during the IRL process.
Transition Regularized Imitation Learning (TRIL)
Off-policy estimation can be biased if the evaluated policy deviates significantly from the data distribution. TRIL provides a "warm-start" by learning an initial near-expert policy via supervised learning. It utilizes a two-channel architecture: one channel predicts the expert's action, while the other predicts the next state (dynamics). This "transition regularization" prevents overfitting to action labels and forces the model to learn a robust feature representation ($\phi$) of the environment's physics.
Experimental Evaluation
The model was evaluated on classical control benchmarks (MountainCar-v0, CartPole-v0, Acrobot-v1) and a real-world clinical task involving sepsis management using the MIMIC-III database.
Classical Control Results
In classical control tasks, the TRIL+DSFN approach demonstrated high data efficiency, outperforming baselines even with a limited number of trajectories. The method shows robustness across different regimes, from single trajectories to large-scale datasets.
Sepsis Management
On the MIMIC-III dataset, which included 17,898 patients with 46 physiological features, DSFN achieved a 79% Top-1 action matching accuracy. This significantly outperformed traditional batch baselines such as LSTD-$\mu$ (39%) and SCIRL (36%). Furthermore, the learned reward function aligned with clinical intuition, penalizing inaction and rewarding vasopressor administration in patients showing signs of septic shock.
Discussion and Conclusion
The core innovation of this work lies in treating the estimation of a policy's "intent" as a value-estimation problem via Successor Features. By using dynamics as a regularizer in TRIL, the authors ensure that learned features capture environmental constraints, mitigating issues like covariate shift. While the current implementation focuses on discrete action spaces and uses dynamics primarily for regularization, future work may extend these methods to continuous actions and more robust model-based simulation.
Summary updated