Strictly Batch Imitation Learning by Energy-based Distribution Matching

Daniel Jarrett, Ioana Bica, Mihaela van der Schaar

Introduction

Imitation Learning (IL) typically falls into two categories: Behavioral Cloning (BC), which is simple and offline but "myopic" (ignoring how actions affect future states), and Apprenticeship Learning, which is "dynamics-aware" but requires millions of interactions with a simulator. In high-stakes fields like healthcare or industrial control, simulators are often unavailable, and real-world trial-and-error is prohibited. This paper addresses the Strictly Batch Imitation Learning (SBIL) problem: learning to mimic an expert using only a fixed dataset with no environment interaction.

The EDM Framework

The authors introduce Energy-based Distribution Matching (EDM), a novel framework that is directly parameterized, dynamics-aware, and intrinsically batch. Unlike traditional methods that require an inner loop of reinforcement learning or complex off-policy evaluation, EDM provides a unified optimization objective.

Comparison of imitation learning frameworks. EDM (d) is intrinsically offline, avoiding the rollout loops required by online methods (a, b) or the complex off-policy adaptations (c).

The core innovation of EDM is identifying the policy's logits as an energy function for state distributions. By representing the policy $\pi_\theta(a|s)$ using a neural network that outputs logits $f_\theta(s)$, the authors define an energy function $E_\theta(s) = -\log \sum_a e^{f_\theta(s)[a]}$. This links the discriminative task (picking an action) with the generative task (modeling state visitation).

The training objective consists of two parts:

  1. Policy Loss: Standard cross-entropy (Behavioral Cloning).
  2. Occupancy Loss: A term that pushes the model to match the expert's state visitation distribution by minimizing the energy difference between expert states and model-preferred states.

To calculate the occupancy loss without a simulator, the authors use Stochastic Gradient Langevin Dynamics (SGLD). This allows the agent to sample from its "imagined" state distribution using only the gradients of the energy function, keeping the entire process 100% offline.

Experimental Evaluation

EDM was evaluated across two distinct domains: OpenAI Gym control tasks (CartPole, Acrobot, LunarLander, and BeamRider) in low-data regimes (1-15 trajectories) and the MIMIC-III healthcare dataset for clinical decision-making.

Key Results

In Gym environments, EDM consistently outperformed BC and state-of-the-art offline methods such as VDICE and DSFN. The performance gap was most dramatic in extremely low-data scenarios (1-3 trajectories). For example, in Acrobot with a single trajectory, EDM achieved nearly double the performance of BC.

Performance comparison across four Gym environments. EDM (orange) consistently achieves higher scaled performance with fewer trajectories compared to BC and other offline baselines.

On the MIMIC-III dataset, EDM achieved higher accuracy (ACC), Area Under the ROC (AUC), and Precision-Recall (APR) in matching clinical decisions compared to all benchmarks. The authors also demonstrated that EDM can benefit from "state-only" data, which is often easier to collect than full expert demonstrations.

Discussion and Insights

EDM's success is attributed to its use of multitask learning as regularization. By forcing the policy network to also model the state distribution, the agent gains a "global" awareness that prevents it from drifting into unseen regions of the state space. This energy-matching objective acts as a smoother, helping the agent understand the underlying manifold of expert behavior even when data is scarce.

Currently, EDM is limited to categorical action spaces due to the log-sum-exp formulation. Extending this framework to continuous control and further stabilizing EBM training are noted areas for future research.

Summary updated

BibTeX
@inproceedings{jarrett2020strictly,
  author = {Daniel Jarrett and Ioana Bica and Mihaela van der Schaar},
  title = {{Strictly Batch Imitation Learning by Energy-based Distribution Matching}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
}
Figure