Introduction
Imitation Learning (IL) typically falls into two categories: Behavioral Cloning (BC), which is simple and offline but "myopic" (ignoring how actions affect future states), and Apprenticeship Learning, which is "dynamics-aware" but requires millions of interactions with a simulator. In high-stakes fields like healthcare or industrial control, simulators are often unavailable, and real-world trial-and-error is prohibited. This paper addresses the Strictly Batch Imitation Learning (SBIL) problem: learning to mimic an expert using only a fixed dataset with no environment interaction.
The EDM Framework
The authors introduce Energy-based Distribution Matching (EDM), a novel framework that is directly parameterized, dynamics-aware, and intrinsically batch. Unlike traditional methods that require an inner loop of reinforcement learning or complex off-policy evaluation, EDM provides a unified optimization objective.
The core innovation of EDM is identifying the policy's logits as an energy function for state distributions. By representing the policy $\pi_\theta(a|s)$ using a neural network that outputs logits $f_\theta(s)$, the authors define an energy function $E_\theta(s) = -\log \sum_a e^{f_\theta(s)[a]}$. This links the discriminative task (picking an action) with the generative task (modeling state visitation).
The training objective consists of two parts:
- Policy Loss: Standard cross-entropy (Behavioral Cloning).
- Occupancy Loss: A term that pushes the model to match the expert's state visitation distribution by minimizing the energy difference between expert states and model-preferred states.
To calculate the occupancy loss without a simulator, the authors use Stochastic Gradient Langevin Dynamics (SGLD). This allows the agent to sample from its "imagined" state distribution using only the gradients of the energy function, keeping the entire process 100% offline.
Experimental Evaluation
EDM was evaluated across two distinct domains: OpenAI Gym control tasks (CartPole, Acrobot, LunarLander, and BeamRider) in low-data regimes (1-15 trajectories) and the MIMIC-III healthcare dataset for clinical decision-making.
Key Results
In Gym environments, EDM consistently outperformed BC and state-of-the-art offline methods such as VDICE and DSFN. The performance gap was most dramatic in extremely low-data scenarios (1-3 trajectories). For example, in Acrobot with a single trajectory, EDM achieved nearly double the performance of BC.
On the MIMIC-III dataset, EDM achieved higher accuracy (ACC), Area Under the ROC (AUC), and Precision-Recall (APR) in matching clinical decisions compared to all benchmarks. The authors also demonstrated that EDM can benefit from "state-only" data, which is often easier to collect than full expert demonstrations.
Discussion and Insights
EDM's success is attributed to its use of multitask learning as regularization. By forcing the policy network to also model the state distribution, the agent gains a "global" awareness that prevents it from drifting into unseen regions of the state space. This energy-matching objective acts as a smoother, helping the agent understand the underlying manifold of expert behavior even when data is scarce.
Currently, EDM is limited to categorical action spaces due to the log-sum-exp formulation. Extending this framework to continuous control and further stabilizing EBM training are noted areas for future research.
Summary updated