A Critique of Strictly Batch Imitation Learning

Gokul Swamy, Sanjiban Choudhury, J. Andrew Bagnell, Zhiwei Steven WuarXiv 2021 · OpenReview

TL;DR

This paper provides a sharp theoretical critique of Energy-based Distribution Matching (EDM), a recently proposed method for offline imitation learning. The authors demonstrate that EDM's objective function relies on a "pseudo-state distribution" that is fundamentally disconnected from the policy's true behavior, which can cause the algorithm to actively move away from the expert policy—even when provided with infinite data.

What Problem Does This Paper Address?

Offline imitation learning (IL) aims to mimic an expert's behavior using a pre-collected dataset of demonstrations, without any further interaction with the environment. The standard baseline, Behavioral Cloning (BC), treats this as supervised learning but famously suffers from compounding errors when the agent drifts into states unseen during training.

To fix this, recent work proposed Energy-Based Distribution Matching (EDM), which attempts to frame offline IL as a joint energy-based model. The goal of EDM is to match the expert's state-action distribution. However, this paper addresses a critical flaw in that formulation: without access to the environment's transition dynamics, it is impossible to truly calculate how a policy change affects state visitation. The authors tackle the misconception that mathematical convenience can bypass the need for environment dynamics in distribution matching.

Methods

As a purely theoretical critique, this paper does not introduce a new algorithm. Instead, it uses mathematical analysis and rigorous counterexamples to expose flaws in the EDM objective function.

The authors dissect the EDM objective, which minimizes the KL divergence from the expert's distribution to a parameterized model distribution ($p_{\pi_\theta}$). This objective splits into two terms:

  1. A standard Behavioral Cloning loss.
  2. A state-distribution loss ($\log p_\theta(s)$).

The authors construct simple, analytical Markov Decision Processes (MDPs) to evaluate how these terms behave under standard policy parameterizations (like the exponential/softmax family). They specifically analyze the gradients of these terms to see if they genuinely guide a learner toward the expert policy.

Datasets

Not discussed in the paper. The paper is entirely theoretical and relies on minimal, hand-constructed analytical MDP examples rather than empirical datasets or benchmarks.

Key Insights and Ideas

  • The Pseudo-Distribution Trap: EDM introduces a mathematically valid probability distribution over states ($p_\theta(s)$), but the authors show this is a "pseudo-distribution." Because it ignores environment transition probabilities, it has no inherent connection to the policy's true state visitation distribution ($d_\pi(s)$).
  • Invariance Flaws: Because softmax policies are invariant to additive shifts in logits, you can alter the pseudo-distribution $p_\theta(s)$ to be completely arbitrary (e.g., a perfectly uniform distribution over all states) without changing the underlying policy or its true state visitation at all.
  • Harmful Parameter Coupling: EDM forces a single set of parameters ($\theta$) to represent both the policy and the state distribution. The authors prove this coupling backfires: optimizing the state distribution term can generate gradients that fight against the behavioral cloning term, pushing the parameters away from the expert.

Key Results

  • Example 1 & 2 (Notational Disconnect): The authors prove that in an MDP with no transitions between different states, EDM's calculated state distribution yields a uniform distribution ($1/|S|$), whereas the true policy visitation is a singleton distribution focused entirely on the initial state ($s_0$).
  • Theorem 1 (Inconsistency Proof): The authors prove that there exists an MDP where the gradient of the EDM objective is non-zero at the exact parameter values of the expert policy ($\nabla_\theta \log p_\theta(s_2)|_{\theta=1} \approx -0.245 \neq 0$). This means even if the algorithm perfectly discovers the expert policy, the gradient will push it away, demonstrating that EDM is statistically inconsistent (unlike Behavioral Cloning).

Limitations and Future Work

As a critique, the paper's primary limitation is that it does not propose a corrected, fully offline distribution-matching algorithm; it only diagnoses why the current energy-based framing fails. The authors note that the creators of EDM acknowledged these notational and conceptual issues in a subsequent blog post response. The underlying takeaway points toward a fundamental limitation of strictly batch imitation learning: true state-distribution matching inherently requires either environment interaction or an explicit model of environment transitions.

Summary updated

Abstract

Recent work by Jarrett et al. attempts to frame the problem of offline imitation learning (IL) as one of learning a joint energy-based model, with the hope of out-performing standard behavioral cloning. We suggest that notational issues obscure how the psuedo-state visitation distribution the authors propose to optimize might be disconnected from the policy's $\textit{true}$ state visitation distribution. We further construct natural examples where the parameter coupling advocated by Jarrett et al. leads to inconsistent estimates of the expert's policy, unlike behavioral cloning.
BibTeX
@misc{swamy2021critique,
  author = {Gokul Swamy and Sanjiban Choudhury and J. Andrew Bagnell and Zhiwei Steven Wu},
  title = {{A Critique of Strictly Batch Imitation Learning}},
  year = {2021},
  eprint = {2110.02063},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.02063},
}
Figure