Non-Adversarial Imitation Learning via Convex Duality

30 concepts18 papers1 proposal

In traditional imitation learning, teaching an agent to behave like an expert typically requires solving a complex, two-player game. Algorithms like Generative Adversarial Imitation Learning (GAIL) set up a competitive loop: a discriminator tries to distinguish between expert trajectories and agent trajectories, while a policy tries to fool the discriminator.

Mathematically, this is framed as a minimax (or saddle-point) optimization problem:

$$\min_{\pi} \max_{D} \mathbb{E}_{\pi}[D(s,a)] - \mathbb{E}_{\pi_E}[D(s,a)]$$

While elegant in theory, this adversarial setup inherits all the notorious instabilities of Generative Adversarial Networks (GANs). The gradient steps of the policy directly conflict with the steps of the discriminator, leading to training oscillations, catastrophic forgetting, and extreme sensitivity to hyperparameters.

A rapidly growing body of research bypasses this adversarial bottleneck entirely. By leveraging mathematical dualities—such as Lagrangian duality, the linear programming formulation of Markov Decision Processes (MDPs), and inverse Bellman operators—researchers have shown that imitation learning and offline policy optimization can be rewritten as single-objective, stable convex minimization problems.

Collapsing the Game via Inverse Q-Learning

The core insight behind avoiding a discriminator is that an agent's policy ($\pi$) and its value function ($Q$) are deeply intertwined. If we can find a value function that implicitly makes the expert's actions look optimal, we do not need to learn an explicit reward function or run an adversarial discriminator loop.

In IQ-Learn: Inverse soft-Q Learning for Imitation, Divyansh Garg et al. demonstrated how to operationalize this insight using the inverse Bellman operator. Instead of alternating between updating a reward function and a policy, the authors showed that the entire imitation objective can be transformed into a single scalar minimization problem directly over the Q-function. Because the objective becomes concave with respect to the Q-value parameters, the training landscape is smoothed out, eliminating the "cat-and-mouse" dynamics of adversarial training. This concept was later integrated into online planning architectures; for instance, Shangzhe Li et al. introduced Reward-free World Models for Online Imitation Learning (IQ-MPC), which plans directly within a learned latent Q-space to bypass explicit reward estimation entirely.

Adversarial IL (e.g., GAIL):
[Policy (Generator)] <---> [Reward (Discriminator)]  (Unstable Min-Max Game)

Convex/Inverse Q-Learning:
[Expert Data] ---> [Direct Q-Function Optimization] ---> [Policy Extraction] (Stable Minimization)

This direct optimization approach proves highly resilient when dealing with imperfect data. In real-world scenarios, pure expert demonstrations are rare; datasets are often flooded with suboptimal or noisy behaviors. To address this, Huy Hoang et al. proposed SPRINQL and SubIQ. The authors formulated imitation learning as a convex optimization problem over Q-functions that explicitly accounts for sub-optimality, allowing agents to robustly recover the optimal policy even when the bulk of the training data comes from non-expert sources.

The benefits of this paradigm also scale to complex multi-agent systems. In Multi-Agent Imitation by Learning and Sampling from Factorized Soft Q-Function, Yi-Chen Li et al. eliminated adversarial training in multi-agent environments by factorizing the joint soft Q-function and treating the optimal policy as an energy-based model, providing stable training curves across both discrete and continuous tasks.

The Dual MDP Perspective: The DICE Family

A parallel strategy for removing adversarial instability involves changing what we optimize. Instead of optimizing the policy parameters directly, we can optimize the stationary distribution (the long-term frequency of visiting specific state-action pairs) of the target policy.

This approach is rooted in the linear programming dual formulation of MDPs. In OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation, Jongmin Lee et al. tackled offline reinforcement learning by estimating the stationary distribution correction ratio. By deriving a closed-form solution for the inner maximization step of the dual MDP, the authors compressed a complex minimax problem into a single, unconstrained convex minimization objective. This mathematical reduction allows standard deep learning optimizers to converge reliably without requiring hand-tuned conservative penalties.

The "DICE" framework has proven to be a versatile mathematical Swiss Army knife for non-adversarial learning:

  • Constrained Safety: In COptiDICE, Jongmin Lee et al. extended this dual formulation to Constrained MDPs. By reducing the problem to a single convex objective, they eliminated the need for the unstable, alternating optimization loops typically required to balance actors, critics, and safety Lagrange multipliers.
  • Learning from Observation (LfO): When action labels are missing from expert trajectories, Geon-Hyeong Kim et al. introduced LobsDICE, which uses Lagrange duality to transform state-only offline imitation into a stable convex minimization task.
  • Negative Demonstrations: In SafeDICE, Youngsoo Jang et al. leveraged non-preferred (unsafe) demonstrations alongside unlabeled data, using a single-stage convex optimization to mathematically filter out dangerous behaviors.

Why Convexity Changes Everything

To understand why this shift works so well, consider how standard deep learning optimizers like Adam or SGD behave. These optimizers are fundamentally designed to roll down hills toward a local or global minimum. When forced into a minimax game, the gradient steps of the two players can endlessly circle the optimum or diverge entirely, because an update for the policy alters the loss landscape for the discriminator.

By applying Lagrangian duality or inverse optimization losses, the problem is structurally altered. For example, in Offline Reinforcement Learning via Inverse Optimization, Ioannis Dimanidis et al. framed policy training as an inverse optimization task where a Q-function is fitted to expert actions using a convex sub-optimality loss. This formulation guarantees a global optimum within the chosen hypothesis class and completely avoids the divergence issues associated with non-convex Bellman residual minimization.

Even outside of value-function and distribution matching, the trend toward non-adversarial objectives holds. In Diffusing States and Matching Scores: A New Framework for Imitation Learning, Runzhe Wu et al. introduced SMILING, a framework that replaces adversarial objectives with a regression-based score-matching loss inspired by diffusion models. By matching the gradients of log-densities between the expert and learner states, they achieve stable training free from sudden performance oscillations or mode collapse.

Ultimately, whether through inverse Bellman operators, stationary distribution corrections, or score matching, the core takeaway remains the same: transforming a competitive two-player game into a cooperative single-objective minimization unlocks the full, stable potential of standard gradient-based optimization.

Key Foundational Work

SQUIRL: Robust and Efficient Learning from Video Demonstration of Long-Horizon Robotic Manipulation Tasks (2020) — contributed 2 concept elements to this direction.
Fisher Efficient Inference of Intractable Models (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_non_adversarial_imitation_learning_cd2eda27,
  title = {non-adversarial imitation learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-adversarial-imitation-learning/rdr_cd2eda2797fd48db97468d79c79f1c82},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_non_adversarial_imitation_learning_cd2eda27_v1,
  title = {non-adversarial imitation learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-adversarial-imitation-learning/rdr_cd2eda2797fd48db97468d79c79f1c82/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake