Traditional imitation learning (IL) has long been dominated by an adversarial "cat-and-mouse" game. In frameworks like Generative Adversarial Imitation Learning (GAIL), a policy tries to fool a discriminator, while the discriminator tries to distinguish between expert and agent behavior. While powerful, this minimax optimization is notoriously unstable, prone to mode collapse, and sensitive to hyperparameter tuning.
A growing body of research is moving away from this competitive dynamic. Instead of balancing two opposing networks, these new methods transform imitation learning into a single, stable optimization problem—often a convex one. By solving for value functions or density ratios directly, these approaches achieve better convergence and performance without the adversarial headache.
The Problem with the Minimax Game
In the adversarial setup, the training signal is a moving target. As the policy improves, the discriminator must also improve to provide a useful gradient. If one outpaces the other, the gradient vanishes or explodes, leading to the "oscillations" often seen in GAN training.
Researchers like Divyansh Garg and Stefano Ermon argue in IQ-Learn: Inverse soft-Q Learning for Imitation that this instability is unnecessary. They show that the entire imitation objective can be collapsed into a single scalar minimization problem over a Q-function. By treating the Q-function as the primary object of interest, the learning process becomes a concave optimization, which is mathematically guaranteed to be more stable than a minimax game.
Direct Value Function Optimization
The core intuition behind this shift is that an expert's behavior implicitly defines a value function. If we can find a Q-function that makes the expert's actions look optimal, we don't need a separate discriminator to tell us what "good" looks like.
This idea is extended to complex settings by Huy Hoang and Tien Mai in SPRINQL, which handles sub-optimal demonstrations by framing the problem as a convex optimization over Q-functions. Similarly, in multi-agent environments, The Viet Bui and colleagues introduced Inverse Factorized Soft Q-Learning. They use "mixing networks" to factorize joint value functions, allowing multiple agents to learn from demonstrations through a unified value-function approach rather than competing against a joint discriminator.
Correcting Distributions Instead of Matching Actions
Another facet of this research direction focuses on the "stationary distribution" of the policy—essentially, how often an agent visits certain states. Instead of matching the expert's actions moment-to-moment (which leads to compounding errors), these methods try to match the expert's long-term state occupancy.
Jongmin Lee and Joelle Pineau pioneered this in OptiDICE. By transforming the reinforcement learning problem into an unconstrained convex minimization of stationary distribution corrections, they avoid the overestimation bias common in offline settings. This mathematical reduction simplifies the optimization landscape, allowing standard deep learning optimizers to reach global optima more reliably. This "DICE" (DIstribution Correction Estimation) framework has been further adapted for safety by Youngsoo Jang in SafeDICE, which uses a stable convex objective to filter out unsafe behaviors from non-expert data.
Regression and Feature Alignment
Beyond Q-learning and distribution matching, other researchers are replacing adversarial games with regression-based objectives.
- Score Matching: Runzhe Wu and Wen Sun propose SMILING, which uses the gradients of log-densities (scores) from diffusion models. By matching the "scores" of the learner's states to the expert's, they achieve a stable regression objective that avoids the mode collapse typical of GANs.
- Feature Matching: Arnav Jain and Glen Berseth show in Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching that aligning "successor features" (expected future feature visits) provides a direct path to recovering rewards without a discriminator.
- Inverse Optimization: Ioannis Dimanidis and Peyman Mohajerin Esfahani take this to the extreme in Offline Reinforcement Learning via Inverse Optimization, using hindsight information to create a non-causal expert and distilling it via a simple convex loss. This approach uses three orders of magnitude fewer parameters than standard deep RL while maintaining performance.
Why It Works: The Power of Convexity
The shift from adversarial minimax to convex optimization is more than a mathematical convenience; it fundamentally changes the training dynamics. In a minimax game, the "solution" is a Nash equilibrium, which is notoriously difficult for gradient descent to find. In a convex problem, any local optimum is a global optimum.
As noted by Luca Viano in Proximal Point Imitation Learning, applying a change of variables to the linear programming formulation of Markov Decision Processes allows us to view imitation as a single, smooth joint optimization. This replaces the "competitive" dynamics with "cooperative" ones, where cost and value parameters are optimized simultaneously toward a single goal. The result is a more robust, efficient, and theoretically grounded framework for teaching machines to learn from observation.