Imitation Learning from Suboptimal Demonstrations

698 concepts497 papers31 proposals

Standard imitation learning operates under a comforting but fragile assumption: the demonstrator is an expert, and every observed action is optimal. Under this expert optimality assumption, the goal of behavioral cloning is simple maximum likelihood estimation—forcing the agent's policy to match the distribution of the demonstrations.

However, real-world data rarely complies. Human operators get distracted or fatigued, creating jittery or slow trajectories, as noted by Chenlin Ming et al. Crowdsourced datasets contain highly varying skill levels, and automated algorithmic solvers often get trapped in local minima (Pau Marquez Julbe et al.). When a standard imitation model is trained on this uncurated data, it suffers from two fatal flaws. First, it treats mistakes as ground-truth targets, inheriting the flaws of its teachers. Second, when faced with conflicting or multimodal actions for the same state, a deterministic policy often averages them together, outputting an entirely dysfunctional action.

To scale imitation learning to massive, low-cost, and uncurated datasets, researchers have developed techniques to break the expert optimality assumption. The core intuition is simple: instead of blindly mimicking the data, the agent must learn to differentiate between what is worth copying and what is an error.

Advantage and Value-Based Filtering

If an environment reward or an offline dataset with rewards is available, we can evaluate the quality of each transition. Instead of treating every state-action pair $(s, a)$ equally, we can weight the imitation loss by how much better an action is compared to the average behavior.

This is the core mechanism behind Monotonic Advantage Re-Weighted Imitation Learning (MARWIL), introduced by Qing Wang et al. in Exponentially Weighted Imitation Learning for Batched Historical Data. MARWIL scales the standard behavioral cloning loss by an exponential function of the estimated advantage:

$$\exp(\beta \hat{A}(s,a))$$

If an action yields a highly positive advantage, its weight is amplified; if it represents a mistake or a suboptimal choice, its advantage is negative, and its influence on the policy gradient vanishes.

Similarly, Shiye Lei et al. leverage action-values ($Q$-values) in Offline Behavior Distillation to filter massive, noisy offline datasets into a compact set of high-quality expert demonstrations. By using value functions as a soft filter, these methods allow the agent to "imitate the best parts" of a suboptimal dataset, effectively outperforming the average quality of the behavior policy. To correct these errors at deployment time, Mitsuhiko Nakamoto proposes value-guided policy steering in Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance, re-ranking action proposals to bypass the imprecise control inherited from human demonstrators.

Unsupervised Expertise and Confidence Scoring

What if we do not have access to an explicit reward function to compute advantages? In large crowdsourced datasets, such as the dataset discussed by Ruohan Zhang in Atari-HEAD: Atari Human Eye-Tracking and Demonstration Dataset, trajectories come from multiple users with completely unknown, varying skill levels. Furthermore, Chongkai Gao notes in Transferring Hierarchical Structures with Dual Meta Imitation Learning that suboptimal demonstrations severely hinder hierarchical skill discovery, leading to non-semantic primitives if left unfiltered.

To handle this, algorithms must infer the quality of the data directly from the structure of the trajectories. Songyuan Zhang et al. developed Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality, which utilizes a bi-level optimization framework to automatically assign confidence scores to different demonstrations. High-confidence trajectories are prioritized during training, while failing or random trajectories are dynamically filtered out.

Taking this a step further, Mark Beliaev et al. developed Imitation Learning by Estimating Expertise of Demonstrators. Rather than assuming human errors are merely random, high-variance noise, the authors recognize that suboptimal demonstrators often make systematic, structured mistakes. Their framework jointly learns the task policy alongside an unsupervised expertise predictor for each demonstrator. By modeling how skilled a specific demonstrator is at different regions of the state space, the framework discounts structured mistakes while absorbing optimal behaviors from the same individual when they happen to perform well.

Relaxed Distribution Matching

Another perspective frames imitation learning as state-action distribution matching. Traditional adversarial imitation methods force the agent's state-action occupancy measure to perfectly match the demonstrator's distribution. When demonstrations are suboptimal, this strict matching forces the agent to replicate errors.

To resolve this, Lantao Yu et al. introduced Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching. Their algorithm replaces strict distribution matching with an asymmetrically-relaxed $f$-divergence.

The intuition here is to change the penalty structure: the agent is heavily penalized for taking actions outside the support of the dataset (preventing dangerous, out-of-distribution behavior), but it is not penalized for failing to match the exact probability mass of the suboptimal demonstrator's mistakes. This allows the agent to remain safe within the data envelope while ignoring the specific errors embedded within it.

Trajectory Stitching and Composition

Often, a dataset does not contain a single fully optimal trajectory, but it contains many mediocre trajectories that excel in different sub-tasks. The challenge is to extract and recombine these successful fragments.

Ajay Mandlekar et al. addressed this in IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data by factorizing control into a high-level goal selector and a low-level goal-conditioned controller. This separation allows the model to "stitch together" successful segments from highly diverse and suboptimal human demonstrations to form a novel, super-expert policy.

More recently, the authors of Robust Offline Imitation Learning Through State-level Trajectory Stitching proposed State-based Behavior Retrieval (SBR), which uses a simplified world model to identify transitions in suboptimal data that lead back toward expert states. By treating suboptimal data as a map of alternative paths and error-recovery trajectories, the agent learns not just how to perform the task perfectly, but how to recover when things go wrong.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 504 papers
  2. Version 1 · 2026-10-04 · 514 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_expert_optimality_assumption_168f5838,
  title = {expert optimality assumption},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expert-optimality-assumption/rdr_168f5838506f44c39ecba34a0b10249d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_expert_optimality_assumption_168f5838_v2,
  title = {expert optimality assumption},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expert-optimality-assumption/rdr_168f5838506f44c39ecba34a0b10249d/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake