Imitation Learning from Suboptimal and Noisy Demonstrations

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2514 papers593 conceptsfrozen 2026-10-04

Standard imitation learning operates on a seductive but flawed premise: that every action in a demonstration dataset is a golden nugget of expert wisdom. In practice, human-collected data is "noisy." It is riddled with hesitations, corrective sub-movements, and outright failures. When a robot is told to "do exactly what the human did," it doesn't just learn the task; it learns the human's distractions and the system's latency.

A growing body of research is moving away from blind mimicry toward intent distillation. The goal is to treat demonstrations not as a ground-truth script, but as a source of experience to be filtered, reweighted, and refined.

The Problem of Mean-Seeking Behavior

When a dataset contains a mixture of expert plays and amateur blunders, standard Behavioral Cloning (BC) tends to "average" the actions. As Eddy Hudson and Peter Stone point out in ABC: Adversarial Behavioral Cloning for Offline Mode-Seeking Imitation Learning, this mean-seeking behavior is dangerous. If an expert goes left around an obstacle and a novice goes right, the average is a collision with the center.

The challenge is that real-world datasets are often "mixed bags" where the "pro" and "novice" labels are missing. Shiqi Lei and Jinkyoo Park argue that treating all trajectories as equally valid causes agents to inherit poor performance, a sentiment echoed by Huazhe Xu and Fisher Yu in their work on Normalized Actor-Critic (NAC).

Reweighting the Best Parts

One of the most effective ways to handle suboptimality is to "imitate the best parts." Instead of a flat loss function, researchers use weights to prioritize high-quality transitions.

Relaxed Distribution Matching

A more formal approach involves "distribution matching"—trying to make the agent's state-action occupancy match the expert's. However, strict matching forces the agent to replicate the expert's mistakes.

In Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching, Lantao Yu and Stefano Ermon introduced RelaxDICE. By using an asymmetrically-relaxed divergence, the algorithm ensures the agent stays within the "support" of the data (avoiding dangerous, unseen actions) without being forced to mimic the specific errors of suboptimal demonstrators. Similarly, Abhinav Jain and Vaibhav V. Unhelkar developed GO-DICE (GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning) to handle long-horizon tasks by combining distribution correction with hierarchical reasoning.

Distilling Intent via Structural Constraints

Sometimes, the best way to handle noise is to change how the data is represented or generated.

  1. KeyTraces and Atomic Skills: In the domain of SAT solvers, Huan Liu and Jun Chen developed ImitSAT (Boolean Satisfiability via Imitation Learning), which collapses cluttered solver runs into "KeyTraces"—concise, conflict-free paths that remove redundant computations. In robotics, the VASK framework (Gentle Manipulation Policy Learning via Demonstrations from VLM Planned Atomic Skills) uses high-level planners to generate "clean" synthetic data, bypassing the noise of human teleoperation entirely.
  2. Formal Logic Guardrails: Jasmine Jerry Aloor and Sebastian A. Scherer use Signal Temporal Logic (STL) (Follow The Rules) to provide a formal grounding. If a human demonstration violates a mandatory procedure (like an aircraft traffic pattern), the STL heuristic guides the agent back to the "intended" logic, preventing the model from internalizing the human's inconsistency.
  3. Self-Supervised "Time Reversal": Oliver Limoyo and Gregory Dudek proposed Placing via Picking (PvP) (Working Backwards: Learning to Place by Picking). By picking objects from target locations and reversing the video, they generate high-quality, noise-free "demonstrations" that outperform traditional kinesthetic teaching.

Uncertainty-Aware Trust

Finally, a key intuition is that an agent should only trust a demonstration when it is certain the demonstration is good. Yujie Zhu and Giovanni Montana developed SPReD (Uncertainty-Based Smooth Policy Regularisation), which uses ensembles of critics to create smooth, uncertainty-aware weights. Instead of a binary "imitate-or-not" filter, the agent gradually shifts its trust from the human teacher to its own experience as it discovers better policies, effectively "breaking free" from the demonstrator's mistakes.

Papers in this version

Showing 60 of 514 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 504 papers
  2. Version 1 · 2026-10-04 · 514 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_expert_optimality_assumption_168f5838,
  title = {expert optimality assumption},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expert-optimality-assumption/rdr_168f5838506f44c39ecba34a0b10249d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_imitation_learning_from_suboptimal_and_n_168f5838_v1,
  title = {Imitation Learning from Suboptimal and Noisy Demonstrations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/imitation-learning-from-suboptimal-and-noisy-demonstrations/rdr_168f5838506f44c39ecba34a0b10249d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}