In the world of robotics and AI, Imitation Learning (IL) offers a powerful shortcut. Instead of forcing a robot to flail around for millions of trials to learn a task via Reinforcement Learning (RL), we can simply show it how a human does it. However, there is a catch: humans are messy. We get distracted, we fumble controllers, and our "expert" demonstrations often contain "noisy" or suboptimal actions.
Standard imitation algorithms like Behavioral Cloning (BC) assume the teacher is perfect. When fed noisy data, they tend to average out the good and the bad, resulting in a mediocre policy. A paper from researchers at Ricoh, published at ICLR 2021, proposes a clever way to filter out this noise without needing extra human labels or expensive environment interactions.
By using an iterative ensemble-weighting approach, their novel algorithm extracts optimal behavior from noisy demonstrations, effectively "seeking the mode" of optimal behavior entirely offline.
The Challenge of Noisy Demonstrations
Most state-of-the-art imitation learning methods (like GAIL or T-REX) have a high "entry fee." They either require the agent to practice in a simulator during training—which is slow and can damage real-world hardware—or they require humans to painstakingly rank demonstrations and provide confidence scores.
The simplest alternative, Behavioral Cloning, is essentially supervised learning: "see state, do action." But BC is notoriously vulnerable to noise. If 20% of your data shows a person driving into a ditch because they sneezed, a standard BC agent will learn to occasionally drive into ditches. The authors aim to solve this "noisy demonstration" problem while keeping the training process as simple, safe, and offline as standard BC.
A Generalized Behavioral Cloning Approach
The core innovation of the paper is a Generalized Behavioral Cloning objective that iteratively filters out bad advice.
-
The Weighted Objective: Instead of just maximizing the log-likelihood of all actions equally, the authors weight the objective: $$\arg \max_\theta \mathbb{E}_{s,a \sim \mathcal{D}} [\log \pi_\theta(a|s) \cdot \hat{R}(s, a)]$$ Here, $\hat{R}(s, a)$ acts as a pseudo-reward, prioritizing better actions.
-
The Iterative Trick: Because they don't have a true reward function, they use a previously learned policy ($\pi_{old}$) as the weight for the new one.
- Ensembles: To stabilize this, they train an ensemble of $K$ policies. In each iteration, the average of the previous ensemble is used to weight the training of the next ensemble.
This creates a virtuous cycle known as Mode Seeking. Because optimal actions are usually more consistent (forming the "mode" or peak of the distribution) than random noisy actions, weighting by the previous policy's probability pushes the new policy to focus even more heavily on those high-probability, optimal actions. It functions similarly to a mean-shift algorithm, shifting the learner's focus toward the peak of the expert's action distribution and ignoring the "tails" (the noise).
Key Insights and Performance
The researchers evaluated their method on standard continuous control robotics tasks (MuJoCo) and discrete control problems (OpenAI Gym). To simulate "noisy" experts, they injected optimal policies with uniform random actions, Gaussian noise, or "lazy" policies that output zeros.
- The "Noisy Regularizer": The authors mathematically demonstrate that noise in demonstrations acts as a "noisy regularizer" that confuses standard learners. By viewing BC through the lens of KL-divergence, they show that standard BC tries to minimize the distance to a mixture of both "good" and "bad" distributions, whereas their method isolates the good.
- Outperforming BC: In almost every task, the proposed algorithm significantly outperformed standard BC. For example, on the Ant-v2 task with 30% noise, the proposed method achieved near-optimal performance, while standard BC failed to reach even half the expert's score.
- Competitive with "Cheating" Baselines: Surprisingly, this offline method performed as well as (or better than) methods like 2IWIL and IC-GAIL, which are given "unfair" advantages like human-provided confidence scores or active environment interaction.
Limitations and Future Directions
While highly effective, the approach does have limitations:
- Computational Cost: Training an ensemble of $K$ policies over $M$ iterations is inherently $K \times M$ times more computationally expensive than a single run of standard BC.
- The "Strong Noise" Failure Case: If the noise is not random but instead a very consistent, "strong" non-optimal behavior (high kurtosis), the algorithm might mistakenly home in on the bad behavior instead of the good.
- State-Dependent Noise: The current model assumes the probability of a mistake is constant. In reality, humans make more mistakes in "hard" parts of a task. Exploring state-dependent noise models remains an open area for future work.
Reference: Sasaki, F., & Yamashina, R. (2021). Behavioral Cloning from Noisy Demonstrations. Published at ICLR 2021.
Summary updated