Imitating Past Successes can be Very Suboptimal

Benjamin Eysenbach, Soumith Udatha, Sergey Levine, Ruslan Salakhutdinov

Abstract

Prior work has proposed a simple strategy for reinforcement learning (RL): label experience with the outcomes achieved in that experience, and then imitate the relabeled experience. These outcome-conditioned imitation learning methods are appealing because of their simplicity, strong performance, and close ties with supervised learning. However, it remains unclear how these methods relate to the standard RL objective, reward maximization. In this paper, we formally relate outcome-conditioned imitation learning to reward maximization, drawing a precise relationship between the learned policy and Q-values and explaining the close connections between these methods and prior EM-based policy search methods. This analysis shows that existing outcome-conditioned imitation learning methods do not necessarily improve the policy, but a simple modification results in a method that does guarantee policy improvement, under some assumptions.

Introduction

Reinforcement Learning (RL) is notoriously difficult to implement and tune. To bypass this, many researchers use a shortcut called Outcome-Conditioned Behavioral Cloning (OCBC). The idea is simple: take any experience gathered, label it with the outcome that actually occurred (hindsight relabeling), and then train a policy to imitate those actions given that specific outcome. While OCBC is easy to implement and performs well on some benchmarks, it lacks the theoretical "policy improvement" guarantees of standard RL. This paper investigates why OCBC often falls short of optimal performance and proposes a solution.

The Suboptimality of OCBC

The authors provide a mathematical bridge between OCBC and standard RL, decomposing the OCBC objective into a two-step process:

  1. The Averaging Step: OCBC takes experience from many different tasks and averages the policies together into a single, task-agnostic "behavioral" policy.
  2. The Improvement Step: It then reweights this average policy by task-specific Q-values (the probability of success).
OCBC decomposes into an averaging step and an improvement step. The averaging step can lead to suboptimal returns compared to the initial policies.

The critical insight is that while the second step is beneficial, the first step is often harmful. If two different tasks require different actions in the same state, averaging them together creates a "blurry" policy that is worse than the originals. The authors suggest that current OCBC methods look good on standard benchmarks only because those benchmarks use uniform goal distributions, which minimizes the negative impact of this averaging step.

Normalized OCBC

To fix this, the authors introduce Normalized OCBC. Instead of just imitating the relabeled data, they train two models: a task-conditioned policy and a task-agnostic "average" policy. By dividing the task-conditioned policy by the average one, they "undo" the harmful averaging step, effectively isolating the policy improvement component. This turns a heuristic imitation method into a provably convergent RL algorithm.

Comparison of policy trajectories: standard OCBC averages policies, leading to suboptimality, while Normalized OCBC correctly converges toward the optimal policies for each task.

Key Results

The authors evaluated their theory on several fronts, including didactic toy problems and standard goal-conditioned RL tasks like Pusher, Door, and Lunar Lander. They also tested "Biased Benchmarks" where action or goal distributions were intentionally skewed.

  • Failure in Simple Tasks: On a 2D navigation task with imbalanced goal commands, standard OCBC failed to learn the optimal path for the rarer goal. Normalized OCBC solved both perfectly.
  • Massive Gains on Biased Data: While Normalized OCBC performed similarly to standard methods on uniform benchmarks, it dominated on imbalanced datasets. On tasks with skewed action distributions, it achieved a +149% median improvement over the baseline (GCSL).
  • OCBC is EM in Disguise: The paper proves that OCBC is mathematically equivalent to reward-weighted regression (an EM-based RL method) if the averaging step is removed through normalization.

Conclusion

By identifying that OCBC implicitly averages policies across tasks, this work explains why simple imitation-based RL can be suboptimal. The proposed "Normalized OCBC" provides a simple yet effective fix. While challenges like importance sampling variance remain, this approach bridges the gap between the simplicity of supervised learning and the theoretical rigor of reinforcement learning.

Summary updated

Abstract

Prior work has proposed a simple strategy for reinforcement learning (RL): label experience with the outcomes achieved in that experience, and then imitate the relabeled experience. These outcome-conditioned imitation learning methods are appealing because of their simplicity, strong performance, and close ties with supervised learning. However, it remains unclear how these methods relate to the standard RL objective, reward maximization. In this paper, we formally relate outcome-conditioned imitation learning to reward maximization, drawing a precise relationship between the learned policy and Q-values and explaining the close connections between these methods and prior EM-based policy search methods. This analysis shows that existing outcome-conditioned imitation learning methods do not necessarily improve the policy, but a simple modification results in a method that does guarantee policy improvement, under some assumptions.
BibTeX
@inproceedings{eysenbach2022imitating,
  author = {Benjamin Eysenbach and Soumith Udatha and Sergey Levine and Ruslan Salakhutdinov},
  title = {{Imitating Past Successes can be Very Suboptimal}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=yk4A_5ZNj-},
}
Figure