Imitating Past Successes can be Very Suboptimal

Benjamin Eysenbach, Soumith Udatha, Russ Salakhutdinov, Sergey Levine

Outcome-conditioned imitation learning methods, such as Goal-Conditioned Supervised Learning (GCSL) or Decision Transformers, offer a popular and simple alternative to complex Reinforcement Learning (RL) algorithms. However, despite their empirical success, they lack a fundamental guarantee of policy improvement. This paper reveals that these methods implicitly "average" policies across different tasks, a mechanism that can inadvertently decrease performance. To address this, the authors propose Normalized OCBC, a simple modification that "undoes" this averaging to provide a provable path to optimal policies.

The Problem with Outcome-Conditioned Behavioral Cloning

Reinforcement Learning is notoriously difficult to implement and tune, primarily because it requires learning complex value functions. A popular "shortcut" is Outcome-Conditioned Behavioral Cloning (OCBC). Instead of maximizing a reward, an agent takes past data, labels it with the outcome that actually happened (hindsight relabeling), and then trains a model to imitate those actions given that specific outcome.

While OCBC is simple and effective in many scenarios, it has a major theoretical gap: it is unclear if it is actually optimizing a control objective. Researchers have found cases where OCBC fails to solve simple tasks or even degrades as it receives more data.

To understand why, the authors decompose OCBC into a two-step mathematical procedure:

  1. The Averaging Step: The method combines experience collected from many different tasks into a single, task-agnostic policy.
  2. The Improvement Step: It then reweights this combined experience based on how well actions lead to specific outcomes.
The averaging step in OCBC can degrade performance compared to the initial policy.

The critical insight is that while the improvement step is beneficial, the averaging step can be destructive. If two tasks require very different actions in the same state, averaging them creates a "blurry" policy that might be worse than either original policy. In multi-task RL, this averaging acts as a bottleneck. If the data distribution is biased (e.g., significantly more data for one goal than another), standard OCBC becomes heavily biased toward the common goal's actions, even when attempting to reach a rare goal.

The Solution: Normalized OCBC

To fix the destructive averaging step, the authors introduce Normalized OCBC. The core idea is to learn two separate models:

  • A standard task-conditioned policy: $\pi(a | s, e)$
  • A task-agnostic "average" policy: $\pi_N(a | s)$

By taking the ratio of these two—$\frac{\pi(a | s, e)}{\pi_N(a | s)}$—the algorithm effectively "cancels out" the harmful averaging step. This ratio turns out to be proportional to the Q-function (the expected future reward). This mathematical relationship allows the agent to perform proper policy iteration without ever explicitly training a value function, enabling "value-free" RL that still enjoys the convergence guarantees of value-based methods.

Normalized OCBC successfully converges to the optimal policy, whereas standard OCBC fails due to the averaging step.

Key Results and Experiments

The authors evaluated their theory and the new algorithm across several environments, ranging from simple Bandit problems and 2D Navigation to standard GCSL benchmarks (like Lunar Lander, Pointmass, Pusher, and Door), as well as artificially biased versions of these benchmarks.

  • Failure Cases of Standard OCBC: In simple bandit and 2D navigation tasks, standard OCBC failed to find the optimal policy. In some cases, it learned a policy that was 13% worse than the starting data, directly illustrating the harm of the averaging step.
  • Performance on Balanced Benchmarks: On standard, well-balanced RL benchmarks, Normalized OCBC performed similarly to existing methods (showing a negligible +3% improvement). This explains why previous researchers hadn't widely noticed the flaw; balanced data hides the averaging bottleneck.
  • Robustness to Bias: The true advantage of Normalized OCBC emerged in "imbalanced" settings. When the action or goal distributions were skewed, Normalized OCBC outperformed the standard version by a staggering 149% (on biased actions) and 62% (on biased goals).

Limitations

While Normalized OCBC provides a strong theoretical fix, it has a few limitations. The theoretical proofs assume "perfect" learning (Bayes optimal); in practice, errors in neural network training could still affect performance. Additionally, the method requires training an additional task-agnostic model, slightly increasing computational overhead. Finally, the paper does not address the exploration problem of which tasks should be commanded during data collection to maximize learning efficiency.

Summary updated

Abstract

Prior work has proposed a simple strategy for reinforcement learning (RL): label experience with the outcomes achieved in that experience, and then imitate the relabeled experience. These outcome-conditioned imitation learning methods are appealing because of their simplicity, strong performance, and close ties with supervised learning. However, it remains unclear how these methods relate to the standard RL objective, reward maximization. In this paper, we prove that existing outcome-conditioned imitation learning methods do not necessarily improve the policy. However, we show that a simple modification results in a method that does guarantee policy improvement. Our aim is not to develop an entirely new method, but rather to explain how a variant of outcome-conditioned imitation learning can be used to maximize rewards
BibTeX
@inproceedings{eysenbach2022imitating,
  author = {Benjamin Eysenbach and Soumith Udatha and Russ Salakhutdinov and Sergey Levine},
  title = {{Imitating Past Successes can be Very Suboptimal}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=yk4A_5ZNj-},
}
Figure