The Difficulty of Passive Learning in Deep Reinforcement Learning

Georg Ostrovski, Pablo Samuel Castro, Will Dabney

Learning to act from observational data without active environmental interaction is a well-known challenge in Reinforcement Learning (RL). While "offline" or "passive" RL is practically vital for fields like healthcare or robotics—where trial-and-error is dangerous or expensive—it remains notoriously difficult.

Previous research often blamed "extrapolation error" (guessing the value of unseen actions) and "bootstrapping" (using your own guesses to update your guesses). However, a nagging question remained: If an active agent successfully learns a task, why can’t a passive agent learn the exact same thing by looking over the active agent's shoulder? This paper seeks to isolate the specific roles of bootstrapping, data distribution, and function approximation to find the root cause of this phenomenon, termed the "tandem effect."

The Tandem RL Framework

To investigate this, the authors introduce a novel experimental framework inspired by Held & Hein's classic 1963 kitten experiment, adapting it for RL:

The Tandem and Forked Tandem experimental setups
  • Tandem Setup: An "active" agent interacts with the environment and learns normally. A "passive" agent receives the exact same sequence of training batches as the active agent but never influences the data generation.
  • Forked Tandem: An agent is trained to a high level of performance, then "forked." The active version is frozen (it just acts to generate data), while the passive version continues training on that data. This tests if an agent can even maintain performance passively.

The researchers used Double-DQN as their baseline and systematically varied components across standard benchmarks (like Atari 57 and Classic Control) to see which factors mitigated or exacerbated the learning gap.

The Root Causes of Passive Failure

The study reveals that the Tandem Effect is pervasive. Across almost all environments, the passive agent significantly underperforms the active agent, even though the data it sees is demonstrably sufficient for success.

Surprisingly, the authors found that bootstrapping is an amplifier, not the root cause. Even when the passive agent was forced to use the active agent’s "correct" targets—removing its own bootstrapping errors—the performance gap persisted.

Instead, the real culprit is the "Deadly Duo" of deep RL: the interaction between non-linear function approximation (deep neural networks) and fixed data distributions. Neural networks have a tendency to exploit gaps in fixed data, leading to wild over-extrapolations for actions not frequently taken by the data-generating policy.

In online RL, interactivity acts as a regularizer. If an agent overestimates an action, it tries it, sees the poor result, and corrects itself. In offline RL, that corrective feedback loop is missing, allowing errors to grow unchecked.

Key Findings

The empirical results strongly support this hypothesis:

Performance gap between active and passive agents in Breakout
  • Policy Disagreement: After 100 million steps of training, active and passive agents disagreed on the best action in over 75% of states across the Atari suite.
  • Maintenance Failure: In the "Forked Tandem" experiments, even when starting with a near-perfect policy, passive training caused performance to collapse rapidly.
  • The Power of "Own" Data: Adding just 10-20% self-generated data to the passive agent's training batches significantly reduced the tandem effect. A 50/50 mix of "own" and "observed" data eliminated the gap entirely.
  • Linear vs. Non-Linear: The tandem effect is much less severe with linear function approximation. As networks get deeper and more non-linear, the gap tends to widen, suggesting deeper models are more prone to the over-generalization that plagues offline learning.

Conclusion and Future Work

This research pinpoints the fundamental instability of passive learning in deep RL, highlighting that interactivity is crucial for correcting the over-extrapolations of non-linear models. While the study provides strong empirical evidence against prior hypotheses focused solely on bootstrapping, it also underscores the need for a rigorous theoretical framework explaining why non-linear models fail on fixed distributions. Future work will need to explore better ways to measure over-generalization directly and determine the minimum interactivity required to stabilize deep RL agents.

Summary updated

Abstract

Learning to act from observational data without active environmental interaction is a well-known challenge in Reinforcement Learning (RL). Recent approaches involve constraints on the learned policy or conservative updates, preventing strong deviations from the state-action distribution of the dataset. Although these methods are evaluated using non-linear function approximation, theoretical justifications are mostly limited to the tabular or linear cases. Given the impressive results of deep reinforcement learning, we argue for a need to more clearly understand the challenges in this setting. In the vein of Held & Hein's classic 1963 experiment, we propose the "tandem learning" experimental paradigm which facilitates our empirical analysis of the difficulties in offline reinforcement learning. We identify function approximation in conjunction with fixed data distributions as the strongest factors, thereby extending but also challenging hypotheses stated in past work. Our results provide relevant insights for offline deep reinforcement learning, while also shedding new light on phenomena observed in the online case of learning control.
BibTeX
@inproceedings{ostrovski2021difficulty,
  author = {Georg Ostrovski and Pablo Samuel Castro and Will Dabney},
  title = {{The Difficulty of Passive Learning in Deep Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=nPHA8fGicZk},
}
Figure