Learning to act from observational data without active environmental interaction is a well-known challenge in Reinforcement Learning (RL). While "offline" or "passive" RL is practically vital for fields like healthcare or robotics—where trial-and-error is dangerous or expensive—it remains notoriously difficult.
Previous research often blamed "extrapolation error" (guessing the value of unseen actions) and "bootstrapping" (using your own guesses to update your guesses). However, a nagging question remained: If an active agent successfully learns a task, why can’t a passive agent learn the exact same thing by looking over the active agent's shoulder? This paper seeks to isolate the specific roles of bootstrapping, data distribution, and function approximation to find the root cause of this phenomenon, termed the "tandem effect."
The Tandem RL Framework
To investigate this, the authors introduce a novel experimental framework inspired by Held & Hein's classic 1963 kitten experiment, adapting it for RL:
- Tandem Setup: An "active" agent interacts with the environment and learns normally. A "passive" agent receives the exact same sequence of training batches as the active agent but never influences the data generation.
- Forked Tandem: An agent is trained to a high level of performance, then "forked." The active version is frozen (it just acts to generate data), while the passive version continues training on that data. This tests if an agent can even maintain performance passively.
The researchers used Double-DQN as their baseline and systematically varied components across standard benchmarks (like Atari 57 and Classic Control) to see which factors mitigated or exacerbated the learning gap.
The Root Causes of Passive Failure
The study reveals that the Tandem Effect is pervasive. Across almost all environments, the passive agent significantly underperforms the active agent, even though the data it sees is demonstrably sufficient for success.
Surprisingly, the authors found that bootstrapping is an amplifier, not the root cause. Even when the passive agent was forced to use the active agent’s "correct" targets—removing its own bootstrapping errors—the performance gap persisted.
Instead, the real culprit is the "Deadly Duo" of deep RL: the interaction between non-linear function approximation (deep neural networks) and fixed data distributions. Neural networks have a tendency to exploit gaps in fixed data, leading to wild over-extrapolations for actions not frequently taken by the data-generating policy.
In online RL, interactivity acts as a regularizer. If an agent overestimates an action, it tries it, sees the poor result, and corrects itself. In offline RL, that corrective feedback loop is missing, allowing errors to grow unchecked.
Key Findings
The empirical results strongly support this hypothesis:
- Policy Disagreement: After 100 million steps of training, active and passive agents disagreed on the best action in over 75% of states across the Atari suite.
- Maintenance Failure: In the "Forked Tandem" experiments, even when starting with a near-perfect policy, passive training caused performance to collapse rapidly.
- The Power of "Own" Data: Adding just 10-20% self-generated data to the passive agent's training batches significantly reduced the tandem effect. A 50/50 mix of "own" and "observed" data eliminated the gap entirely.
- Linear vs. Non-Linear: The tandem effect is much less severe with linear function approximation. As networks get deeper and more non-linear, the gap tends to widen, suggesting deeper models are more prone to the over-generalization that plagues offline learning.
Conclusion and Future Work
This research pinpoints the fundamental instability of passive learning in deep RL, highlighting that interactivity is crucial for correcting the over-extrapolations of non-linear models. While the study provides strong empirical evidence against prior hypotheses focused solely on bootstrapping, it also underscores the need for a rigorous theoretical framework explaining why non-linear models fail on fixed distributions. Future work will need to explore better ways to measure over-generalization directly and determine the minimum interactivity required to stabilize deep RL agents.
Summary updated