Abstract
Learning to act from observational data without active environmental interaction is a well-known challenge in Reinforcement Learning (RL). Recent approaches involve constraints on the learned policy or conservative updates, preventing strong deviations from the state-action distribution of the dataset. Although these methods are evaluated using non-linear function approximation, theoretical justifications are mostly limited to the tabular or linear cases. Given the impressive results of deep reinforcement learning, we argue for a need to more clearly understand the challenges in this setting. In the vein of Held & Hein's classic 1963 experiment, we propose the "tandem learning" experimental paradigm which facilitates our empirical analysis of the difficulties in offline reinforcement learning. We identify function approximation in conjunction with fixed data distributions as the strongest factors, thereby extending but also challenging hypotheses stated in past work. Our results provide relevant insights for offline deep reinforcement learning, while also shedding new light on phenomena observed in the online case of learning control.
Introduction: The Passive Learning Challenge
In Offline Reinforcement Learning (RL), the goal is to train agents using pre-existing datasets without any environmental interaction. This is critical for applications like autonomous driving or healthcare, where exploration is costly or dangerous. However, "passive" learners—those trained on fixed datasets—consistently underperform compared to "active" learners that interact with their environment, even when the passive agent is trained on the exact same data generated by a successful active agent. This performance gap is termed the "Tandem Effect."
While previous literature has attributed this failure to extrapolation error and bootstrapping, this study isolates these factors to identify the fundamental cause: the interaction between non-linear function approximators (neural networks) and fixed data distributions.
Experimental Paradigm: Tandem Learning
To investigate the Tandem Effect, we introduce two experimental setups inspired by Held & Hein’s (1963) kitten experiment, which demonstrated that self-produced movement is necessary for the development of visually guided behavior.
- Tandem RL: Two agents, Active and Passive, are trained simultaneously. The Active agent interacts with the environment, populating a replay buffer. Both agents sample identical batches of data from this buffer to update their weights. Crucially, only the Active agent's actions influence future data collection.
- Forked Tandem: A successful agent is "forked" into two. The Active agent is frozen and continues to generate data, while the Passive agent attempts to maintain or improve performance using that fixed data stream.
Key Insights: Why Passive Learning Fails
Our analysis reveals several critical findings regarding the nature of deep RL:
- Bootstrapping as an Amplifier: While learning from one's own potentially erroneous value estimates (bootstrapping) is often blamed for offline RL failure, we found that even when passive agents are provided with "perfect" target values from the active agent, they still fail. Bootstrapping exacerbates the issue but is not its primary cause.
- Exploitation of Data Gaps: Neural networks are highly expressive and tend to find patterns in "gaps"—state-action pairs poorly represented in the dataset. Passive agents over-extrapolate into these gaps, creating "imaginary" high-value actions that do not translate to real-world performance.
- Interactivity as a Regularizer: Active interaction serves as a corrective feedback loop. When an active agent overestimates an action, it executes it, observes the poor outcome, and corrects its model. Passive agents lack this "reality check," leading to unconstrained model divergence.
- The Role of Non-linearity: The Tandem Effect is largely absent in linear models. It is a specific characteristic of Deep RL, where non-linear function approximation allows for wild, unconstrained extrapolations in the absence of interactive feedback.
Empirical Results
Experiments across Atari 57, Classic Control, and MinAtar benchmarks demonstrate the severity of the Tandem Effect.
- Performance Collapse: Passive agents frequently experience a total collapse in performance. In the Forked Tandem setup, even agents starting with optimal weights often see their performance plummet immediately upon being restricted to a fixed data stream.
- Policy Disagreement: After 100 million frames, passive agents disagreed with their active counterparts on the optimal action in over 75% of states.
- Interaction Threshold: We observed that allowing the passive agent to generate as little as 10-20% of its own data significantly mitigates the Tandem Effect. At a 50% interaction rate, the performance gap disappears.
Conclusion
The Tandem RL paradigm highlights that the dynamics of deep RL are fundamentally unstable on fixed data distributions. The failure of passive learning is not merely a result of "bad data" but an inherent property of how non-linear function approximators exploit gaps in static datasets. Future work in Offline RL must focus on regularizing these networks to prevent over-generalization without the benefit of a live environment's corrective feedback.
Summary updated