Introduction
Off-policy reinforcement learning (RL) holds immense potential for real-world applications where active data collection is prohibitive, such as healthcare or autonomous driving. While standard off-policy algorithms like Q-learning should theoretically learn from static datasets, they often fail catastrophically in practice. This failure is primarily attributed to bootstrapping error. During the Bellman backup, Q-functions are evaluated on out-of-distribution (OOD) actions suggested by the current policy. These unreliable estimates lead to compounding overestimation errors, causing Q-values to diverge and the policy to collapse.
The BEAR Algorithm: Support-Matching over Distribution-Matching
To address bootstrapping error, we introduce BEAR (Bootstrapping Error Accumulation Reduction). The core innovation of BEAR is a shift in how we constrain the learned policy relative to the behavior policy that generated the dataset.
Prior methods, such as Batch-Constrained Q-learning (BCQ), attempt to force the learned policy to match the exact distribution of the behavior policy. However, if the training data is suboptimal, this approach limits the agent to mimicking mediocre behavior. BEAR instead employs support-matching: it restricts the learned policy to choose actions that have a non-zero probability in the training data (the support set), without being forced to match the specific action frequencies of the behavior policy.
BEAR implements this constraint using the Maximum Mean Discrepancy (MMD). By calculating MMD with a small number of samples (e.g., 4 or 5), the algorithm effectively relaxes a strict distribution-matching constraint into a support-matching one. This allows the agent the flexibility to piece together optimal trajectories from diverse, even suboptimal, data sources while avoiding the pitfalls of OOD actions.
Experimental Evaluation
BEAR was evaluated on standard MuJoCo continuous control benchmarks (e.g., HalfCheetah-v2, Hopper-v2) using datasets of varying quality to test robustness:
- Random: Data collected by a completely random policy.
- Medium-Quality: Data collected by a partially trained, mediocre policy.
- Optimal: Data collected by a fully trained expert policy.
Key Results and Stability
Experimental results demonstrate that BEAR is uniquely robust across all data qualities. Unlike BCQ, which fails on random data by cloning noise, BEAR successfully extracts performant policies. On medium-quality data—the most common real-world scenario—BEAR significantly outperforms behavioral cloning and naive RL baselines.
Crucially, BEAR stabilizes the learning process. While naive RL methods often see Q-values diverge to infinity, BEAR's Q-estimates remain closely aligned with actual Monte Carlo returns. This stability is vital for reliable offline deployment where overestimation can lead to dangerous real-world behavior.
Conclusion
BEAR provides a principled framework for offline RL by identifying and mitigating bootstrapping error through support-matching. By leveraging MMD as a proxy for support, BEAR maintains the flexibility to improve upon suboptimal data while remaining grounded in the training distribution. Future work includes addressing long-term performance degradation and scaling these methods to high-dimensional, real-world datasets in robotics and autonomous driving.
Summary updated