Introduction
Offline reinforcement learning (also known as Batch RL) promises the ability to train high-performing agents from static, pre-collected datasets—such as driving logs or robot movements—without requiring further interaction with the environment. However, standard off-policy algorithms like Soft Actor-Critic (SAC) or TD3 often fail miserably in this setting.
The primary culprit is bootstrapping error. During training, an agent updates its estimate of a state's value (Q-value) based on the maximum Q-value of the next state. If the agent considers an "out-of-distribution" (OOD) action—one not present in the training dataset—the Q-function might produce a wildly inaccurate, overly optimistic value. The agent then "bootstraps" from this error, leading to a feedback loop where it pursues imaginary rewards until the policy completely diverges.
To address this, researchers introduce BEAR (Bootstrapping Error Accumulation Reduction), an algorithm that stabilizes learning by constraining the agent to only choose actions within the "support" of the training data, allowing it to outperform both standard RL and simple imitation learning.
The BEAR Algorithm
BEAR builds on the concept of distribution-constrained backups, but introduces a critical innovation in how it defines "closeness" to the training data.
Support-Set Matching vs. Distribution Matching
Previous offline RL methods, such as BCQ, attempted to make the learned policy closely match the exact distribution of the behavior policy that collected the data. BEAR argues this is too restrictive. If the data was collected by a mediocre or highly exploratory agent, copying its exact action probabilities limits the new agent's potential. Instead, BEAR simply requires the agent to stay within the range (or "support") of actions previously taken. You don't need to follow the data's exact distribution to be safe; you just need to avoid actions the Q-function knows nothing about.
Maximum Mean Discrepancy (MMD)
To enforce this support constraint without needing to know the exact probability density of the original data, BEAR uses Maximum Mean Discrepancy (MMD). MMD is a kernel-based distance measure between two sets of samples. By minimizing the MMD between the agent's proposed actions and the dataset's actions, BEAR ensures the agent stays within known regions of the action space. Using MMD with a limited number of samples effectively acts as a support constraint rather than a strict distribution constraint, providing the flexibility needed for the agent to improve upon the data collector.
Q-Ensemble
To further mitigate risk, BEAR uses an ensemble of $K$ Q-functions. During the Bellman backup, it takes a conservatively weighted minimum of these estimates, reducing the chance of overestimating the value of OOD actions.
Experimental Results
The researchers evaluated BEAR on several continuous control tasks from the MuJoCo physics engine (HalfCheetah, Walker2d, Hopper, Ant). They tested three specific types of offline datasets to measure robustness:
- Random: Data collected by a completely random policy.
- Medium: Data from a partially trained, mediocre policy (simulating imperfect human demonstrations).
- Expert: Data from a fully optimized policy.
Key Findings
- Robustness Across Data Types: BEAR was the only algorithm that performed consistently well across all data types (Random, Medium, and Expert).
- Outperforming the "Teacher": On "Medium" quality data, BEAR consistently learned policies that were significantly better than the behavior policy that generated the data. For example, in the Walker2d task, BEAR achieved a score of ~3000, while the underlying data averaged only ~1000.
- Beating Strict Distribution Matching: While previous state-of-the-art methods like BCQ performed well on expert data, they struggled on random or mediocre data. Their constraints were too aggressive, essentially forcing the agent into behavioral cloning rather than allowing it to optimize within the support set.
Limitations and Future Work
While BEAR represents a significant step forward for offline RL, challenges remain:
- Performance Degradation: In very long training runs, performance can sometimes still degrade. Developing an "early stopping" condition for RL—similar to validation sets in supervised learning—is a critical next step.
- High-Dimensional Action Spaces: While BEAR works well on standard MuJoCo tasks, scaling support-set matching to higher-dimensional or more complex action spaces remains an open challenge.
- State Distribution Shift: BEAR focuses strictly on action constraints. However, the agent might still visit states that are OOD. Future work could explore directly constraining the state distribution to further stabilize offline learning.
Summary updated