Introduction
Offline Reinforcement Learning (RL) presents a unique challenge: learning an optimal policy from a static dataset without environment interaction. The primary obstacle is distribution shift. When an agent evaluates an action not present in the training data (out-of-distribution, or OOD), the value function (Q-network) often produces erroneously high estimates. Without the ability to test these actions, the agent becomes trapped in a cycle of overestimation, leading to catastrophic failure during deployment.
Traditional approaches mitigate this through regularization, forcing the learned policy to remain close to the behavior policy that generated the data. However, methods like Conservative Q-Learning (CQL) can be overly restrictive, potentially stifling the discovery of superior strategies. This paper introduces Proximity-based Bi-direction Reward Shifting (PBRS), an adaptive framework that balances safety with flexibility.
Proximity-based Bi-direction Reward Shifting (PBRS)
The core of the proposed method is Bi-direction Reward Shifting (BRS), which employs a "carrot and stick" approach to reward modification:
- The Carrot (In-Distribution): A positive bias ($b^+$) is added to the rewards of actions existing in the dataset, encouraging the exploitation of known-good behaviors.
- The Stick (Out-of-Distribution): A negative bias ($b^-$) is applied to actions the agent selects that are absent from the dataset, penalizing exploration into unreliable regions of the state-action space.
PBRS refines this by making the shift magnitude, $b_k(s)$, adaptive. It is calculated based on the KL-divergence between the current learned policy and the original behavior policy. Early in training, when the policy is likely to deviate significantly, the penalty is high. As the policy aligns with the data, the constraints are relaxed.
Theoretical and Empirical Insights
A key theoretical contribution is the demonstration that CQL is essentially a special case of BRS where reward shifts are held constant. The authors argue that constant penalties are suboptimal; Theorem 1 in the paper proves that a static, large penalty can inadvertently decrease the probability of selecting an optimal action if that action happens to be OOD. By contrast, the adaptive nature of PBRS ensures that the agent is not permanently discouraged from high-value regions.
Empirically, PBRS was evaluated on the D4RL MuJoCo benchmark. It achieved a total normalized score of 1076.3, surpassing CQL (1027.1) and IQL (1006.9). The performance was particularly strong in "medium-replay" scenarios, where the agent must distinguish between various levels of performance within a single buffer.
Analysis of Policy Diversity
One of the most significant advantages of PBRS over traditional regularization is the increased diversity of the learned policy. Because the reward shifting is adaptive and directly influences the Q-values rather than imposing a hard constraint on the actor, the agent maintains a higher degree of "action distance" from the behavior policy while remaining within safe bounds.
As shown in the figure above, PBRS allows the agent to explore a broader distribution of actions. This flexibility is crucial for surpassing the performance of the behavior policy, as it allows the agent to "stitch" together the best parts of different trajectories more effectively than a rigid, conservative model.
Conclusion
PBRS offers a dynamic alternative to static regularization in offline RL. By adaptively shifting rewards based on policy proximity, it prevents overestimation of OOD actions without sacrificing the agent's ability to improve upon the source data. While the method requires careful tuning of the scaling coefficient $c$ and accurate modeling of the behavior policy, its state-of-the-art results on D4RL suggest it is a powerful tool for robust offline learning.
Summary updated