Adaptive Reward Shifting Based on Behavior Proximity for Offline Reinforcement Learning

Zhe Zhang, Xiaoyang TanIJCAI 2023 · OpenReview

Introduction

Offline Reinforcement Learning (RL) presents a unique challenge: learning an optimal policy from a static dataset without environment interaction. The primary obstacle is distribution shift. When an agent evaluates an action not present in the training data (out-of-distribution, or OOD), the value function (Q-network) often produces erroneously high estimates. Without the ability to test these actions, the agent becomes trapped in a cycle of overestimation, leading to catastrophic failure during deployment.

Traditional approaches mitigate this through regularization, forcing the learned policy to remain close to the behavior policy that generated the data. However, methods like Conservative Q-Learning (CQL) can be overly restrictive, potentially stifling the discovery of superior strategies. This paper introduces Proximity-based Bi-direction Reward Shifting (PBRS), an adaptive framework that balances safety with flexibility.

Proximity-based Bi-direction Reward Shifting (PBRS)

The core of the proposed method is Bi-direction Reward Shifting (BRS), which employs a "carrot and stick" approach to reward modification:

  1. The Carrot (In-Distribution): A positive bias ($b^+$) is added to the rewards of actions existing in the dataset, encouraging the exploitation of known-good behaviors.
  2. The Stick (Out-of-Distribution): A negative bias ($b^-$) is applied to actions the agent selects that are absent from the dataset, penalizing exploration into unreliable regions of the state-action space.

PBRS refines this by making the shift magnitude, $b_k(s)$, adaptive. It is calculated based on the KL-divergence between the current learned policy and the original behavior policy. Early in training, when the policy is likely to deviate significantly, the penalty is high. As the policy aligns with the data, the constraints are relaxed.

The mean reward shift $b_k(s)$ adaptively decreases as training progresses, reflecting the relaxation of constraints as the policy stabilizes.

Theoretical and Empirical Insights

A key theoretical contribution is the demonstration that CQL is essentially a special case of BRS where reward shifts are held constant. The authors argue that constant penalties are suboptimal; Theorem 1 in the paper proves that a static, large penalty can inadvertently decrease the probability of selecting an optimal action if that action happens to be OOD. By contrast, the adaptive nature of PBRS ensures that the agent is not permanently discouraged from high-value regions.

Empirically, PBRS was evaluated on the D4RL MuJoCo benchmark. It achieved a total normalized score of 1076.3, surpassing CQL (1027.1) and IQL (1006.9). The performance was particularly strong in "medium-replay" scenarios, where the agent must distinguish between various levels of performance within a single buffer.

Analysis of Policy Diversity

One of the most significant advantages of PBRS over traditional regularization is the increased diversity of the learned policy. Because the reward shifting is adaptive and directly influences the Q-values rather than imposing a hard constraint on the actor, the agent maintains a higher degree of "action distance" from the behavior policy while remaining within safe bounds.

Comparison of action distance distributions on the Walker2d-Medium task. PBRS (ours) allows for a wider range of actions compared to the more constrained CQL.

As shown in the figure above, PBRS allows the agent to explore a broader distribution of actions. This flexibility is crucial for surpassing the performance of the behavior policy, as it allows the agent to "stitch" together the best parts of different trajectories more effectively than a rigid, conservative model.

Conclusion

PBRS offers a dynamic alternative to static regularization in offline RL. By adaptively shifting rewards based on policy proximity, it prevents overestimation of OOD actions without sacrificing the agent's ability to improve upon the source data. While the method requires careful tuning of the scaling coefficient $c$ and accurate modeling of the behavior policy, its state-of-the-art results on D4RL suggest it is a powerful tool for robust offline learning.

Summary updated

Abstract

One of the major challenges of the current offline reinforcement learning research is to deal with the distribution shift problem due to the change in state-action visitations for the new policy. To address this issue, we present a novel reward shifting-based method. Specifically, to regularize the behavior of the new policy at each state, we modify the reward to be received by the new policy by shifting it adaptively according to its proximity to the behavior policy, and apply the reward shifting along opposite directions for in-distribution actions and the ones not. In this way we are able to guide the learning procedure of the new policy itself by influencing the consequence of its actions explicitly, helping it to achieve a better balance between behavior constraints and policy improvement. Empirical results on the popular D4RL benchmarks show that the proposed method obtains competitive performance compared to the state-of-art baselines.
BibTeX
@inproceedings{zhang2023adaptive,
  author = {Zhe Zhang and Xiaoyang Tan},
  title = {{Adaptive Reward Shifting Based on Behavior Proximity for Offline Reinforcement Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=8Ma9tWPC2l},
}
Figure