Efficient and Stable Offline-to-online Reinforcement Learning via Continual Policy Revitalization

Rui Kong, Chenyang Wu, Chen-Xiao Gao, Zongzhang Zhang, Ming Li

Abstract

Transitioning reinforcement learning (RL) agents from offline pre-training to online fine-tuning often results in performance degradation due to distribution shifts and "primacy bias," where overtrained networks lose plasticity. We introduce Continual Policy Revitalization (CPR), a framework that periodically resets the policy network to restore learning capacity while maintaining a "policy set" to preserve accumulated knowledge. Our approach ensures stable, monotonic improvement without requiring access to the original offline dataset during online updates. Evaluated on D4RL benchmarks, CPR consistently outperforms state-of-the-art methods in both efficiency and final performance.

Introduction

The offline-to-online (O2O) transition is a critical step for deploying RL agents in real-world scenarios. While offline RL provides a safe starting point using historical data, online fine-tuning is necessary to reach peak performance. However, this transition faces two primary hurdles: distribution shift, where the agent encounters unfamiliar states and overestimates their value, and primacy bias, where the network becomes too rigid to incorporate new information effectively. This loss of plasticity often leads to a sharp performance drop immediately after the transition begins.

Continual Policy Revitalization (CPR)

To address these challenges, CPR employs a strategy of periodic network resets combined with a knowledge preservation mechanism. Unlike previous methods that attempt to fine-tune a single rigid model, CPR acknowledges that resetting parameters can be more effective for restoring learning capacity.

The CPR framework: Offline pre-training is followed by an online phase where a policy set maintains historical knowledge while a current policy is periodically revitalized to maintain plasticity.

The core components of CPR include:

  • Policy Revitalization: Periodically re-initializing the policy network to a "blank slate" to restore its ability to learn from new online samples.
  • Policy Set & Composition: Maintaining a collection of policies (the original offline policy and subsequent versions). Actions are selected via a Boltzmann distribution weighted by Q-values, ensuring the agent utilizes the best available knowledge while the newest network is being trained.
  • Adaptive Constraints: The training loss for the new policy includes a constraint to stay near the composed behavior of the policy set, providing a safety mechanism during the revitalization phase.

Experimental Results

We evaluated CPR on the D4RL MuJoCo locomotion benchmarks, covering various data quality levels from random to medium-expert. The results demonstrate that CPR effectively mitigates the performance crash typically seen in O2O transitions.

Comparative performance of CPR against state-of-the-art baselines across 12 D4RL tasks. CPR (red line) demonstrates superior stability and final returns.

Key findings include:

  • Superior Stability: Unlike many baselines that suffer from a performance "dip" at the start of online training, CPR maintains stable or improving performance throughout the transition.
  • Efficiency on Low-Quality Data: CPR shows significant advantages in "Random" and "Medium" settings, where the initial offline policy is weak. By restoring plasticity, it quickly surpasses the performance of the original model.
  • Primacy Bias Mitigation: Our results confirm that maintaining network plasticity through revitalization is more effective than standard fine-tuning for long-term improvement.

Conclusion

Continual Policy Revitalization provides a robust solution for the O2O transition by explicitly managing network plasticity and knowledge retention. By decoupling the learning of new behaviors from the preservation of old ones, CPR achieves state-of-the-art performance across a wide range of tasks without the need for the original offline data. This approach offers a scalable and efficient path for refining pre-trained models in dynamic environments.

Summary updated

BibTeX
@inproceedings{kong2024efficient,
  author = {Rui Kong and Chenyang Wu and Chen-Xiao Gao and Zongzhang Zhang and Ming Li},
  title = {{Efficient and Stable Offline-to-online Reinforcement Learning via Continual Policy Revitalization}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure