Abstract
Transitioning reinforcement learning (RL) agents from offline pre-training to online fine-tuning often results in performance degradation due to distribution shifts and "primacy bias," where overtrained networks lose plasticity. We introduce Continual Policy Revitalization (CPR), a framework that periodically resets the policy network to restore learning capacity while maintaining a "policy set" to preserve accumulated knowledge. Our approach ensures stable, monotonic improvement without requiring access to the original offline dataset during online updates. Evaluated on D4RL benchmarks, CPR consistently outperforms state-of-the-art methods in both efficiency and final performance.
Introduction
The offline-to-online (O2O) transition is a critical step for deploying RL agents in real-world scenarios. While offline RL provides a safe starting point using historical data, online fine-tuning is necessary to reach peak performance. However, this transition faces two primary hurdles: distribution shift, where the agent encounters unfamiliar states and overestimates their value, and primacy bias, where the network becomes too rigid to incorporate new information effectively. This loss of plasticity often leads to a sharp performance drop immediately after the transition begins.
Continual Policy Revitalization (CPR)
To address these challenges, CPR employs a strategy of periodic network resets combined with a knowledge preservation mechanism. Unlike previous methods that attempt to fine-tune a single rigid model, CPR acknowledges that resetting parameters can be more effective for restoring learning capacity.
The core components of CPR include:
- Policy Revitalization: Periodically re-initializing the policy network to a "blank slate" to restore its ability to learn from new online samples.
- Policy Set & Composition: Maintaining a collection of policies (the original offline policy and subsequent versions). Actions are selected via a Boltzmann distribution weighted by Q-values, ensuring the agent utilizes the best available knowledge while the newest network is being trained.
- Adaptive Constraints: The training loss for the new policy includes a constraint to stay near the composed behavior of the policy set, providing a safety mechanism during the revitalization phase.
Experimental Results
We evaluated CPR on the D4RL MuJoCo locomotion benchmarks, covering various data quality levels from random to medium-expert. The results demonstrate that CPR effectively mitigates the performance crash typically seen in O2O transitions.
Key findings include:
- Superior Stability: Unlike many baselines that suffer from a performance "dip" at the start of online training, CPR maintains stable or improving performance throughout the transition.
- Efficiency on Low-Quality Data: CPR shows significant advantages in "Random" and "Medium" settings, where the initial offline policy is weak. By restoring plasticity, it quickly surpasses the performance of the original model.
- Primacy Bias Mitigation: Our results confirm that maintaining network plasticity through revitalization is more effective than standard fine-tuning for long-term improvement.
Conclusion
Continual Policy Revitalization provides a robust solution for the O2O transition by explicitly managing network plasticity and knowledge retention. By decoupling the learning of new behaviors from the preservation of old ones, CPR achieves state-of-the-art performance across a wide range of tasks without the need for the original offline data. This approach offers a scalable and efficient path for refining pre-trained models in dynamic environments.
Summary updated