Efficient and Stable Offline-to-online Reinforcement Learning via Continual Policy Revitalization

Rui Kong, Chenyang Wu, Chen-Xiao Gao, Zongzhang Zhang, Ming Li

Abstract

Offline-to-online (O2O) reinforcement learning (RL) aims to bridge the gap between static dataset pre-training and real-world interaction fine-tuning. However, this transition is frequently hampered by distribution shifts and a phenomenon known as "loss of plasticity," where neural networks overtrained on static data become brittle and unable to adapt to new information. We introduce Continual Policy Revitalization (CPR), a framework that addresses these challenges by periodically resetting the policy network while archiving historical models in a policy library. By employing an adaptive policy composition mechanism, CPR maintains high baseline performance while restoring the agent's capacity for continuous online learning.

Introduction

In the standard RL pipeline, the transition from offline datasets to online interaction is a critical step for deploying agents in complex environments. Ideally, an agent should inherit a strong baseline from offline data and improve monotonically. In practice, two primary hurdles exist:

  1. Distribution Shift and Extrapolation Error: Online interaction exposes the agent to states and actions outside the offline distribution. This often leads to over-optimistic value estimates (Q-values), causing the policy to collapse during the initial stages of online fine-tuning.
  2. Primacy Bias (Loss of Plasticity): Recent research highlights that extensive training on static datasets can cause neural networks to lose their ability to learn from new samples. The network weights become "stiff," making subsequent online learning inefficient or impossible.

CPR is designed to circumvent these issues by treating the online phase as a series of revitalization steps, ensuring the agent remains adaptable without discarding its prior knowledge.

The CPR Framework

Overview of the CPR framework, showing the interaction between offline training, policy revitalization, and online composition.

Continual Policy Revitalization (CPR) consists of three core architectural innovations:

Periodic Revitalization

To combat the loss of plasticity, CPR periodically re-initializes the policy network to a "blank slate." This reset restores the network's capacity to integrate new online samples. Unlike standard resets that might lose progress, CPR preserves knowledge through its policy set.

Policy Set Management

CPR maintains a library, $\Pi$, containing the initial offline-trained policy and all subsequent versions frozen at each revitalization step. This ensures that even as the current active network is reset, the agent's cumulative "wisdom" remains accessible.

Policy Composition and Adaptive Constraints

During online interaction, the agent does not rely solely on the newest (and potentially unstable) policy. Instead, it uses a Boltzmann distribution based on current Q-values to select actions from the entire policy set. This composition mechanism naturally favors high-performing historical policies when the new policy is still maturing. Furthermore, the optimization process utilizes a modified TD3+BC objective, where the Behavioral Cloning (BC) term acts as an adaptive constraint, guiding the new policy toward successful actions identified across the entire policy library.

Experimental Evaluation

We evaluated CPR on the D4RL MuJoCo Locomotion benchmarks, covering HalfCheetah, Hopper, and Walker2d across various data quality levels (Random, Medium, Medium-Replay, and Medium-Expert).

Performance comparison of CPR against state-of-the-art baselines across twelve D4RL locomotion tasks.

Key Results

  • State-of-the-Art Performance: As shown in the benchmarking results, CPR consistently outperforms established baselines such as IQL, AWAC, and PEX. It demonstrates superior sample efficiency and higher asymptotic performance across nearly all tasks.
  • Stability and Robustness: CPR avoids the characteristic "performance dip" often seen in O2O transitions. By leveraging the policy set, the agent maintains its offline baseline from the very first online update. It is particularly effective in "Random" and "Medium" data regimes, where traditional fine-tuning often fails to improve upon the initial policy.
  • Plasticity Restoration: The periodic resets effectively mitigate primacy bias. While overtraining on offline data typically hinders online adaptation, CPR's revitalization steps allow the agent to continue learning effectively regardless of the initial offline training duration.

Conclusion and Future Work

Continual Policy Revitalization provides a robust solution to the instability and plasticity issues inherent in offline-to-online RL. By combining periodic weight resets with a structured policy library, CPR achieves a balance between knowledge retention and new information integration.

Future research will focus on reducing the computational overhead associated with maintaining multiple policy networks and exploring the application of CPR to larger-scale architectures, such as Transformers, where plasticity management is increasingly critical.

Summary updated

BibTeX
@inproceedings{kong2024efficient,
  author = {Rui Kong and Chenyang Wu and Chen-Xiao Gao and Zongzhang Zhang and Ming Li},
  title = {{Efficient and Stable Offline-to-online Reinforcement Learning via Continual Policy Revitalization}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure