Towards Fast Safe Online Reinforcement Learning via Policy Finetuning

Keru Chen, Honghao Wei, Zhigang Deng, Sen Lin

TL;DR

Marvel is a novel offline-to-online (O2O) safe reinforcement learning framework designed to enable fast and safe online policy finetuning from static datasets. By combining Value Pre-Alignment to correct offline Q-estimation errors with an Adaptive PID controller to manage Lagrange multipliers, Marvel achieves near-optimal rewards with minimal safety violations in just a fraction of the steps required by traditional methods.

What Problem Does This Paper Address?

Deploying reinforcement learning (RL) in real-world systems like autonomous driving or healthcare requires strict safety guarantees. Online safe RL algorithms learn by trial and error, which is dangerously expensive when errors mean physical crashes or system failures. Offline safe RL mitigates this by training purely on static, pre-collected datasets. However, offline policies are notoriously conservative and limited by the quality of the available data.

The natural next step is Offline-to-Online (O2O) safe RL: pretrain safely offline, then finetune online. However, the authors show that naively "warm-starting" an online agent with offline weights fails dramatically. This failure stems from two unique challenges:

  1. Erroneous Q-estimations: Offline algorithms heavily penalize out-of-distribution (OOD) actions to keep the agent safe. Combined with sparse cost data, this creates a highly distorted cost landscape that traps the agent in a hyper-conservative bubble, paralyzing online exploration.
  2. Lagrangian Mismatch: Safe RL relies on Lagrange multipliers ($\lambda$) to balance rewards and costs. The optimal $\lambda$ values calculated offline rarely align with online dynamics, leading to massive safety violations or training stagnation if initialized poorly.

Methods

To solve these issues, the authors introduce Marvel (warM-stArt safe Reinforcement learning with Value prE-aLignment), which introduces two key components:

  • Value Pre-Alignment (VPA): Before the agent interacts with the online environment, Marvel runs an offline calibration phase. Using Fitted Q Evaluation with an entropy-regularized Bellman operator, VPA strips away the artificial offline pessimism penalties. This aligns the pretrained Q-functions with the actual online objective, smoothing out the value landscape and encouraging the agent to safely explore the boundaries of the cost limit.
  • Adaptive PID Control (aPID): Traditional online safe RL uses standard dual ascent to update the Lagrange multiplier. Marvel replaces this with a tailored PID controller that dynamically adjusts the control gains ($K_p, K_i, K_d$) based on a short buffer of recent cost violations and variance. When the agent ventures into high-reward but higher-risk zones, aPID rapidly scales up the multiplier to suppress cost spikes, stabilizing the learning process.

Datasets

The framework was evaluated on the DSRL benchmark, utilizing ten distinct environments from Bullet Safety Gym and Safety Gymnasium:

  • Environments: BallCircle, BallRun, CarCircle, CarRun, HalfCheetah, AntCircle, AntRun, DroneCircle, Hopper, and Swimmer.
  • Data Composition: The training sets consisted of a mixture of logged expert/intermediate data from DSRL and completely random trajectories to simulate varying dataset qualities. The safety cost threshold was strictly set to 20 across all tasks.

Key Insights and Ideas

  • Pessimism is a double-edged sword: While offline pessimism keeps an agent safe during static training, it acts as an invisible wall during online finetuning. VPA's core insight is that we must explicitly unlearn this pessimism before starting online updates.
  • Counter-intuitive Optimism: By adding entropy regularization during VPA, the agent becomes slightly overoptimistic about uncertain actions. While dangerous in purely offline RL, this optimism is exactly what triggers the necessary exploration to find high-reward, safe paths online.
  • Dynamic Safety Nets: The combination of VPA and aPID acts as an accelerator and a brake working in tandem. VPA pushes the agent to explore higher-reward regions, while aPID dynamically steps in to damp out the resulting cost spikes before they violate safety thresholds.

Key Results

Marvel demonstrated exceptional sample efficiency and safety across the board:

  • Instantaneous Adaptation: In environments like BallCircle and CarCircle, Marvel achieved near-optimal performance in as few as 20 gradient steps, whereas training from scratch or using standard warm-starts took hundreds of steps or failed entirely.
  • Superior Reward-Safety Balance: Across multiple environments, Marvel consistently delivered the highest rewards while strictly adhering to the cost threshold of 20. For instance, in BallCircle, Marvel achieved a reward of 603.9 with a cost of 19.8, outperforming baselines like SO2, JSRL, and PEX, which frequently suffered massive cost violations (costs exceeding 100+).
  • Robustness to Data Quality: Marvel proved highly resilient, successfully finetuning agents regardless of whether the offline pretrained policy was high-performing or mediocre.
Figure 1: “Steps" on the x-axis represent the number of policy gradient updates (i.e., optimizer updates). For each update, the agent interacts with the environment for 3 episodes. This convention is followed in the subsequent figures. In (a), we evaluate these methods in BallCircle from Bullet Safety Gym (Gronauer, 2022), with a cost limit of 20. As shown, while “Warm Start” begins with a reasonably good initial policy, it performs poorly and overly conservatively, even worse than “From Scratch” where the policy and Q-functions are initialized randomly. This implies that directly finetuning the pretrained policy and Q-functions may hinder online learning. In contrast, “Marvel” achieves impressive results, finding a policy with much higher return in just a few online steps while adhering to the cost limit. In (b), t-SNE visualization of state in the environment, reduced to 2D space. Each point represents a state, with rewards uniformly distributed across the space, while costs are sparse, appearing as isolated points or clusters, reflecting their limited association with states.

Limitations and Future Work

Despite its success, Marvel struggles in highly complex, high-dimensional, non-stationary environments such as AntRun. In these scenarios, the adaptive PID controller does not perform optimally, and the agent fails to significantly improve its reward and cost metrics during finetuning. The authors note that future work will focus on enhancing VPA to boost exploratory behavior in these harder tasks and exploring strategies to explicitly reset or re-initialize the derivative gain ($K_d$) when environment dynamics shift unexpectedly.

Summary updated

Abstract

The high costs and risks involved in extensive environment interactions hinder the practical application of current online safe reinforcement learning (RL) methods. While offline safe RL addresses this by learning policies from static datasets, the performance therein is usually limited due to reliance on data quality and challenges with out-of-distribution (OOD) actions. Inspired by recent successes in offline-to-online (O2O) RL, it is crucial to explore whether offline safe RL can be leveraged to facilitate faster and safer online policy learning, a direction that has yet to be fully investigated. To fill this gap, we first demonstrate that naively applying existing O2O algorithms from standard RL would not work well in the safe RL setting due to two unique challenges: \emph{erroneous Q-estimations}, resulted from offline-online objective mismatch and offline cost sparsity, and \emph{Lagrangian mismatch}, resulted from difficulties in aligning Lagrange multipliers between offline and online policies. To address these challenges, we introduce \textbf{Marvel}, a novel framework for O2O safe RL, comprising two key components that work in concert: \emph{Value Pre-Alignment} to align the Q-functions with the underlying truth before online learning, and \emph{Adaptive PID Control} to effectively adjust the Lagrange multipliers during online finetuning. Extensive experiments demonstrate that Marvel significantly outperforms existing baselines in both reward maximization and safety constraint satisfaction. By introducing the first policy-finetuning based framework for O2O safe RL, which is compatible with many offline and online safe RL methods, our work has the great potential to advance the field towards more efficient and practical safe RL solutions.
BibTeX
@article{chen2025towards,
  author = {Keru Chen and Honghao Wei and Zhigang Deng and Sen Lin},
  title = {{Towards Fast Safe Online Reinforcement Learning via Policy Finetuning}},
  journal = {Accepted by TMLR},
  year = {2025},
  url = {https://openreview.net/forum?id=1SO7vmLFUq},
}
Figure