FOSP: Fine-tuning Offline Safe Policy through World Models

Chenyang Cao, Yucheng Xin, Silang Wu, Longxiang He, Zichen Yan, Junbo Tan, Xueqian Wang

Abstract

Offline Safe Reinforcement Learning (RL) seeks to address safety constraints by learning from static datasets and restricting exploration. However, these approaches heavily rely on the dataset and struggle to generalize to unseen scenarios safely. In this paper, we aim to improve safety during the deployment of vision-based robotic tasks through online fine-tuning an offline pretrained policy. To facilitate effective fine-tuning, we introduce model-based RL, which is known for its data efficiency. Specifically, our method employs in-sample optimization to improve offline training efficiency while incorporating reachability guidance to ensure safety. After obtaining an offline safe policy, a safe policy expansion approach is leveraged for online fine-tuning. The performance of our method is validated on simulation benchmarks with five vision-only tasks and through real-world robot deployment using limited data. It demonstrates that our approach significantly improves the generalization of offline policies to unseen safety-constrained scenarios. To the best of our knowledge, this is the first work to explore offline-to-online RL for safe generalization tasks.

Introduction

In the world of robotics, Offline Reinforcement Learning (RL) is highly attractive because it allows agents to learn from pre-collected data without the risk of breaking hardware during exploration. However, offline datasets are rarely perfect; they can't capture every possible real-world scenario. When an offline-trained robot encounters an "out-of-distribution" (OOD) situation—like a new obstacle it hasn't seen before—it often fails or violates safety constraints.

The challenge lies in Safe Offline-to-Online Fine-tuning. Standard fine-tuning often leads to a "performance dip" or a spike in safety violations as the agent starts exploring the real world. Existing methods are either too conservative (resulting in poor performance) or too aggressive (resulting in dangerous collisions). FOSP (Fine-tuning Offline Safe Policy) aims to bridge this gap, providing a way to leverage offline data for a head start while ensuring the robot remains "persistently safe" during online adaptation.

Methodology

FOSP is built on a model-based RL architecture (specifically extending the DreamerV3 framework). Its novel approach consists of three main pillars:

  1. In-Sample Optimization for World Models: To combat the common problem of "value overestimation" (where the agent thinks an unknown action is better than it actually is), FOSP uses expectile regression. This ensures the agent's "critic" stays grounded in the actions actually present in the offline dataset.
  2. Reachability-Guided Safety: Instead of just using a simple penalty for hitting obstacles, FOSP incorporates a Reachability Estimation Function (REF). This function predicts the probability of future constraint violations. The optimization is split: in the "feasible set" (safe states), the agent maximizes reward; in the "infeasible set" (dangerous states), the agent's sole priority is to return to safety as quickly as possible.
  3. Safe Policy Expansion (PEX): Rather than directly overwriting the offline policy during fine-tuning, FOSP "expands" it. It initializes a new online policy while keeping the frozen offline policy as a safety guide. A softmax distribution based on Q-values determines which policy to trust at any given moment, ensuring a smooth and safe transition.
The FOSP framework overview, illustrating the transition from offline pretraining to online fine-tuning using a world model and policy expansion.

Experimental Evaluation

The authors evaluated FOSP using both simulation and real-world hardware:

  • Simulation: They used the Safety-Gymnasium benchmark, focusing on five vision-only tasks: PointGoal1, PointGoal2, PointButton1, PointPush1, and CarGoal2. The offline datasets were composed of a 1:1:1 mix of trajectories from "safe," "unsafe," and "random" policies.
  • Real-World: A Franka Panda 7-DOF robotic arm was used for a task called SafeReach. The robot had to reach a target while avoiding green rectangular obstacles.
  • Generalization Tasks: The authors introduced "unseen" scenarios (e.g., SafetyFading) where targets disappear over time, requiring the agent to rely on its world model's memory and prediction capabilities.

Key Results and Discussion

FOSP demonstrates superior performance in maintaining safety while adapting to new environments. In simulation, FOSP achieved nearly zero safety violations (cost) during both the offline and online phases, significantly outperforming baselines like Recover-RL and SafeDreamer.

Performance comparison across five vision-only tasks, showing the transition from offline (left of arrow) to online (right of arrow) performance in terms of reward and cost.

Key insights from the results include:

  • World Models as Safety Simulators: By learning a "world model" (a latent imagination of how the environment works), the agent can predict the consequences of its actions before taking them. This "look-ahead" ability is the secret sauce for maintaining safety in high-dimensional, vision-based tasks.
  • Superior Generalization: FOSP was the only method that could successfully adapt to the SafetyFading task, where the environment changes dynamically.
  • Real-World Efficiency: On the Franka robot, FOSP generalized to completely unseen obstacle configurations in just 40 trials. In contrast, the baseline SafeDreamer frequently violated constraints or failed to reach the goal in the same setting.

Conclusion

FOSP provides a robust framework for safe offline-to-online reinforcement learning. By combining in-sample optimization, reachability guidance, and policy expansion, it achieves nearly zero safety violations during deployment. While highly effective, future work will address vision challenges such as occluded obstacles and extend the framework to more complex manipulation tasks like grasping and pulling.

Summary updated

Abstract

Offline Safe Reinforcement Learning (RL) seeks to address safety constraints by learning from static datasets and restricting exploration. However, these approaches heavily rely on the dataset and struggle to generalize to unseen scenarios safely. In this paper, we aim to improve safety during the deployment of vision-based robotic tasks through online fine-tuning an offline pretrained policy. To facilitate effective fine-tuning, we introduce model-based RL, which is known for its data efficiency. Specifically, our method employs in-sample optimization to improve offline training efficiency while incorporating reachability guidance to ensure safety. After obtaining an offline safe policy, a safe policy expansion approach is leveraged for online fine-tuning. The performance of our method is validated on simulation benchmarks with five vision-only tasks and through real-world robot deployment using limited data. It demonstrates that our approach significantly improves the generalization of offline policies to unseen safety-constrained scenarios. To the best of our knowledge, this is the first work to explore offline-to-online RL for safe generalization tasks.
BibTeX
@inproceedings{cao2025fosp,
  author = {Chenyang Cao and Yucheng Xin and Silang Wu and Longxiang He and Zichen Yan and Junbo Tan and Xueqian Wang},
  title = {{FOSP: Fine-tuning Offline Safe Policy through World Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=dbuFJg7eaw},
}
Figure