Safe Continual Domain Adaptation after Sim2Real Transfer of Reinforcement Learning Policies in Robotics

Josip Josifovski, Shangding Gu, Mohammadhossein Malmir, Haoliang Huang, Sayantan Auddy, Nicolás Navarro-Guerrero, Costas J. Spanos, Alois KnollarXiv 2025 · DOI · arXiv

Abstract

Bridging the "reality gap" remains a significant challenge in robotics. While reinforcement learning (RL) policies trained in simulation can be transferred to the real world, they often suffer from suboptimality or instability when faced with evolving environmental conditions. This paper presents Safe Continual Domain Adaptation (SCDA), a framework that enables robots to safely and continuously fine-tune their control policies after deployment. By combining safe RL with continual learning techniques, SCDA allows for environment-specific optimization while maintaining strict safety constraints and preventing the "catastrophic forgetting" of initial training.

Introduction

Training robotic systems directly in the real world is often impractical due to high costs, time requirements, and the risk of hardware damage. Consequently, simulation-to-reality (Sim2Real) transfer is the standard approach. However, simulations are imperfect models of reality. Techniques like Domain Randomization (DR) attempt to solve this by training policies to be robust across a wide range of simulated parameters. While effective for transfer, DR often results in "conservative" policies that lack peak performance in any specific real-world setting. Furthermore, real-world conditions—such as mechanical wear or temperature fluctuations—change over time, necessitating post-deployment adaptation. Standard online RL adaptation is often too dangerous for physical hardware due to its unstable exploration phases.

Methodology: The SCDA Framework

The SCDA framework addresses these challenges by integrating three core components to ensure that adaptation is both effective and safe.

Conceptual overview of the Safe Continual Domain Adaptation (SCDA) framework, illustrating the transition from simulation pretraining to real-world adaptation.
  1. Safe Reinforcement Learning (PCRPO): To mitigate the risks of online exploration, the authors utilize Policy Constraint Reward Policy Optimization. This algorithm incorporates a "safety critic" that enforces predefined safety bounds (e.g., joint limits or collision avoidance) during both simulation and real-world adaptation.
  2. Continual Learning (EWC): To prevent the robot from losing the general capabilities learned in simulation while adapting to a specific environment, the framework employs Elastic Weight Consolidation. EWC uses the Fisher Information Matrix (FIM) to identify neural network weights critical to the general task. During adaptation, changes to these "important" weights are penalized, while less critical weights are allowed to adjust to the new domain.
  3. Domain Randomization: Wide DR is used during the initial pretraining phase to provide a robust baseline and to calculate the initial parameter importance (FIM) required for EWC.

Experimental Evaluation

The framework was evaluated using a KUKA LBR iiwa 14 robotic arm on two distinct tasks:

  • Reach-and-Balance: A 2-degree-of-freedom (DOF) task focused on reaching targets quickly while strictly avoiding joint limits.
  • Grasping: A complex 7-DOF task involving a 3-finger Robotiq gripper, where the robot must successfully grasp a box while avoiding collisions with its stand.

Testing was conducted in both a high-fidelity "realistic" simulator and on physical hardware to validate the Sim2Real transfer and subsequent adaptation.

Key Insights and Results

A primary technical insight of this work is the effectiveness of selective plasticity. By using EWC, the model distinguishes between weights necessary for the core task and those that can be modified for environmental tuning.

Visualization of selective plasticity: parameter changes relative to their Fisher importance.

As shown in the experimental results, SCDA significantly outperforms baseline methods:

  • Performance Gains: In the grasping task, SCDA improved the real-world success rate from a 20% zero-shot baseline to 60% after adaptation.
  • Safety Maintenance: Unlike standard adaptation methods that often violate constraints to maximize rewards, SCDA reduced safety costs to zero during the adaptation process.
  • Stability: The framework maintained high performance across a variety of targets, avoiding the overfitting issues seen in non-continual adaptation methods.

Conclusion and Future Work

SCDA demonstrates that safe, continuous adaptation in the real world is achievable by combining safety-constrained optimization with importance-aware weight updates. This allows robots to evolve from "jack-of-all-trades" robust policies to environment-specific "experts."

Future research will focus on automating the detection of domain shifts to eliminate the need for manual triggering of the adaptation mode. Additionally, the authors aim to integrate formal methods to provide even stronger safety guarantees during the earliest stages of policy learning.

Summary updated

Abstract

Domain randomization has emerged as a fundamental technique in reinforcement learning (RL) to facilitate the transfer of policies from simulation to real-world robotic applications. Many existing domain randomization approaches have been proposed to improve robustness and sim2real transfer. These approaches rely on wide randomization ranges to compensate for the unknown actual system parameters, leading to robust but inefficient real-world policies. In addition, the policies pretrained in the domain-randomized simulation are fixed after deployment due to the inherent instability of the optimization processes based on RL and the necessity of sampling exploitative but potentially unsafe actions on the real system. This limits the adaptability of the deployed policy to the inevitably changing system parameters or environment dynamics over time. We leverage safe RL and continual learning under domain-randomized simulation to address these limitations and enable safe deployment-time policy adaptation in real-world robot control. The experiments show that our method enables the policy to adapt and fit to the current domain distribution and environment dynamics of the real system while minimizing safety risks and avoiding issues like catastrophic forgetting of the general policy found in randomized simulation during the pretraining phase. Videos and supplementary material are available at https://safe-cda.github.io/.
BibTeX
@misc{josifovski2025safe,
  author = {Josip Josifovski and Shangding Gu and Mohammadhossein Malmir and Haoliang Huang and Sayantan Auddy and Nicolás Navarro-Guerrero and Costas J. Spanos and Alois Knoll},
  title = {{Safe Continual Domain Adaptation after Sim2Real Transfer of Reinforcement Learning Policies in Robotics}},
  year = {2025},
  eprint = {2503.10949},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.10949},
}
Figure