Context-Aware Safe Reinforcement Learning for Non-Stationary Environments

Baiming Chen, Zuxin Liu, Jiacheng Zhu, Mengdi Xu, Wenhao Ding, Ding Zhao

Introduction

Deploying Reinforcement Learning (RL) in real-world scenarios requires addressing two critical challenges: safety and non-stationarity. While RL agents often excel in static simulations, real-world environments frequently involve changing dynamics—such as a drone navigating variable winds or a healthcare robot interacting with a moving patient. Traditional safe RL methods often fail to adapt quickly to these "non-stationary disturbances," leading to safety violations during the adaptation phase. This paper introduces CASRL (Context-Aware Safe Reinforcement Learning), a meta-learning framework designed to maintain safety while rapidly adapting to environmental shifts.

Methodology

CASRL integrates three core components to achieve robust performance:

  1. Context-Aware Latent Variable Model: Utilizing an Attentive Neural Process (ANP), the agent infers hidden environmental parameters (e.g., object mass or movement speed) from a "context" of recent state-action transitions. This allows for online Bayesian inference of the environment's current state.
  2. Dynamics Decomposition: The system models dynamics as the sum of a known prior model (basic physics) and a learned disturbance model (the context-aware neural network). This structure ensures that even before the neural network fully learns a new disturbance, the agent can rely on fundamental physical constraints.
  3. Risk-Averse MPC with Prior Constraints: Decision-making is handled via Model Predictive Control (MPC) using Conditional Value at Risk (CVaR). By focusing on worst-case outcomes and applying safety constraints derived from both the prior and learned models, CASRL provides a safety net throughout the learning process.

Experimental Setup

The framework was evaluated on two challenging benchmarks: a modified Cart-Pole swingup task with randomized physical properties and a high-dimensional Healthcare Feeding Robot task.

The healthcare feeding robot environment, where the agent must safely deliver food despite unpredictable patient head movements.

In the feeding task, a 7-DOF robot arm must deliver food to a patient's mouth while avoiding forceful contact. The non-stationarity arises from the patient's head moving at varying speeds and rotations across episodes.

Results and Discussion

CASRL demonstrated significant improvements in both safety and adaptation speed compared to baselines like Model-Agnostic Meta-Learning (MAML) and PETS.

CASRL demonstrating adaptive safety across different non-stationary disturbances (forward and rotational head movements).

In the healthcare robot simulation, CASRL maintained a safety violation rate of approximately 1%, whereas MAML experienced spikes up to 5% during adaptation. The use of prioritized sampling—over-sampling rare near-miss data—allowed the model to learn safety boundaries more effectively. Furthermore, CASRL proved robust across the entire disturbance space, successfully handling extreme parameters where other models failed.

Conclusion

CASRL provides a robust framework for safe RL in changing environments by combining context-aware latent modeling with risk-averse control. By leveraging prior physical knowledge as a safety shield, it minimizes catastrophic failures during adaptation. Future work may explore extending this approach to more complex, image-based disturbance spaces using latent dynamics models.

Summary updated

Abstract

Safety is a critical concern when deploying reinforcement learning agents for realistic tasks. Recently, safe reinforcement learning algorithms have been developed to optimize the agent’s performance while avoiding violations of safety constraints. However, few studies have addressed the nonstationary disturbances in the environments, which may cause catastrophic outcomes. In this paper, we propose the context-aware safe reinforcement learning (CASRL) method, a metal-earning framework to realize safe adaptation in non-stationary environments. We use a probabilistic latent variable model to achieve fast inference of the posterior environment transition distribution given the context data. Safety constraints are then evaluated with uncertainty-aware trajectory sampling. Prior safety constraints are formulated with domain knowledge to improve safety during exploration. The algorithm is evaluated in realistic safety-critical environments with non-stationary disturbances. Results show that the proposed algorithm significantly outperforms existing baselines in terms of safety and robustness.
BibTeX
@inproceedings{chen2021context,
  author = {Baiming Chen and Zuxin Liu and Jiacheng Zhu and Mengdi Xu and Wenhao Ding and Ding Zhao},
  title = {{Context-Aware Safe Reinforcement Learning for Non-Stationary Environments}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=ruLmCSN8jD8},
}
Figure