Context-Aware Safe Reinforcement Learning for Non-Stationary Environments

Baiming Chen, Zuxin Liu, Jiacheng Zhu, Mengdi Xu, Wenhao Ding, Liang Li, Ding Zhao

Introduction

Deploying Reinforcement Learning (RL) in real-world scenarios requires addressing two critical challenges: safety and non-stationarity. While RL agents often excel in static simulations, real-world environments frequently involve changing dynamics—such as a drone navigating variable winds or a healthcare robot interacting with a moving patient. Traditional safe RL methods often fail to adapt quickly to these "non-stationary disturbances," leading to safety violations during the adaptation phase. This paper introduces CASRL (Context-Aware Safe Reinforcement Learning), a meta-learning framework designed to maintain safety while rapidly adapting to environmental shifts.

Methodology

CASRL integrates three core components to achieve robust performance:

  1. Context-Aware Latent Variable Model: Utilizing an Attentive Neural Process (ANP), the agent infers hidden environmental parameters (e.g., object mass or movement speed) from a "context" of recent state-action transitions. This allows for online Bayesian inference of the environment's current state.
  2. Dynamics Decomposition: The system models dynamics as the sum of a known prior model (basic physics) and a learned disturbance model (the context-aware neural network). This structure ensures that even before the neural network fully learns a new disturbance, the agent can rely on fundamental physical constraints.
  3. Risk-Averse MPC with Prior Constraints: Decision-making is handled via Model Predictive Control (MPC) using Conditional Value at Risk (CVaR). By focusing on worst-case outcomes and applying safety constraints derived from both the prior and learned models, CASRL provides a safety net throughout the learning process.

Experimental Setup

The framework was evaluated on two challenging benchmarks: a modified Cart-Pole swingup task with randomized physical properties and a high-dimensional Healthcare Feeding Robot task.

The healthcare feeding robot environment, where the agent must safely deliver food despite unpredictable patient head movements.

In the feeding task, a 7-DOF robot arm must deliver food to a patient's mouth while avoiding forceful contact. The non-stationarity arises from the patient's head moving at varying speeds and rotations across episodes.

Results and Discussion

CASRL demonstrated significant improvements in both safety and adaptation speed compared to baselines like Model-Agnostic Meta-Learning (MAML) and PETS.

CASRL demonstrating adaptive safety across different non-stationary disturbances (forward and rotational head movements).

In the healthcare robot simulation, CASRL maintained a safety violation rate of approximately 1%, whereas MAML experienced spikes up to 5% during adaptation. The use of prioritized sampling—over-sampling rare near-miss data—allowed the model to learn safety boundaries more effectively. Furthermore, CASRL proved robust across the entire disturbance space, successfully handling extreme parameters where other models failed.

Conclusion

CASRL provides a robust framework for safe RL in changing environments by combining context-aware latent modeling with risk-averse control. By leveraging prior physical knowledge as a safety shield, it minimizes catastrophic failures during adaptation. Future work may explore extending this approach to more complex, image-based disturbance spaces using latent dynamics models.

Summary updated

BibTeX
@inproceedings{chen2021context,
  author = {Baiming Chen and Zuxin Liu and Jiacheng Zhu and Mengdi Xu and Wenhao Ding and Liang Li and Ding Zhao},
  title = {{Context-Aware Safe Reinforcement Learning for Non-Stationary Environments}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=ruLmCSN8jD8},
}
Figure