Safe Reinforcement Learning via Statistical Model Predictive Shielding

Osbert Bastani, Shuo LiRobotics: Science and Systems 2021 · OpenReview

Introduction

Reinforcement Learning (RL) has achieved remarkable success in complex robotic tasks, yet the deployment of Deep Neural Network (DNN) policies on physical hardware remains a significant challenge. These "black-box" policies often prioritize cumulative reward over strict safety constraints. In real-world scenarios, a single failure—such as a bipedal robot falling or an autonomous vehicle colliding—can be catastrophic.

Existing safety frameworks typically rely on either exhaustive ahead-of-time verification, which scales poorly with system complexity, or real-time shielding. Traditional shielding methods, such as Robust Model Predictive Shielding (RMPS), often depend on expensive reachability analysis that is too computationally intensive for high-dimensional systems in real-time. This paper introduces Statistical Model Predictive Shielding (SMPS), a framework that guarantees safety with high probability while operating up to 100x faster than previous state-of-the-art methods.

The SMPS Framework

The core philosophy of SMPS is to maintain a safety invariant: the robot should only execute an action if there is a verified "backup plan" capable of returning the system to a safe equilibrium (e.g., a car at rest or a drone landed).

Conceptual overview of SMPS state spaces and the distinction between recoverable and irrecoverable states for a mobile robot.

Key Components

  • The Shield: At each time step, the shield evaluates the action proposed by the RL policy. It simulates the trajectory and checks if a backup policy can successfully transition the robot from the resulting state back into a known safe invariant set ($\mathcal{X}_{inv}$).
  • Statistical Verification: Rather than using exhaustive formal methods to prove safety against all possible disturbances, SMPS employs statistical sampling. By simulating $K$ possible future trajectories under stochastic noise and applying concentration inequalities, the system determines the exact number of samples required to guarantee safety at a specific confidence level (e.g., 99%).
  • LQR and Taylor’s Theorem: To define safe zones near equilibrium points, the authors utilize Linear-Quadratic Regulator (LQR) theory. They use Taylor’s theorem to rigorously bound errors introduced by linearizing nonlinear dynamics, ensuring the invariant sets are mathematically sound.
  • Recovery Policy: In addition to the primary RL policy and the backup policy, a specific "recovery" policy is trained to steer the robot back toward safe equilibrium points if it drifts into the recovery set ($\mathcal{X}_{rec}$).

Experimental Evaluation

The SMPS algorithm was tested across several challenging robotics benchmarks, including Cart-pole balancing, a 4D Bicycle Model for navigation, and high-dimensional MuJoCo environments like "Half-Cheetah" and "Ant." To test robustness, the authors introduced "distribution shifts"—such as increased noise or altered obstacle positions—where standard RL policies typically fail.

Key Results

SMPS demonstrated a significant leap in both computational efficiency and safety. In terms of speed, SMPS achieved a ~145x speedup over Robust MPS, taking only 0.39 seconds per step compared to 56.92 seconds for RMPS at a 99% confidence level. This efficiency allows SMPS to handle long time horizons that are computationally intractable for formal verification methods.

Comparison of reward and safety probability across different shielding configurations.

As shown in the results, while a standard RL policy ("No Shield") may achieve some reward, it offers zero safety guarantees in modified environments. Conversely, a system with only a backup policy ("No Recovery") is perfectly safe but fails to achieve any task reward. SMPS ("Ours") provides the best of both worlds: maintaining 100% safety (Pr[Safe] = 1.0) while still achieving high rewards by allowing the RL policy to operate within verified safe boundaries.

Conclusion and Future Work

SMPS proves that real-time safety for complex robots does not require verifying the complex RL policy itself, but rather verifying the viability of a backup safety plan. By shifting from reachability analysis to statistical sampling, SMPS enables safe deployment on high-dimensional systems. Future work will focus on reducing the dependence on perfect nominal dynamics models and extending the framework to systems that lack static equilibrium points.

Summary updated

BibTeX
@inproceedings{bastani2021safe,
  author = {Osbert Bastani and Shuo Li},
  title = {{Safe Reinforcement Learning via Statistical Model Predictive Shielding}},
  booktitle = {Robotics: Science and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=hi0_aMsBg2D},
}
Figure