Abstract
Deploying deep reinforcement learning in safety-critical settings requires developing algorithms that obey hard constraints during exploration. This paper contributes a first approach toward enforcing formal safety constraints on end-to-end policies with visual inputs. Our approach draws on recent advances in object detection and automated reasoning for hybrid dynamical systems. The approach is evaluated on a novel benchmark that emphasizes the challenge of safely exploring in the presence of hard constraints. Our benchmark draws from several proposed problem sets for safe learning and includes problems that emphasize challenges such as reward signals that are not aligned with safety constraints. On each of these benchmark problems, our algorithm completely avoids unsafe behavior while remaining competitive at optimizing for as much reward as is safe. We also prove that our method of enforcing the safety constraints preserves all safe policies from the original environment.
Introduction
Reinforcement Learning (RL) typically relies on extensive exploration, which often involves "learning by failing." In safety-critical domains like autonomous robotics or self-driving cars, even a single failure during training can be catastrophic. Existing "Safe RL" methods often rely on reward shaping—which is brittle and doesn't guarantee safety—or Formally Constrained RL (FCRL). However, FCRL usually assumes "Oracle" access to the environment's internal state (e.g., precise coordinates). In real-world applications, agents must operate based on raw, noisy visual inputs. This paper introduces Verifiably Safe Reinforcement Learning (VSRL), a framework that enforces hard formal constraints on agents learning directly from pixels.
Methodology: The VSRL Framework
VSRL acts as a safety wrapper around standard RL algorithms like Proximal Policy Optimization (PPO). The architecture bridges the gap between high-dimensional visual inputs and rigorous symbolic safety proofs through three integrated stages.
- Object Detection (Perception): A pre-trained detector identifies safety-relevant objects from images and outputs an estimated symbolic state (e.g., positions of hazards).
- Formal Verification (Reasoning): Using Differential Dynamic Logic ($dL$) and the KeYmaera X theorem prover, the system models environment physics to synthesize a "safety monitor." This monitor defines mathematical guards for safe actions.
- Runtime Safety Guard (Execution): During exploration, the RL agent proposes an action $a_t$. The safety monitor checks this action against the symbolic state $o_t$. If unsafe, the guard intercepts and substitutes a safe action $a'_t$.
Crucially, the framework accounts for perception noise by incorporating an uncertainty bound ($\epsilon$) into the formal proofs, ensuring safety as long as the detector's error remains bounded.
Experimental Evaluation
The approach was evaluated on four environments designed to test safety and performance:
- XO: A grid-based navigation task.
- Adaptive Cruise Control (ACC): Maintaining safe distance in a continuous car-following scenario.
- Goal-Finding (GF): 2D navigation avoiding obstacles.
- Pointmesses (PM): A benchmark designed to test resistance to "reward hacking," where crashing normally yields higher rewards.
Results and Discussion
The primary achievement of VSRL is the complete elimination of safety violations during the entire training process.
As shown in the results, VSRL encountered zero accidents across all benchmarks, while the standard PPO baseline committed thousands of violations. Furthermore, VSRL achieved competitive or superior rewards. In the ACC environment, the safety monitor actually accelerated learning by pruning the search space of dangerous, unproductive actions. In the "Pointmesses" environment, VSRL successfully resisted reward hacking, forcing the agent to find safe ways to maximize utility.
Conclusion and Future Work
VSRL demonstrates that formal safety guarantees can be extended to end-to-end RL with visual inputs. While effective, the system's safety is currently tied to the reliability of the object detector. Future work will focus on improving perception robustness against novel hazards and reducing the computational overhead of real-time symbolic checking. This research marks a significant step toward deploying verifiably safe autonomous systems in complex, vision-based environments.
Summary updated