Constrained Reinforcement Learning and Formal Verification for Safe Colonoscopy Navigation

Davide Corsi, Luca Marzari, Ameya Pore, Alessandro Farinelli, Alicia Casals, Paolo Fiorini, Diego Dall'Alba

Introduction

Colonoscopies are critical for colorectal cancer screening but remain technically challenging and potentially painful. While robotic flexible endoscopes (FEs) aim to automate navigation, traditional "lumen tracking" methods often fail during sharp turns or when the endoscope is in close proximity to the colon wall. Deep Reinforcement Learning (DRL) offers a promising alternative for learning complex navigation from visual data. However, the "black-box" nature of standard DRL poses significant safety risks in surgical settings, where even a 1% failure rate is unacceptable. This work introduces a framework that combines Constrained Reinforcement Learning (CRL) with Formal Verification (FV) to produce provably safe autonomous agents.

The Two-Layered Safety Framework

The researchers propose a dual-loop architecture to ensure both performance and safety.

Overview of the proposed two-loop framework, featuring an inner loop for training with soft constraints and an outer loop for formal verification and model selection.

Inner Loop: Training with Soft Constraints

The inner loop utilizes Lagrangian Proximal Policy Optimization (L-PPO). Unlike standard RL, which only maximizes a reward, L-PPO incorporates a cost function that penalizes wall collisions. The objective is to maximize navigation rewards while maintaining the cumulative cost below a predefined safety threshold during the training phase.

Outer Loop: Formal Verification and Model Selection

To move beyond "soft" probabilistic safety, the outer loop employs VeriNet, a formal verification tool. This stage treats the trained neural network as a mathematical object to prove it adheres to "Hard Constraints." For example, a rule might state: "If the top of the visual field is saturated (indicating wall contact), the agent must not move upward."

Input Discretization for Verification

Formal verification of high-dimensional neural networks is computationally prohibitive. To address this, the researchers discretized the endoscopic view into a 4x4 grid of brightness values. This simplification allows VeriNet to exhaustively check all possible inputs within defined bounds to guarantee safety properties are never violated.

The navigation action space (a) and the simplified 4x4 brightness grid (b) that enables formal verification of the agent's policy.

Experimental Results and the "Safety Gap"

The framework was evaluated using high-fidelity simulations in Unity3D and SOFA, utilizing 3D colon models derived from real patient CT scans. While both standard PPO and L-PPO achieved a 100% success rate in reaching the target, a significant "safety gap" was observed.

Out of 300 trained policies, zero standard PPO policies passed the formal safety check. Even among L-PPO policies, which were trained with safety costs, only three were found to be entirely "provably safe." This highlights the necessity of formal verification as a gatekeeper for clinical deployment. The identified "Safe Three" policies not only guaranteed safety but also demonstrated higher efficiency than traditional lumen-centralization baselines.

Conclusion and Future Work

This research demonstrates that combining CRL with formal verification can bridge the gap between high-performance AI and the rigorous safety requirements of surgery. Future work will focus on closing the "sim-to-real" gap by testing these provably safe policies on physical robotic platforms and under more complex, dynamic environmental conditions, such as varying illumination and the presence of fluids or polyps.

Summary updated

BibTeX
@inproceedings{corsi2023constrained,
  author = {Davide Corsi and Luca Marzari and Ameya Pore and Alessandro Farinelli and Alicia Casals and Paolo Fiorini and Diego Dall'Alba},
  title = {{Constrained Reinforcement Learning and Formal Verification for Safe Colonoscopy Navigation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=rVk0TsJTu7},
}
Figure