Safe Reinforcement Learning using Formal Verification for Tissue Retraction in Autonomous Robotic-Assisted Surgery

Ameya Pore, Davide Corsi, Enrico Marchesini, Diego Dall'Alba, Alicia Casals, Alessandro Farinelli, Paolo Fiorini

Abstract

This research introduces a Safe-DRL framework designed to automate tissue retraction in robotic surgery by integrating Deep Reinforcement Learning (DRL) with formal verification. By employing reward shaping during training and a novel interval-analysis tool for evaluation, the system significantly mitigates the risk of surgical instruments impacting critical anatomical structures while maintaining high task performance.

Introduction: The Challenge of Surgical Automation

Automating surgical subtasks, such as tissue retraction—where fat or connective tissue is moved to expose a tumor—has the potential to reduce surgeon fatigue and enhance procedural precision. However, Deep Reinforcement Learning (DRL), while powerful for complex robotics, remains difficult to deploy in safety-critical environments due to its "black box" nature.

Standard DRL agents maximize rewards through trial and error, but in surgery, an error can lead to catastrophic outcomes like organ puncture. Current methods often lack provable safety guarantees, performing well in standard simulations but failing under unusual conditions. This study bridges the gap between high-performance DRL and the rigorous safety requirements of the operating room.

The simulated surgical environment for partial nephrectomy, featuring the robotic PSM, deformable fat tissue, and the target tumor.

Methods: Safe-DRL and Formal Verification

The researchers focused on a partial nephrectomy (kidney surgery) scenario using the UnityFlexML simulation environment, which accurately models the physics of deformable tissues.

1. Safe-DRL via Reward Shaping

A Proximal Policy Optimization (PPO) agent was trained with integrated safety constraints. The team defined a "safe workspace" represented by a virtual cylinder. If the robotic arm exited this cylinder or contacted a restricted organ, the agent received a significant numerical penalty ($c$), incentivizing the discovery of safe trajectories.

Visualization of the safe workspace (blue cylinder) used to constrain the robotic arm's movement during training and verification.

2. Formal Verification Tool

The core innovation is a verification tool based on interval algebra (specifically Moore’s comparison rules). Unlike standard testing, this tool mathematically analyzes the entire possible input space of the neural network. By recursively subdividing the input domain into smaller "sub-areas," the authors calculated a "violation rate," providing a probabilistic upper bound on risk rather than a simple binary pass/fail.

Experimental Setup

The study utilized a custom virtual surgical scene simulating the DaVinci Surgical System (DVSS) interacting with a kidney and perirenal fat. Performance was evaluated using Tumor Exposure (TE), a metric measuring the visibility of the target tumor following the robot's retraction maneuver across a 5x5 grid of starting positions.

Key Results and Insights

  • Safety Without Performance Loss: Contrary to the assumption that strict safety constraints hinder performance, the "Safe-PPO" agent achieved higher final rewards than the unsafe version by learning more stable and efficient trajectories. Both models achieved nearly identical Tumor Exposure scores (~0.41).
  • Significant Risk Reduction: The Safe-PPO model reduced the overall violation rate to 3.07%, a dramatic improvement over the 27.02% violation rate observed in the Unsafe-PPO model.
  • Granular Failure Analysis: The verification tool identified that most safety violations occurred at specific distances from the target, allowing researchers to pinpoint hazardous states before any real-world deployment.

Conclusion and Future Work

While the current formal verification is an offline process, future work aims to develop an online safety controller capable of real-time intervention. Additionally, the researchers plan to transition from simulation to physical testing using synthetic phantoms and real robotic hardware. Future iterations will also explore weighting safety properties based on clinical risk (e.g., prioritizing the avoidance of major arteries over less critical structures).

Summary updated

BibTeX
@inproceedings{pore2021safe,
  author = {Ameya Pore and Davide Corsi and Enrico Marchesini and Diego Dall'Alba and Alicia Casals and Alessandro Farinelli and Paolo Fiorini},
  title = {{Safe Reinforcement Learning using Formal Verification for Tissue Retraction in Autonomous Robotic-Assisted Surgery}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=2JNOJfju69g},
}
Figure