In the field of surgical robotics and computer-assisted intervention, high-fidelity simulations are essential for navigation and autonomous task execution. However, making these simulations match the dynamic and often unpredictable environment of the operating room remains a significant challenge. This paper, presented at MICCAI 2021, introduces a real-time pipeline designed to update the "boundary conditions" (BCs)—such as adhesions and tissue attachments—of patient-specific biomechanical simulations using intra-operative vision data.
The Challenge: Dynamic Boundary Conditions
Patient-specific Biomechanical Models (PBMs) utilize pre-operative scans (CT or MRI) to provide geometric data. However, these scans rarely capture the specific points where organs are attached to one another via connective tissue or adhesions. Furthermore, these boundary conditions change during surgery as tissues are dissected. If a simulation does not account for these anchors, its predictions regarding tissue deformation and reaction to force will be fundamentally inaccurate.
Proposed Methodology
The researchers developed a two-process framework that integrates a real-time physics simulation (using the SOFA framework) with a continuous parameter update pipeline. The update process consists of five distinct stages:
- Pre-processing: Raw 3D point clouds from an RGB-D camera are segmented and aligned with the pre-operative model.
- Displacement Estimation: Utilizing the ZoomOut method, the system calculates Intra-Operative Displacement (IOD), mapping deformed surface points back to their original positions.
- Voxelization: The 3D geometry and displacement fields are converted into a $64 \times 64 \times 64$ grid to facilitate deep learning processing.
- BANet (Boundary Attachment Network): A U-Net-style Deep Neural Network (DNN) predicts which voxels represent attachment points based on observed surface deformations.
- PBM Update: The predicted BCs are mapped back to the simulation, creating an "Intra-Operative Model" (IOM) that reflects the current surgical state.
Experimental Validation
The system was evaluated using two primary datasets:
- Synthetic Dataset: 600 samples of adipose tissue manipulation were generated to provide ground truth knowledge for training BANet.
- Ex Vivo Validation: The system was tested on a real human specimen involving the kidney, pararenal fat, and Gerota’s fascia. This test provided a realistic assessment of the system's performance against sensor noise, occlusions, and complex tissue behaviors.
Key Results and Insights
The central innovation of this work is the realization that deformation can serve as a proxy for connection. By observing how tissue stretches when manipulated, the neural network can "reverse-engineer" the location of hidden anchors.
- Accuracy: In ex vivo human tissue tests, the average Root Mean Square Error (RMSE) remained below 5 mm, meeting the threshold for clinical acceptability in minimally invasive surgery.
- Performance Gain: Simulations with updated BCs were significantly more accurate than unconstrained models, where errors reached as high as 31 mm.
- Latency: The total pipeline latency was approximately 1.44 seconds. While displacement estimation accounts for the majority of this time (1.39s), the BANet inference is nearly instantaneous at 13 milliseconds.
Limitations and Future Directions
While the system demonstrates robust performance, several areas for improvement remain. The pipeline currently struggles with significant occlusions from surgical tools and high levels of surface noise during extreme deformations. Future work may involve incorporating temporal dynamics—treating the surgery as a sequence rather than static snapshots—and optimizing the system to handle complex topological changes such as tissue tearing or suturing.
Summary updated