Abstract
For Minimally Invasive Surgical (MIS) robots, accurate haptic interaction force feedback is essential for ensuring the safety of interacting with soft tissue. However, most existing MIS robotic systems cannot facilitate direct measurement of the interaction force with hardware sensors due to space limitations. This letter introduces an effective vision-based scheme that utilizes a One-Shot structured light projection with a designed pattern on soft tissue coupled with haptic information processing through a trained image-to-force neural network. The images captured from the endoscopic stereo camera are analyzed to reconstruct high-resolution 3D point clouds for soft tissue deformation. Based on this, a modified PointNet-based force estimation method is proposed, which excels in representing the complex mechanical properties of soft tissue. Numerical force interaction experiments are conducted on three silicon materials with different stiffness. The results validate the effectiveness of the proposed scheme.
Introduction
In Minimally Invasive Surgery (MIS), surgeons operate through small incisions using robotic tools. A significant limitation of current systems is the "loss of touch"—the absence of haptic feedback. Without the ability to feel the resistance of delicate tissues, surgeons risk applying excessive force during tasks like suturing or retraction.
While physical force sensors are a potential solution, they are difficult to integrate into tiny surgical end-effectors due to space constraints, high costs, and the need to withstand harsh sterilization environments. Previous vision-based attempts to estimate force by observing tissue deformation have struggled with textureless human organs and have often focused solely on compression (pushing) rather than the complex traction (pulling) tasks common in surgery.
Methods
The proposed Vision-Based Force Estimation (VBFE) framework consists of two primary stages: active 3D reconstruction and force regression via a modified neural network.
Active 3D Reconstruction
To overcome the challenge of featureless tissue surfaces, the system employs a "One-Shot" structured light projection. A specific color pattern, based on a De Bruijn sequence, is projected onto the tissue. This creates artificial features that enable a stereo camera to reconstruct a dense, pixel-wise 3D point cloud of the tissue's surface in real-time.
Modified PointNet for Force Regression
While standard PointNet architectures are typically used for object classification, this study modifies the network for regression to predict continuous force values.
Key architectural enhancements include:
- Input Preprocessing: Point clouds are normalized to a unit sphere to ensure a consistent representation regardless of absolute position.
- Activation Functions: The standard ReLU is replaced with ELU (Exponential Linear Unit). ELU is better suited for capturing the non-linear mechanical properties of soft tissue and prevents the "dying neuron" problem.
- Optimizer: The Nadam optimizer is utilized for faster convergence and improved accuracy in force prediction.
Experimental Setup and Datasets
The system was validated using a custom dataset collected with a Toumai laparoscopic surgical robot.
- Hardware: Ground truth force measurements were provided by a force-sensing trocar (Model TRF85D), while visual data was captured by a 3D electronic endoscope.
- Materials: Three silicone tubes with varying stiffness (40 N/m, 80 N/m, and 200 N/m) were used to simulate different biological tissues.
- Open Source: The dataset is available on GitHub to support further research in medical haptics.
Key Results and Discussion
Validation involved 40 repeated traction experiments for each material. The system demonstrated high accuracy, particularly on softer materials:
- Performance on Soft Tissue: For the 40 N/m material, the Mean Absolute Error (MAE) was 0.4055 N, with a Root Mean Square Error (RMSE) of 0.3023 N.
- Stiffness Correlation: Accuracy was found to be inversely proportional to material stiffness. Stiffer materials (e.g., 200 N/m) exhibited higher error (MAE of 0.8044 N) because they deform less under the same force, resulting in a weaker visual signal for the network to interpret.
- Traction Capabilities: Unlike models limited to compression, this system successfully estimated forces during "pull-and-hold" sequences, which are critical for surgical manipulation.
Conclusion and Future Work
This research presents a viable path toward sensorless haptic feedback in MIS. By projecting structured light, the system generates high-resolution 3D data even on featureless surfaces, which a modified PointNet then translates into accurate force estimates.
Future work will focus on miniaturizing the projection hardware for clinical use and incorporating temporal modeling (such as LSTMs) to better account for the viscoelastic properties of real biological tissue.
Summary updated