Abstract
This work describes a regression model based on Convolutional Neural Networks (CNN) and Long-Short Term Memory (LSTM) networks for tracking objects from monocular video sequences. The target application being pursued is Vision-Based Sensor Substitution (VBSS). In particular, the tool-tip position and velocity in 3D space of a pair of surgical robotic instruments (SRI) are estimated for three surgical tasks, namely suturing, needle-passing and knot-tying. The CNN extracts features from individual video frames and the LSTM network processes these features over time and continuously outputs a 12-dimensional vector with the estimated position and velocity values. A series of analyses and experiments are carried out in the regression model to reveal the benefits and drawbacks of different design choices. First, the impact of the loss function is investigated by adequately weighing the Root Mean Squared Error (RMSE) and Gradient Difference Loss (GDL), using the VGG16 neural network for feature extraction. Second, this analysis is extended to a Residual Neural Network designed for feature extraction, which has fewer parameters than the VGG16 model, resulting in a reduction of ~96.44 % in the neural network size. Third, the impact of the number of time steps used to model the temporal information processed by the LSTM network is investigated. Finally, the capability of the regression model to generalize to the data related to "unseen" surgical tasks (unavailable in the training set) is evaluated. The aforesaid analyses are experimentally validated on the public dataset JIGSAWS. These analyses provide some guidelines for the design of a regression model in the context of VBSS, specifically when the objective is to estimate a set of 1D time series signals from video sequences.
Introduction
In Robotic Assisted Minimally Invasive Surgery (RAMIS), precise tracking of surgical instruments is essential for automated skill assessment and gesture recognition. While physical sensors can provide kinematic data in laboratory settings, their clinical application is hindered by sterilization requirements, cost, and size constraints. Vision-Based Sensor Substitution (VBSS) offers a solution by estimating these physical variables directly from video feeds. However, recovering 3D information from a 2D monocular image is an ill-posed problem due to the loss of depth information. Furthermore, estimating velocity requires the model to interpret temporal dynamics across multiple frames.
Methodology
The proposed architecture utilizes a serial combination of a Convolutional Neural Network (CNN) and a Long Short-Term Memory (LSTM) network.
Pre-processing and Feature Extraction
To emphasize motion, the system subtracts the background "mean frame" and concatenates three consecutive grayscale frames into a single "space-time image." For feature extraction, the study compares a standard VGG16 model with a custom, lightweight Residual Network (ResNet). The ResNet design significantly reduces computational overhead, achieving a 96.44% reduction in parameter count (4.92M vs. 138M) while maintaining competitive performance.
Temporal Modeling
The extracted spatial features are processed by an LSTM variant featuring Coupled Input-Forget Gates (CIFG). This network models the temporal dependencies of the surgical tools to output a continuous 12-dimensional vector representing the X, Y, and Z position and velocity for both the left and right instruments. The training process explored a hybrid loss function combining Root Mean Squared Error (RMSE) with Gradient Difference Loss (GDL) to improve signal "sharpness" and velocity estimation.
Results and Discussion
The models were evaluated using the JIGSAWS dataset, which contains 206 video sequences of suturing, needle-passing, and knot-tying tasks.
Performance Metrics
The system achieved high precision in position tracking, with Pearson Correlation Coefficients (PCC) often exceeding 0.90. Velocity estimation proved more challenging, with PCC scores ranging from 0.20 to 0.70, reflecting the inherent noise and complexity of derivative signals.
Key Findings
- Model Efficiency: The custom ResNet proved that massive architectures are not required for specialized regression tasks in VBSS.
- Temporal Window: Increasing the LSTM memory beyond 32 time steps did not yield significant improvements, suggesting that surgical tool motion has relatively short-term temporal dependencies.
- Task Specificity: Models trained on one task (e.g., suturing) generalized poorly to others (e.g., knot-tying), highlighting the need for diverse training data to handle the full spectrum of surgical maneuvers.
Conclusion
This research demonstrates the feasibility of using deep learning for high-precision 3D tracking of surgical tools from monocular video. While the CNN-LSTM framework provides a robust foundation, future work must address the generalization gap between different surgical tasks. Additionally, incorporating Explainable AI (XAI) techniques will be crucial for ensuring safety and transparency in clinical environments.
Summary updated