Abstract
Accurate future frame prediction in robot-assisted surgery (RAS) is critical for safety and automation. This paper introduces the Ternary Prior Guided Variational Autoencoder (TPG-VAE), a stochastic model that disentangles scene representation into content, motion, and high-level surgical gesture labels. By incorporating gesture labels as an invariant prior, TPG-VAE effectively models the complex, entangled movements of dual-arm surgical robots. Experiments on the JIGSAWS dataset demonstrate that TPG-VAE outperforms state-of-the-art methods in both quantitative metrics and visual fidelity, particularly over long-term prediction horizons.
Introduction
Predicting the "next few seconds" of a surgical video is a foundational challenge for medical AI. If a system can accurately anticipate a surgeon's next move, it can provide proactive safety alerts, automate sub-tasks, or assist in training junior surgeons via imitation demonstrations.
However, surgical video prediction is uniquely difficult compared to natural videos. Surgical tasks involve dual robotic arms (such as the da Vinci system) that move in highly coordinated, "entangled" ways. Furthermore, tool movements are often tiny and precise, making them easy for standard models to ignore as noise. Because the tissue background remains largely static, many models fail by simply "copy-pasting" the previous frame rather than predicting actual motion.
Methodology
The authors propose the Ternary Prior Guided Variational Autoencoder (TPG-VAE), a stochastic video prediction model that structures its understanding of a scene into three distinct parts (the "Ternary" prior):
- Content ($C_t$): A VGG-based encoder captures the spatial features of the scene, identifying the tools and tissue.
- Motion ($M_t$): Instead of raw frames, the model analyzes the difference between consecutive frames ($\Delta x_t$). This acts as an attention mechanism, forcing the model to focus on the small movements of the instruments.
- Class Label ($L_t$): This represents high-level surgical gestures (e.g., "positioning needle"). These labels act as an invariant prior to constrain the model's latent space, ensuring predicted movements are consistent with the current surgical phase.
The model uses LSTMs to handle temporal dependencies and is trained by maximizing the variational lower bound, using a combination of KL divergence to align learned priors and L1 reconstruction loss to ensure visual accuracy.
Experimental Setup
The model was validated using the JIGSAWS (JHU-ISI Gesture and Skill Assessment Working Set) dataset, specifically the "Suturing" task performed on a da Vinci surgical robot. The study utilized 470 sequences for training and 142 for testing, employing a "Leave-One-User-Out" (LOUO) cross-validation setup. Videos were downsampled to 64x64 resolution to manage computational load.
Results and Key Insights
TPG-VAE was compared against several baselines, including MCnet (deterministic) and SVG-LP (stochastic).
- Quantitative Performance: TPG-VAE achieved superior scores in PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index), especially as the prediction horizon grew. At 20 frames into the future ($t=30$), TPG-VAE maintained a PSNR of 23.67, while MCnet dropped to 18.79.
- Qualitative Fidelity: Visual results demonstrate that while other models produced blurry "ghosts" of the robotic arms or failed to move them at all, TPG-VAE generated crisp frames where the arms followed the actual ground-truth trajectories.
A key insight from the study is that Motion acts as Attention. By explicitly encoding frame differences, the model avoids blurry predictions and learns that moving pixels are more critical than the static background. Furthermore, the inclusion of the gesture class label ($L_t$) significantly reduced uncertainty; ablation studies showed that knowing the surgical intent was more helpful for long-term prediction than raw motion data alone.
Conclusion and Future Work
While TPG-VAE proves the value of domain-specific priors, limitations remain. The current experiments were conducted at 64x64 resolution, whereas clinical applications require higher fidelity. Additionally, the model currently relies on ground-truth gesture labels. Future research will focus on end-to-end systems that predict gestures and frames simultaneously, potentially enabling real-time safety monitoring where AI could intervene if a predicted future indicates a potential collision or tissue damage.
Summary updated