Future Frame Prediction for Robot-assisted Surgery

Xiaojie Gao, Yueming Jin, Zixu Zhao, Qi Dou, Pheng-Ann Heng

Abstract

Accurate future frame prediction in robot-assisted surgery (RAS) is critical for safety and automation. This paper introduces the Ternary Prior Guided Variational Autoencoder (TPG-VAE), a stochastic model that disentangles scene representation into content, motion, and high-level surgical gesture labels. By incorporating gesture labels as an invariant prior, TPG-VAE effectively models the complex, entangled movements of dual-arm surgical robots. Experiments on the JIGSAWS dataset demonstrate that TPG-VAE outperforms state-of-the-art methods in both quantitative metrics and visual fidelity, particularly over long-term prediction horizons.

Introduction

Predicting the "next few seconds" of a surgical video is a foundational challenge for medical AI. If a system can accurately anticipate a surgeon's next move, it can provide proactive safety alerts, automate sub-tasks, or assist in training junior surgeons via imitation demonstrations.

However, surgical video prediction is uniquely difficult compared to natural videos. Surgical tasks involve dual robotic arms (such as the da Vinci system) that move in highly coordinated, "entangled" ways. Furthermore, tool movements are often tiny and precise, making them easy for standard models to ignore as noise. Because the tissue background remains largely static, many models fail by simply "copy-pasting" the previous frame rather than predicting actual motion.

Methodology

The authors propose the Ternary Prior Guided Variational Autoencoder (TPG-VAE), a stochastic video prediction model that structures its understanding of a scene into three distinct parts (the "Ternary" prior):

  • Content ($C_t$): A VGG-based encoder captures the spatial features of the scene, identifying the tools and tissue.
  • Motion ($M_t$): Instead of raw frames, the model analyzes the difference between consecutive frames ($\Delta x_t$). This acts as an attention mechanism, forcing the model to focus on the small movements of the instruments.
  • Class Label ($L_t$): This represents high-level surgical gestures (e.g., "positioning needle"). These labels act as an invariant prior to constrain the model's latent space, ensuring predicted movements are consistent with the current surgical phase.
The TPG-VAE architecture disentangles scene representation into content, motion, and gesture-based class labels to improve prediction accuracy.

The model uses LSTMs to handle temporal dependencies and is trained by maximizing the variational lower bound, using a combination of KL divergence to align learned priors and L1 reconstruction loss to ensure visual accuracy.

Experimental Setup

The model was validated using the JIGSAWS (JHU-ISI Gesture and Skill Assessment Working Set) dataset, specifically the "Suturing" task performed on a da Vinci surgical robot. The study utilized 470 sequences for training and 142 for testing, employing a "Leave-One-User-Out" (LOUO) cross-validation setup. Videos were downsampled to 64x64 resolution to manage computational load.

Results and Key Insights

TPG-VAE was compared against several baselines, including MCnet (deterministic) and SVG-LP (stochastic).

  • Quantitative Performance: TPG-VAE achieved superior scores in PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index), especially as the prediction horizon grew. At 20 frames into the future ($t=30$), TPG-VAE maintained a PSNR of 23.67, while MCnet dropped to 18.79.
  • Qualitative Fidelity: Visual results demonstrate that while other models produced blurry "ghosts" of the robotic arms or failed to move them at all, TPG-VAE generated crisp frames where the arms followed the actual ground-truth trajectories.
Qualitative comparison showing TPG-VAE's ability to maintain crisp tool trajectories compared to the blurry or static predictions of baseline models.

A key insight from the study is that Motion acts as Attention. By explicitly encoding frame differences, the model avoids blurry predictions and learns that moving pixels are more critical than the static background. Furthermore, the inclusion of the gesture class label ($L_t$) significantly reduced uncertainty; ablation studies showed that knowing the surgical intent was more helpful for long-term prediction than raw motion data alone.

Conclusion and Future Work

While TPG-VAE proves the value of domain-specific priors, limitations remain. The current experiments were conducted at 64x64 resolution, whereas clinical applications require higher fidelity. Additionally, the model currently relies on ground-truth gesture labels. Future research will focus on end-to-end systems that predict gestures and frames simultaneously, potentially enabling real-time safety monitoring where AI could intervene if a predicted future indicates a potential collision or tissue damage.

Summary updated

Abstract

Predicting future frames for robotic surgical video is an interesting, important yet extremely challenging problem, given that the operative tasks may have complex dynamics. Existing approaches on future prediction of natural videos were based on either deterministic models or stochastic models, including deep recurrent neural networks, optical flow, and latent space modeling. However, the potential in predicting meaningful movements of robots with dual arms in surgical scenarios has not been tapped so far, which is typically more challenging than forecasting independent motions of one arm robots in natural scenarios. In this paper, we propose a ternary prior guided variational autoencoder (TPG-VAE) model for future frame prediction in robotic surgical video sequences. Besides content distribution, our model learns motion distribution, which is novel to handle the small movements of surgical tools. Furthermore, we add the invariant prior information from the gesture class into the generation process to constrain the latent space of our model. To our best knowledge, this is the first time that the future frames of dual arm robots are predicted considering their unique characteristics relative to general robotic videos. Experiments demonstrate that our model gains more stable and realistic future frame prediction scenes with the suturing task on the public JIGSAWS dataset.
BibTeX
@inproceedings{gao2021future,
  author = {Xiaojie Gao and Yueming Jin and Zixu Zhao and Qi Dou and Pheng-Ann Heng},
  title = {{Future Frame Prediction for Robot-assisted Surgery}},
  booktitle = {IPMI},
  year = {2021},
  url = {https://openreview.net/forum?id=C7FqHRCSJlQ},
}
Figure