Feature-Space Latent Forecasting for Endoscopic Surgical Video Understanding

Songtao Hu, Hang SuarXiv 2026 · arXiv

TL;DR

Researchers have developed a new framework for surgical video anticipation that skips the "pretty pictures" of future frame synthesis to predict semantic features instead. By forecasting latent representations within a frozen Vision Foundation Model (VFM), the system achieves significantly better accuracy in predicting surgical phases and tool usage than traditional pixel-based methods, even in the presence of smoke and specular highlights.

What Problem Does This Paper Address?

In the operating room, a reactive AI is a late AI. For a system to be truly useful—warning of upcoming phase transitions or preparing instruments—it must be anticipatory.

Most current video prediction models try to "imagine" future frames in pixel space. While this works for natural videos (like a person walking in a park), it fails miserably in endoscopic surgery. Laparoscopic environments are chaotic: smoke from cauterization, specular highlights on wet tissue, and sudden occlusions make pixel-level statistics incredibly unstable.

The authors argue that we don't actually need to see a photorealistic "hallucination" of the next 5 seconds of surgery. What we need is the semantic essence of that future. The gap between visual fidelity (how real it looks) and downstream utility (how useful it is for a task) is the core challenge this paper tackles.

Methods

The authors propose a Feature-Space Latent Forecasting framework. Instead of training a model to generate pixels, they train it to predict the internal "thoughts" (features) of a Vision Foundation Model (VFM).

  1. Frozen Backbone: They use a pre-trained Vision Transformer (ViT) as a feature extractor and keep it frozen. This ensures the model relies on robust, pre-learned visual concepts.
  2. Hierarchical Representation: They don't just take the final layer of the ViT. They concatenate features from multiple layers to capture both fine-grained details (like tool tips) and high-level procedural context.
  3. Dimensionality Reduction: To make the target space manageable, they use an offline PCA (Principal Component Analysis) projection to create a compact "target feature space."
  4. Masked Transformer Predictor: This is the heart of the system. They take a sequence of frames, keep the observed ones as they are, and replace the future frames with learnable [MASK] tokens. A transformer then predicts what the features in those masked slots should be.
  5. Plug-and-Play Heads: The predicted future features are fed into lightweight "heads" that perform specific tasks like phase recognition or tool detection.

Datasets

The model was evaluated on Cholec80, the gold-standard benchmark for laparoscopic cholecystectomy (gallbladder removal). It consists of 80 videos with dense annotations for:

  • Surgical Phase: (e.g., Preparation, Calot Triangle Dissection).
  • Tool Presence: Seven different instruments (e.g., Grasper, Hook, Scissors).
  • Remaining Time: Estimating how long until the next phase transition.

Key Insights and Ideas

  • Semantic Stability > Visual Fidelity: By forecasting in feature space, the model ignores "transient photometric artifacts" (like a flash of light or a puff of smoke) that would confuse a pixel-based model but don't change the underlying surgical logic.
  • Decoupling is Key: By freezing the VFM and only training the temporal "forecaster," the authors decoupled representation learning from temporal modeling. This makes the system much more training-efficient and less prone to overfitting on low-level appearance noise.
  • The "Oracle" Gap: The paper uses an "Oracle" baseline (using actual future features) to show the upper bound of performance. Their method closes the gap between simple persistence (assuming the future looks like the present) and the Oracle significantly better than RGB-based models.

Key Results

The proposed method consistently outperformed RGB-based forecasting (like SimVP) across all metrics:

  • Phase Anticipation: Achieved 67.53% accuracy, a massive jump over the RGB baseline's 51.18%.
  • Tool Presence: Improved the macro-F1 score from 0.4145 (RGB) to 0.5521. It was particularly good at identifying context-sensitive tools like the Bipolar cautery and Clippers.
  • Horizon Sensitivity: Remarkably, the model's predictions for 5 seconds into the future were more accurate than the RGB baseline's predictions for just 1 second into the future.
  • Remaining Time to Transition (RTT): The model reduced the weighted Mean Absolute Error (wMAE) across all horizons (2, 3, and 5 minutes), proving it can track long-range procedural evolution.

Limitations and Future Work

  • Frozen Bias: The system is only as good as its frozen VFM. If the VFM wasn't trained on enough surgical data, it might miss subtle tissue changes or uncommon instruments.
  • Deterministic Nature: The current model is deterministic—it predicts one "average" future. Surgery is often non-linear; future work could involve probabilistic forecasting to account for multiple possible surgical paths.
  • PCA Compression: While PCA makes the model efficient, it might discard low-variance information that is actually critical for rare, high-stakes clinical events. The authors suggest exploring more selective compression strategies in the future.

Summary updated

Abstract

Surgical workflow assistance is inherently anticipatory: systems must reason about how procedures evolve, not just recognize the current frame. However, many prior approaches perform prediction in pixel space by synthesizing future frames, which primarily focuses on appearance fidelity rather than the semantic representations required for downstream reasoning. To address this limitation, we propose a feature-space semantic forecasting framework that models temporal dynamics directly within the representations of a frozen Vision Foundation Model (VFM). Specifically, we build a hierarchical target representation from multi-layer VFM features, replace future-frame feature tokens with learnable \texttt{[MASK]} tokens, and train a masked transformer to forecast the corresponding future latent features from observed context, thereby formulating procedural evolution as masked future feature prediction. Experiments on public laparoscopic benchmarks demonstrate consistent improvements in phase anticipation, multi-label tool presence prediction, and remaining-time estimation over RGB-based forecasting baselines, while enabling a plug-and-play representation interface for lightweight downstream heads.
BibTeX
@misc{hu2026feature,
  author = {Songtao Hu and Hang Su},
  title = {{Feature-Space Latent Forecasting for Endoscopic Surgical Video Understanding}},
  year = {2026},
  eprint = {2611.18750},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18750},
}
Figure