Surgical-MambaLLM: Mamba2-Enhanced Multimodal Large Language Model for VQLA in Robotic Surgery

Pengfei Hao, Hongqiu Wang, Shuaibo Li, Zhaohu Xing, Guang Yang, Kaishun Wu, Lei Zhu

Introduction

In the high-stakes environment of robotic surgery, real-time decision support systems can significantly enhance surgical safety and training. A critical component of such systems is Surgical Visual Question Localized-Answering (Surgical-VQLA). This task requires an AI model to not only answer complex questions regarding the surgical scene—such as the state or function of a specific instrument—but also to provide precise spatial grounding through bounding boxes.

Existing solutions for Surgical-VQLA primarily utilize Transformer-based architectures. While effective at capturing global context, Transformers often struggle with the fine-grained local details necessary for identifying small surgical tool tips within a cluttered surgical cavity. Furthermore, standard vision models typically process images using linear scanning patterns (e.g., top-to-bottom), which fail to account for the unique radial geometry of surgical scenes where instruments typically converge from the periphery toward a central operative site. To address these limitations, we introduce Surgical-MambaLLM, the first framework to integrate the Mamba2 state-space model with a Large Language Model (LLM) specifically optimized for the surgical domain.

Methodology

Surgical-MambaLLM leverages the computational efficiency and long-sequence modeling capabilities of the Mamba2 architecture to improve multimodal fusion and spatial awareness.

Cross-modal Bidirectional Mamba2 Integration (CBMI)

Traditional multimodal models often employ "lazy" fusion, simply concatenating visual and textual features before feeding them into a reasoning engine. Surgical-MambaLLM introduces the CBMI module, which utilizes bidirectional Mamba2 blocks to deeply integrate visual tokens with textual queries. This approach allows the model to establish stronger dependencies between the specific wording of a question and the relevant pixels in the surgical feed.

Surgical Instrument Perception (SIP) Scanning

A key innovation of this framework is the Surgical Instrument Perception (SIP) scanning mode. Recognizing that surgical instruments are generally oriented radially, the SIP mode replaces standard 1D or cross-scanning patterns with a radial scan that moves from the center of the image outward in four directions. This ensures that the features of a single instrument remain contiguous in the data sequence, preserving the structural integrity of the tools and enhancing the model's ability to localize them accurately.

Two-Stage Training Strategy

The model follows a structured training pipeline to maximize the reasoning potential of the underlying LLM (InternLM-7B):

  1. Alignment Stage: The vision encoder, CBMI module, and projector are trained to align visual features with the LLM's embedding space while the LLM remains frozen.
  2. Fine-tuning Stage: The LLM is fine-tuned using Low-Rank Adaptation (LoRA), allowing the model to specialize its linguistic reasoning for surgical terminology and procedural logic without the computational cost of full parameter updates.

Experimental Evaluation

Datasets

The model was evaluated on two benchmark datasets derived from the MICCAI Endoscopic Vision Challenges:

  • EndoVis-18-VQLA: Comprising video sequences from 14 surgeries, this dataset includes 12,741 training and 3,820 test QA pairs.
  • EndoVis-17-VQLA: Used for external validation to assess the model's ability to generalize to unseen surgical environments, containing 708 QA pairs.

Performance Results

Surgical-MambaLLM achieved state-of-the-art (SOTA) results across both benchmarks:

  • EndoVis-18: The model reached an accuracy of 69.64% and a mean Intersection over Union (mIoU) of 0.8027, significantly outperforming the previous leading model, Surgical-VQLA++ (65.73% accuracy).
  • Generalization: On the EndoVis-17 dataset, the model maintained a superior accuracy of 51.91%, demonstrating robust performance in novel surgical scenarios.
  • Ablation Studies: Comparisons confirmed that the SIP radial scanning mode outperformed standard 1D, Bi-Scan, and Cross-Scan methods, validating the hypothesis that geometry-aware scanning is crucial for surgical tool perception.

Discussion and Future Work

The success of Surgical-MambaLLM highlights the potential of State Space Models (SSMs) like Mamba2 in specialized multimodal domains. By moving away from the "one-size-fits-all" approach of standard Transformers and adopting geometry-aware scanning, the model achieves a more nuanced understanding of the surgical field.

However, limitations remain. While the model leads in accuracy, its mIoU on the external EndoVis-17 dataset was slightly lower than some specialized grounding models, suggesting that the precision of its bounding boxes can be further refined for unseen data. Future research will focus on enhancing these grounding capabilities, expanding the training data to include a wider variety of surgical procedures, and further optimizing the reasoning efficiency of the LLM within the operating room environment.

Summary updated

BibTeX
@inproceedings{hao2025surgical,
  author = {Pengfei Hao and Hongqiu Wang and Shuaibo Li and Zhaohu Xing and Guang Yang and Kaishun Wu and Lei Zhu},
  title = {{Surgical-MambaLLM: Mamba2-Enhanced Multimodal Large Language Model for VQLA in Robotic Surgery}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2025},
  doi = {10.1007/978-3-032-05114-1\_55},
}
Figure