One to Many: Adaptive Instrument Segmentation via Meta Learning and Dynamic Online Adaptation in Robotic Surgical Video

Zixu Zhao, Yueming Jin, Bo Lu, Chi-Fai Ng, Qi Dou, Yun-Hui Liu, Pheng-Ann Heng

TL;DR

This paper introduces MDAL, a novel meta-learning framework for adapting surgical instrument segmentation models to new robotic surgery videos with minimal annotations. By learning to adapt quickly from a few examples, MDAL significantly improves label efficiency and performance in real-world surgical scenarios where data distribution shifts are common.

What Problem Does This Paper Address?

Surgical instrument segmentation is crucial for robotic-assisted surgery (RAS), enabling tasks like tool tracking, pose estimation, and even autonomous camera control. Traditional deep learning models for this task require large amounts of annotated data. However, collecting and annotating data for every new surgical setting (e.g., different surgical robots, procedures, or even individual surgeons) is prohibitively expensive and time-consuming. This "domain shift" problem, where models trained on one dataset perform poorly on another, is a major hurdle. Existing methods often assume training and testing data come from the same domain or require extensive fine-tuning on new data, which is impractical given the scarcity of surgical annotations. This paper tackles the challenge of adapting a single source model to multiple new target domains with only the first frame of a new video annotated.

Methods

The proposed method, MDAL (Meta-learning based Dynamic Online Adaptive Learning), operates in two stages:

  1. Meta-Training (Offline): This stage trains a "meta-learner" on a diverse set of surgical videos (source domain). The goal is not just to learn to segment instruments but to learn how to adapt to new segmentation tasks quickly. A "video-level task space" is introduced, where each task involves segmenting a pair of frames from a video. This encourages the model to learn generalizable instrument representations. The meta-learner learns optimal initial parameters and a step size for fast adaptation.
  2. Dynamic Online Adaptation (Testing): When presented with a new surgical video (target domain), MDAL first uses the learned meta-parameters to quickly adapt the segmentation model to the first annotated frame. This "meta-adaptation" requires only a few gradient descent steps. To handle variations in subsequent frames, MDAL employs a "dynamic online adaptation" strategy. This involves generating "pseudo-masks" for frames beyond the first. Crucially, a "gradient gate" is used to selectively update the model. This gate assesses the quality of the pseudo-mask by comparing the model's performance on the first frame with and without the update. If the update is detrimental (due to noisy pseudo-masks), the model is not updated for that frame, preventing performance degradation. This selective adaptation improves both accuracy and efficiency.

Datasets

  • Source Domain: EndoVis17 dataset, a public challenge dataset from 2017 MICCAI EndoVis Instrument Segmentation, containing in-vivo surgical video sequences from da Vinci Xi surgical systems.
  • Target Domains:
    • EndoVis18: A more complex dataset from the 2018 MICCAI challenge, featuring porcine procedures.
  • HKPWH: A newly collected real-world RAS dataset from Hong Kong Prince of Wales Hospital, containing in-vivo prostatectomy surgeries.
  • DSS (dVRK-based suturing videos) and PSH (phantom-based simulated hysterectomy videos): Used to demonstrate downstream application potential.

Key Insights and Ideas

  • Learning to Adapt is Key: The core idea is that instead of training a model to perform well on a specific task, we train it to become a good "learner" that can quickly adapt to new tasks with minimal data. This meta-learning approach is crucial for handling the domain shift problem in RAS.
  • First Frame is Gold: Leveraging the single annotated first frame as the primary source of supervision for adaptation is a practical and efficient strategy, significantly reducing annotation burden.
  • Dynamic Adaptation with a "Gradient Gate": The proposed gradient gate is a clever mechanism to prevent noisy pseudo-masks from corrupting the model's performance. By selectively updating only when beneficial, it ensures more stable and robust online adaptation.
  • Domain Shift is a Major Challenge: The paper strongly emphasizes that real-world surgical videos from different sources exhibit significant distribution shifts, making direct deployment of models trained on one dataset impractical.

Key Results

  • MDAL achieves state-of-the-art performance on the EndoVis18 and HKPWH datasets, with average IoU of 75.5% and Dice of 84.9%.
  • The meta-adaptation stage significantly reduces the need for fine-tuning iterations. While a standard fine-tuned model might need 100 iterations, MDAL achieves superior results with just 5 iterations.
  • The dynamic online adaptation with the gradient gate shows consistent improvement over naive online adaptation, particularly in the later frames of a sequence, demonstrating its effectiveness in handling temporal variations.
  • Qualitative results show MDAL's ability to handle challenging scenarios like fast motion and multiple instruments, producing robust segmentation.
  • The method shows promise for downstream tasks like robot-assisted suturing and autonomous camera control.

Limitations and Future Work

  • Inference Speed: While the current inference speed (around 4 fps) is promising for some tasks, the authors acknowledge that increasing this speed is necessary for wider application in robotic automation.
  • Future Work: The authors plan to explore incorporating lightweight networks and learning to update only parts of the parameters to further improve inference speed and adaptation efficiency.

Summary updated

BibTeX
@inproceedings{zhao2021one,
  author = {Zixu Zhao and Yueming Jin and Bo Lu and Chi-Fai Ng and Qi Dou and Yun-Hui Liu and Pheng-Ann Heng},
  title = {{One to Many: Adaptive Instrument Segmentation via Meta Learning and Dynamic Online Adaptation in Robotic Surgical Video}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  doi = {10.1109/icra48506.2021.9561690},
}

Loading related content...

Figure