A Foundation Model for General Moving Object Segmentation in Medical Images

Zhongnuo Yan, Tong Han, Yuhao Huang, Lian Liu, Han Zhou, Jiongquan Chen, Wenlong Shi, Yan Cao, Xin Yang, Dong Ni

Abstract

Medical image segmentation aims to delineate the anatomical or pathological structures of interest, playing a crucial role in clinical diagnosis. A substantial amount of high-quality annotated data is crucial for constructing high-precision deep segmentation models. However, medical annotation is highly cumbersome and time-consuming, especially for medical videos or 3D volumes, due to the huge labeling space and poor inter-frame consistency. Recently, a fundamental task named Moving Object Segmentation (MOS) has made significant advancements in natural images. Its objective is to delineate moving objects from the background within image sequences, requiring only minimal annotations. In this paper, we propose the first foundation model, named iMOS, for MOS in medical images. Extensive experiments on a large multi-modal medical dataset validate the effectiveness of the proposed iMOS. Specifically, with the annotation of only a small number of images in the sequence, iMOS can achieve satisfactory tracking and segmentation performance of moving objects throughout the entire sequence in bi-directions. We hope that the proposed iMOS can help accelerate the annotation speed of experts, and boost the development of medical foundation models.

Introduction

In medical imaging, 3D volumes (like CT/MRI) and videos (like Ultrasound/Endoscopy) are essentially sequences of 2D slices. Traditionally, segmenting these requires experts to annotate dozens or hundreds of frames manually—a process that is notoriously slow, prone to inter-frame inconsistency, and mentally taxing. While general-purpose models like the Segment Anything Model (SAM) have revolutionized 2D segmentation, they often struggle with temporal consistency, requiring new prompts for every frame. The iMOS model addresses this "annotation bottleneck" by introducing Moving Object Segmentation (MOS) to the medical domain, aiming to propagate a single manual mask across an entire sequence automatically.

Methods

The iMOS architecture is built upon XMem, a state-of-the-art framework for video object segmentation. The core innovation lies in its multi-tier memory management and domain adaptation strategy.

The iMOS architecture featuring a three-tier memory system and parameter-efficient adapters.

1. The Three-Tier Memory System

To handle sequences of varying lengths, iMOS uses three distinct memory stores:

  • Sensory Memory: Updates every frame to maintain short-term temporal consistency using Gated Recurrent Units (GRUs).
  • Working Memory: Stores high-resolution features from recent frames for precise matching.
  • Long-term Memory: Uses "Prototype Selection" to store only the most representative features of the object over the entire sequence, preventing GPU memory overflow.

2. Parameter-Efficient Fine-Tuning (PEFT)

Instead of retraining a massive model, the authors used Adapters. By keeping pre-trained weights from natural images fixed and inserting small, trainable "Adapter modules" (parallel convolutional layers), the model learns medical-specific nuances while retaining robust tracking logic.

3. Bi-directional Inference

Unlike standard video models, iMOS propagates masks both forward and backward. If a clinician annotates a slice in the middle of a volume, the model segments the slices above and below it simultaneously.

Datasets and Results

The researchers validated iMOS on a massive, multi-modal dataset spanning five modalities: CT, MRI, Ultrasound, Endoscopy, and Electron Microscopy. This diversity ensures the model acts as a general-purpose foundation model rather than a specialized organ segmenter.

Examples of the diverse medical imaging modalities (CT, MRI, Ultrasound, Endoscope, Electron Microscopy) used to validate iMOS.

Key Findings

  • Performance Boost: Fine-tuning with adapters led to significant gains, such as a 28.7% improvement in the $J\&F$ score for MRI scans compared to the base model.
  • High Accuracy: Across most categories, the model achieved $F$-scores above 0.8, with some reaching 0.95 (e.g., mitochondria in electron microscopy).
  • Zero-Shot Generalization: The model demonstrated an ability to segment organs it had never seen during training (e.g., the esophagus or aorta), provided it was given an initial first-frame mask.

Conclusion

iMOS represents a significant step toward reducing the manual burden of medical image annotation. By leveraging the logic of natural video tracking and adapting it through efficient memory management and PEFT, it achieves high-precision bi-directional segmentation. Future work may explore unsupervised MOS and the integration of interactive prompting systems to further enhance clinical utility.

Summary updated

Abstract

Medical image segmentation aims to delineate the anatomical or pathological structures of interest, playing a crucial role in clinical diagnosis. A substantial amount of high-quality annotated data is crucial for constructing high-precision deep segmentation models. However, medical annotation is highly cumbersome and time-consuming, especially for medical videos or 3D volumes, due to the huge labeling space and poor inter-frame consistency. Recently, a fundamental task named Moving Object Segmentation (MOS) has made significant advancements in natural images. Its objective is to delineate moving objects from the background within image sequences, requiring only minimal annotations. In this paper, we propose the first foundation model, named iMOS, for MOS in medical images. Extensive experiments on a large multi-modal medical dataset validate the effectiveness of the proposed iMOS. Specifically, with the annotation of only one image in the sequence, iMOS can achieve satisfactory tracking and segmentation performance of moving objects throughout the entire sequence in bi-directions. We hope that iMOS can help accelerate the annotation speed of experts, and boost the development of medical foundation models.
BibTeX
@misc{yan2023foundation,
  author = {Zhongnuo Yan and Tong Han and Yuhao Huang and Lian Liu and Han Zhou and Jiongquan Chen and Wenlong Shi and Yan Cao and Xin Yang and Dong Ni},
  title = {{A Foundation Model for General Moving Object Segmentation in Medical Images}},
  year = {2023},
  eprint = {2309.17264},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.17264},
}
Figure