ProMamba: Prompt-Mamba for polyp segmentation

Jianhao Xie, Ruofan Liao, Ziang Zhang, Sida Yi, Yuesheng Zhu, Guibo LuoarXiv 2024 · OpenReview

TL;DR

This paper introduces ProMamba, a novel polyp segmentation model that combines the powerful Vision-Mamba architecture with prompt-based segmentation. ProMamba achieves state-of-the-art performance, particularly on unseen datasets, demonstrating superior generalization capabilities compared to existing methods.

What Problem Does This Paper Address?

Accurate polyp segmentation during colonoscopy is crucial for early cancer detection and treatment planning. However, this task is challenging due to the diverse shapes and colors of polyps, their often unclear boundaries with surrounding tissues, and the significant differences across various medical imaging datasets, which hinder the generalization of existing models.

Methods

The ProMamba model is built upon a Vision-Mamba backbone, a recent architecture that has shown promise in computer vision tasks by efficiently processing sequential data. The authors are the first to apply Vision-Mamba to polyp segmentation. They integrate prompt technology, inspired by models like SAM (Segment Anything Model), to guide the segmentation process. Specifically, they use box prompts during training, which they found to be more stable than point prompts. The model comprises three main components: an image encoder (using Vision-Mamba), a prompt encoder, and a mask decoder. The loss function is a combination of Dice loss and Focal loss, chosen for their effectiveness in handling unbalanced data and improving segmentation accuracy.

Datasets

The experiments were conducted on a merged dataset comprising six public datasets: ETIS, CVC-ClinicDB/CVC-612, CVC-ColonDB, EndoScene-CVC-300, Kvasir, and BKAI. The data was partitioned into training, validation, and test sets, with BKAI specifically added to the test set.

Key Insights and Ideas

  • Vision-Mamba for Medical Imaging: This paper pioneers the use of Vision-Mamba in polyp segmentation, showcasing its potential for extracting complex features in medical images.
  • Prompting for Generalization: The integration of prompt technology, particularly box prompts, significantly enhances the model's ability to generalize to unseen datasets, a critical challenge in medical image analysis.
  • Efficient Architecture: ProMamba demonstrates that even smaller versions of the model, with fewer parameters, can outperform larger, established models, highlighting architectural efficiency.
  • Bidirectional SSM: The inclusion of a backward SSM branch in the Vision-Mamba encoder is noted as a way to better capture image feature values, contributing to improved performance.

Key Results

ProMamba achieves an average of 5% improvement in segmentation accuracy across six datasets compared to previous state-of-the-art methods. Notably, it exhibits superior performance on unseen datasets, indicating strong generalization capabilities. The paper also presents scaled versions of their model, showing that even a model with as few as 11M parameters can outperform larger models like ParNet and nnU-Net.

Limitations and Future Work

The authors acknowledge that their model's performance on the seen validation dataset is slightly lower than the best existing methods, attributing this to the relatively small training data size for those specific datasets. They plan to release their code and trained weights. Future work includes exploring other types of prompts (e.g., text prompts) and developing pre-trained models with more data to further enhance results and generalization.

Summary updated

Abstract

Detecting polyps through colonoscopy is an important task in medical image segmentation, which provides significant assistance and reference value for clinical surgery. However, accurate segmentation of polyps is a challenging task due to two main reasons. Firstly, polyps exhibit various shapes and colors. Secondly, the boundaries between polyps and their normal surroundings are often unclear. Additionally, significant differences between different datasets lead to limited generalization capabilities of existing methods. To address these issues, we propose a segmentation model based on Prompt-Mamba, which incorporates the latest Vision-Mamba and prompt technologies. Compared to previous models trained on the same dataset, our model not only maintains high segmentation accuracy on the validation part of the same dataset but also demonstrates superior accuracy on unseen datasets, exhibiting excellent generalization capabilities. Notably, we are the first to apply the Vision-Mamba architecture to polyp segmentation and the first to utilize prompt technology in a polyp segmentation model. Our model efficiently accomplishes segmentation tasks, surpassing previous state-of-the-art methods by an average of 5% across six datasets. Furthermore, we have developed multiple versions of our model with scaled parameter counts, achieving better performance than previous models even with fewer parameters. Our code and trained weights will be released soon.
BibTeX
@misc{xie2024promamba,
  author = {Jianhao Xie and Ruofan Liao and Ziang Zhang and Sida Yi and Yuesheng Zhu and Guibo Luo},
  title = {{ProMamba: Prompt-Mamba for polyp segmentation}},
  year = {2024},
  eprint = {2403.13660},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.13660},
}
Figure