TL;DR
This paper introduces SimTxtSeg, a novel framework for weakly-supervised medical image segmentation that leverages simple text cues to generate high-quality pseudo-labels. By effectively converting text into visual prompts and integrating them into a segmentation model via a text-vision hybrid attention mechanism, SimTxtSeg achieves state-of-the-art performance on colonic polyp and brain tumor segmentation tasks with significantly reduced annotation costs.
What Problem Does This Paper Address?
Medical image segmentation is crucial for diagnosis and treatment planning, but it traditionally requires extensive pixel-level annotations, which are costly and time-consuming to obtain from expert clinicians. While weakly-supervised methods aim to alleviate this burden using less precise labels (e.g., image-level, point-level, bounding boxes), their performance often falls short of fully-supervised approaches. The authors identify a gap in generating high-quality, low-cost pseudo-labels that can effectively train segmentation models, particularly by exploring the potential of simple text descriptions as supervision.
Methods
SimTxtSeg comprises two core components:
- Textual-to-Visual Cue Converter (TVCC): Inspired by GroundingDINO, this module takes an image and its corresponding text description as input. It uses separate image and text backbones (Swin-T and BERT-BASE, respectively) to extract features, fuses them, and then employs a language-guided query selection and a cross-modality decoder to refine bounding box predictions. This converter is pre-trained on medical datasets to accurately localize regions of interest based on text.
- Text-Vision Hybrid Attention (TVHA) Decoder: This module is integrated into a segmentation model (specifically, a UNet-like decoder). It takes the visual features from an image encoder and the text embeddings from the TVCC. The TVHA decoder features a dual-way cross-modal attention mechanism that allows for bidirectional interaction between text and visual features. It also incorporates channel attention to highlight relevant features. This allows the model to directly infuse semantic information from text into the segmentation process, going beyond just using text to generate pseudo-labels.
The overall pipeline first uses the TVCC to generate bounding box prompts from text, which are then fed into the Segment Anything Model (SAM) to produce pseudo-masks. These pseudo-masks, along with the original images and text cues, are used to train the final segmentation model equipped with the TVHA decoder.
Datasets
The paper evaluates SimTxtSeg on two medical image segmentation tasks:
- Colonic Polyp Segmentation: A combined dataset from CVC-ClinicDB, CVC-ColonDB, ETIS-LaribPolypDB, Kvasir, and PolypGen, totaling 3,784 images.
- MRI Brain Tumor Segmentation: The LGG Segmentation Dataset from The Cancer Imaging Archive, comprising 3,929 brain MRI images.
For pre-training the TVCC, the authors mention using a domain-specific medical dataset in the ODVG format, which includes bounding boxes and corresponding phrases.
Key Insights and Ideas
- Text as a Powerful Weak Supervision Signal: The paper demonstrates that simple text cues, even without explicit spatial information, can be effectively used to generate high-quality pseudo-labels for medical image segmentation, significantly reducing annotation effort.
- Bridging Modalities with Hybrid Attention: The Text-Vision Hybrid Attention mechanism is crucial for infusing semantic understanding from text directly into the visual segmentation task, leading to improved performance beyond just using text for pseudo-mask generation.
- SAM's Adaptability and Limitations: While SAM is a powerful general-purpose segmentation model, its performance in medical imaging is improved by fine-tuning (SAM-med2d). However, the paper shows that even the general SAM-huge can be effectively utilized with text-generated prompts.
- Sentence Cues Outperform Class Names: Using descriptive sentences as text prompts for the TVCC yields better results than using simple class names, suggesting that richer textual context aids in more accurate localization.
Key Results
- State-of-the-Art Performance: SimTxtSeg achieves state-of-the-art results on both colonic polyp and brain tumor segmentation tasks, outperforming existing weakly-supervised methods.
- Polyp Segmentation Improvement: On the polyp dataset, SimTxtSeg achieved a +1.38% improvement in mDice and a +3.36% improvement in mIoU compared to other SOTA weakly-supervised models.
- Brain Tumor Segmentation Improvement: On the brain tumor dataset, the method achieved a +4.1% improvement in mDice and a +3.94% improvement in mIoU.
- Pseudo-Label Quality: The pseudo-labels generated by the TVCC+SAM combination are shown to be competitive with, and sometimes even better than, those from SOTA fully-supervised models.
- Ablation Study Findings: The TVHA module significantly contributes to performance, with both the dual-way cross-modal attention and channel attention playing important roles. Using both modules together yields the best results.
Limitations and Future Work
- End-to-End Integration: The authors suggest that fusing the TVCC and SAM into a more integrated, end-to-end fashion could lead to further improvements.
- Broader Applications: They plan to extend their method to other medical image analysis areas.
- Prompt Granularity: While sentences were found to be effective, further exploration of optimal text prompt granularity could be beneficial.
- SAM-med2d Bias: The paper notes that SAM-med2d, while pre-trained on medical data, exhibits a bias towards polyp images and has poor generalizability for other tasks.
Summary updated