Abstract
Medical image segmentation is critical for AI-assisted diagnosis but suffers from high annotation costs and suboptimal multimodal fusion. We present a novel framework that leverages medical text annotations to enhance image segmentation. By modeling features as distributions to reduce information redundancy and incorporating frequency-domain "prompts" to sharpen boundaries, our method achieves state-of-the-art performance on multiple benchmarks while maintaining computational efficiency.
Introduction
Accurate segmentation of organs and lesions is a cornerstone of medical imaging. However, expert pixel-level labeling is expensive and time-consuming. While multimodal approaches incorporating text descriptions offer a solution, they often struggle with information redundancy or ignore fine-grained modality-specific details. Traditional fusion methods like feature addition or contrastive learning often fail to capture the optimal joint representation, leading to noise or loss of unique modality-specific information.
Methodology
The proposed framework consists of three primary components designed to optimize multimodal representation and boundary precision.
Multimodal Feature Distribution Learning
Unlike simple element-wise addition, our approach treats text and vision features as probability distributions ($p$ and $q$). We introduce "learnable features" that act as a bridge in the joint feature space. By minimizing the Kullback-Leibler (KL) divergence between these distributions, the model effectively reduces redundancy while preserving unique, task-relevant information from both modalities. This process "squeezes out" noise, ensuring that the fused representation is both concise and informative.
Frequency Prompt Encoder
To address the "low-frequency bias" common in deep learning models—which often leads to blurry boundaries—we utilize the frequency domain. Using a Fast Fourier Transform (FFT), we extract the phase spectrum of the image. This phase information, which captures structural "blueprints" and edges, is processed through a Prompt Encoder and fed into the decoder as a spatial prompt to help the model identify sharp boundaries more accurately.
System Architecture
The architecture utilizes a frozen CLIP text encoder and a vision encoder. The fused distribution features and frequency-domain prompts are integrated within a cross-modality processor and a U-shaped decoder to produce the final segmentation mask. This design allows the model to leverage pre-trained semantic knowledge while refining spatial details.
Experimental Results
The framework was evaluated on three diverse datasets: MoNuSeg (nuclear segmentation), GlaS (gland segmentation), and MosMedData+ (COVID-19 lung infections).
- Performance: On MoNuSeg, the model achieved an 80.96% Dice score, outperforming the previous SOTA (LViT) by 0.81%. On GlaS, it reached 91.08%, significantly surpassing other multimodal methods like LAVT and TGANet.
- Efficiency: The model is remarkably lightweight, using only 29.2M parameters on MoNuSeg, compared to 105M for TransUNet and 91M for SAM, while delivering superior accuracy.
Conclusion
By shifting from simple feature fusion to distribution learning and leveraging frequency-domain structural information, this framework effectively addresses the challenges of redundancy and boundary precision in medical segmentation. The results demonstrate that even simple text descriptions can significantly bolster model performance, especially in data-scarce scenarios, providing a robust path forward for multimodal medical AI.
Summary updated