Multimodal Representation Distribution Learning for Medical Image Segmentation

Chao Huang, Weichao Cai, Qiuping Jiang, Zhihua WangIJCAI 2024

Abstract

Medical image segmentation is a critical component of clinical diagnosis, yet it is often hindered by the scarcity of high-quality, pixel-level annotations. While multimodal approaches leveraging medical text have shown promise, they frequently suffer from feature redundancy and information loss. This paper presents a novel framework that models text and vision features as probability distributions rather than static vectors. By incorporating a frequency-domain "prompt encoder" to preserve high-frequency boundary details, the proposed method achieves state-of-the-art performance across multiple medical benchmarks, including nuclei and lung infection segmentation.

Introduction

Accurate segmentation of biological structures and lesions is foundational to modern medical imaging. However, the manual creation of ground-truth masks is labor-intensive and requires expert knowledge. To mitigate this, researchers have turned to multimodal learning, using descriptive medical text to guide vision models.

Existing fusion strategies, such as simple feature addition or contrastive learning (e.g., CLIP), face significant challenges. Feature addition often introduces task-irrelevant noise (redundancy), while contrastive learning tends to focus solely on shared information, discarding modality-specific details essential for precise boundary detection. This work addresses these limitations by treating multimodal features as distributions within a joint space, allowing for a more nuanced balance between shared and unique information.

Conceptual comparison of feature fusion methods: (a) Feature Addition, (b) Contrastive Learning, and (c) the proposed Distribution Learning approach.

Methodology

Multimodal Representation Distribution Learning

The core innovation lies in moving from "point-based" features to a probabilistic representation. The model treats text features ($p$) and vision features ($q$) as Gaussian distributions. By minimizing the Kullback-Leibler (KL) divergence between these distributions, the framework aligns the modalities while utilizing a set of learnable features to preserve modality-specific information that might otherwise be lost during fusion.

Frequency Prompt Encoder ($Enc_P$)

To address the common issue of blurry boundaries in deep learning models, the authors introduce a Frequency Prompt Encoder. By applying a Fast Fourier Transform (FFT) to the input images, the model extracts the phase spectrum. This spectrum contains critical high-frequency information regarding edges and shapes, acting as a spatial prompt for the segmentation decoder.

The overall architecture of the proposed framework, detailing the Multi-Modal Feature Fusion and the Frequency Representation modules.

Architecture

The framework adopts a U-Net-style backbone enhanced with a cross-modality processor. This processor integrates the fused distribution features with the frequency-domain prompts, ensuring that the decoder has access to both high-level semantic guidance from text and low-level structural details from the image phase.

Experimental Results

Datasets and Benchmarks

The model was evaluated on three diverse medical imaging datasets:

  • MoNuSeg: Nuclei segmentation in microscopic tissue images.
  • GlaS: Gland segmentation in colon histology.
  • MosMedData+: COVID-19 related lung infection segmentation in CT slices.

Performance Analysis

The proposed method consistently outperformed both single-modal and multimodal state-of-the-art (SOTA) models:

  • MoNuSeg: Achieved a 80.96% Dice score, a 0.81% improvement over the previous best multimodal model (LViT).
  • GlaS: Reached a 91.08% Dice score with lower computational complexity (FLOPs) than many transformer-based alternatives.
  • MosMedData+: Demonstrated robustness on 3D-derived data with a 1.45% improvement in Dice score over the second-best method.

In addition to accuracy, the model maintains efficiency with approximately 29M parameters and 57 GFLOPs, making it more practical for clinical deployment than larger foundation models like MedSAM.

Discussion and Conclusion

By representing multimodal features as distributions, this framework effectively reduces redundancy while retaining the modality-specific details necessary for high-precision segmentation. The inclusion of the frequency-domain phase spectrum further ensures that the model can accurately delineate complex boundaries. While the model currently relies on the availability of medical text, its ability to act as a regularizer for vision features suggests that distribution learning is a powerful tool for maximizing the utility of limited medical data. Future work will explore the scalability of this approach within larger foundation models.

Summary updated

BibTeX
@inproceedings{huang2024multimodal,
  author = {Chao Huang and Weichao Cai and Qiuping Jiang and Zhihua Wang},
  title = {{Multimodal Representation Distribution Learning for Medical Image Segmentation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure