A Multimodal Approach Combining Structural and Cross-domain Textual Guidance for Weakly Supervised OCT Segmentation

Jiaqi Yang, Nitish Mehta, Xiaoling Hu, Chao Chen, Chia-Ling TsaiarXiv 2024 · arXiv · OpenReview

Abstract

Accurate segmentation of Optical Coherence Tomography (OCT) images is crucial for diagnosing and monitoring retinal diseases. However, the labor-intensive nature of pixel-level annotation limits the scalability of supervised learning with large datasets. Weakly Supervised Semantic Segmentation (WSSS) provides a promising alternative by leveraging image-level labels. In this study, we propose a novel WSSS approach that integrates structural guidance with text-driven strategies to generate high-quality pseudo labels, significantly improving segmentation performance. By fusing visual and textual components within a multimodal framework, our approach enhances lesion localization accuracy. Experimental results on three OCT datasets demonstrate that our method achieves state-of-the-art performance, highlighting its potential to improve diagnostic accuracy and efficiency in medical imaging.

Introduction

Optical Coherence Tomography (OCT) is the gold standard for imaging the retina, but identifying lesions such as fluid or detachments usually requires experts to manually trace boundaries—a process that is slow, expensive, and prone to human error. While "Weakly Supervised Semantic Segmentation" (WSSS) allows models to learn from simple "yes/no" image labels, the resulting maps are often coarse and lack the precision needed for medical diagnosis. This paper tackles the challenge of making these "weak" labels "stronger" by feeding the model extra information: the anatomical structure of the eye and descriptive text.

Methodology

The proposed multimodal framework processes four distinct streams of information to generate high-quality Class Activation Maps (CAMs), which are then converted into pseudo-labels for training:

  1. Dual Visual Modules: A primary module processes the raw OCT image using a Mix Transformer (MiT) backbone, while a structural module identifies retinal layers. This provides the "where" context—lesions usually appear in specific layers, preventing the AI from "hallucinating" lesions in impossible locations.
  2. Dual Textual Modules: The model uses CLIP to encode medical descriptions and BLIP (a captioning model) to describe the OCT image as if it were a natural scene. This "childhood metaphor" approach uses general spatial and relational cues (e.g., "the object in the middle") to help the model group similar-looking regions even without specialized medical terminology.
  3. Cross-Attention Fusion: A transformer-based mechanism allows the visual and structural modules to communicate, ensuring detected lesions align with the actual anatomy of the eye.

Results and Key Insights

The proposed method outperformed existing weakly supervised methods across all tested datasets. On the RESC dataset, it achieved a mean Intersection over Union (mIoU) of 61.15%, significantly beating the previous state-of-the-art of 54.87%. On the Duke dataset, it reached 68.11% mIoU.

Visual comparison of the segmentation process: (a) original OCT, (b) ground truth, (c) baseline CAM, (d) proposed CAM heatmap, and (e) final generated pseudo-label.

Ablation studies confirmed that peak performance is only achieved when visual, structural, and textual cues are combined. The textual cues refine the "what," while the structural cues refine the "where."

Conclusion and Future Work

By integrating structural anchors and cross-domain textual guidance, this framework bridges the gap between simple image-level labels and precise pixel-level segmentation. While currently optimized for retinal anatomy, the success of the text-driven approach suggests a flexible path forward for other medical imaging modalities. Future work may involve integrating actual medical reports to further bridge the gap between general AI and specialized clinical reasoning.

Summary updated

Abstract

Accurate segmentation of Optical Coherence Tomography (OCT) images is crucial for diagnosing and monitoring retinal diseases. However, the labor-intensive nature of pixel-level annotation limits the scalability of supervised learning with large datasets. Weakly Supervised Semantic Segmentation (WSSS) provides a promising alternative by leveraging image-level labels. In this study, we propose a novel WSSS approach that integrates structural guidance with text-driven strategies to generate high-quality pseudo labels, significantly improving segmentation performance. In terms of visual information, our method employs two processing modules that exchange raw image features and structural features from OCT images, guiding the model to identify where lesions are likely to occur. In terms of textual information, we utilize large-scale pretrained models from cross-domain sources to implement label-informed textual guidance and synthetic descriptive integration with two textual processing modules that combine local semantic features with consistent synthetic descriptions. By fusing these visual and textual components within a multimodal framework, our approach enhances lesion localization accuracy. Experimental results on three OCT datasets demonstrate that our method achieves state-of-the-art performance, highlighting its potential to improve diagnostic accuracy and efficiency in medical imaging.
BibTeX
@misc{yang2024multimodal,
  author = {Jiaqi Yang and Nitish Mehta and Xiaoling Hu and Chao Chen and Chia-Ling Tsai},
  title = {{A Multimodal Approach Combining Structural and Cross-domain Textual Guidance for Weakly Supervised OCT Segmentation}},
  year = {2024},
  eprint = {2411.12615},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.12615},
}
Figure