A multi-modal multi-branch framework for retinal vessel segmentation using ultra-widefield fundus photographs

Qihang Xie, Xuefei Li, Yuanyuan Li, Jiayi Lu, Shaodong Ma, Yitian Zhao, Jiong ZhangFrontiers in Cell and Developmental Biology 2025 · DOI

TL;DR

Researchers have developed M3B-Net, a multi-modal deep learning framework that significantly improves retinal vessel segmentation in ultra-widefield (UWF) fundus images. By using style transfer to "borrow" high-contrast features from invasive angiography (FFA) and employing a multi-scale fusion strategy, the model outperforms current state-of-the-art methods in detecting fine vascular structures.

What Problem Does This Paper Address?

Retinal vessel segmentation is a foundational task in ophthalmology, essential for diagnosing diseases like diabetic retinopathy. While traditional color fundus photos cover only about 45 degrees of the eye, Ultra-Widefield (UWF) imaging captures a panoramic 200-degree view.

However, UWF images present two major hurdles for AI:

  1. Low Contrast: Vessels often blend into complex backgrounds, making fine capillaries nearly invisible to standard algorithms.
  2. High Resolution vs. Context: UWF images are massive (e.g., 4000x4000 pixels). To process them, models usually "cut" the image into small patches. This causes a loss of global context, leading to fragmented vessel segments in the final output.

While Fundus Fluorescence Angiography (FFA) is the "gold standard" for contrast, it requires injecting dye into the patient—an invasive procedure with potential side effects. The authors wanted to get FFA-level segmentation quality using only non-invasive UWF photographs.

Methods

The authors propose M3B-Net, a multi-modal, multi-branch framework. Its architecture is built on three novel pillars:

  • Style Transfer Branch: The model uses a CycleGAN-based architecture to transform UWF images into a latent FFA feature space. This allows the segmentation network to leverage high-contrast vascular features without the patient ever needing an actual FFA injection.
  • Selective Fusion Module (SFM): Not all generated FFA features are useful. The SFM calculates the Euclidean distance between UWF and FFA feature maps, selectively integrating only the most relevant high-contrast information into the segmentation path.
  • Local Perception Fusion Module (LPFM): To solve the "patch-processing" context problem, the model extracts three concentric patches (Small, Middle, Large) around the same center point. These are processed through parallel ResNet encoders, ensuring the model sees both the fine details and the surrounding neighborhood simultaneously.
  • Attention-Guided Upsampling Module (AUM): During the decoding (upsampling) phase, this module uses attention mechanisms to prevent the loss of fine vessel details that often occur during standard convolution operations.

Datasets

The study validated the framework on two datasets:

  1. Private UWF-SEG Dataset: Collected from Ningbo Yinzhou People’s Hospital, containing 65 high-resolution UWF images (3900 × 3072) with expert pixel-level annotations and paired FFA images for training the style transfer.
  2. PRIME-FP20: A publicly available dataset of 15 high-resolution UWF images used to test the model’s generalizability.

Key Insights and Ideas

  • Cross-Modality Feature Extraction: The "aha moment" here is that we don't need the actual second modality (FFA) at inference time. By training a style transfer branch, the model learns the characteristics of what makes a vessel visible in FFA and applies that "knowledge" to segmenting standard UWF photos.
  • Context via Multi-Scale Patches: Instead of just looking at a 256x256 window, the LPFM forces the model to look at a 768x768 area (downsampled) alongside the 256x256 window. This provides the "big picture" needed to keep long vessels continuous.
  • Similarity-Based Filtering: By using a similarity threshold (0.5) in the SFM, the model avoids being "distracted" by artifacts or noise generated during the style transfer process, ensuring only high-fidelity vascular features are fused.

Key Results

M3B-Net consistently outperformed established models like U-Net, TransUnet, and ConvUNext across all metrics:

  • UWF-SEG (Private): Achieved a Dice score of 78.54% and a Balanced Accuracy (BACC) of 90.27%. This represents a ~5.19% improvement in Dice score over the high-performing ConvUNext model.
  • PRIME-FP20 (Public): Achieved a Dice score of 83.02%, outperforming the previous state-of-the-art by over 1%.
  • Qualitative Success: Visual results show that M3B-Net is significantly better at connecting "broken" vessels and identifying tiny terminal capillaries that other models miss or mistake for noise.

Limitations and Future Work

The authors acknowledge that while the multi-branch framework is highly accurate, it is computationally expensive. Running multiple encoders for different patch sizes and a style transfer branch requires significant GPU resources.

Future Work: The team aims to optimize the model for better computational efficiency to make it more viable for real-time clinical use. They also plan to explore how this enhanced segmentation can directly improve the automated diagnosis of specific conditions like retinal vein occlusion.

Summary updated

Abstract

Extensive experimental evaluations demonstrate that our approach significantly outperforms existing state-of-the-art methods for UWF fundus image segmentation.
BibTeX
@inproceedings{xie2025multi,
  author = {Qihang Xie and Xuefei Li and Yuanyuan Li and Jiayi Lu and Shaodong Ma and Yitian Zhao and Jiong Zhang},
  title = {{A multi-modal multi-branch framework for retinal vessel segmentation using ultra-widefield fundus photographs}},
  booktitle = {Frontiers in Cell and Developmental Biology},
  year = {2025},
  doi = {10.3389/fcell.2024.1532228},
}
Figure