FuseNet: Self-Supervised Dual-Path Network for Medical Image Segmentation

Amirhossein Kazerouni, Sanaz Karimijafarbigloo, Reza Azad, Yury Velichko, Ulas Bagci, Dorit Merhof

Abstract

Semantic segmentation, a crucial task in computer vision, often relies on labor-intensive and costly annotated datasets for training. In response to this challenge, we introduce FuseNet, a dual-stream framework for self-supervised semantic segmentation that eliminates the need for manual annotation. FuseNet leverages the shared semantic dependencies between the original and augmented images to create a clustering space, effectively assigning pixels to semantically related clusters, and ultimately generating the segmentation map. Additionally, FuseNet incorporates a cross-modal fusion technique that extends the principles of CLIP by replacing textual data with augmented images. This approach enables the model to learn complex visual representations, enhancing robustness against variations similar to CLIP's text invariance. To further improve edge alignment and spatial consistency between neighboring pixels, we introduce an edge refinement loss. This loss function considers edge information to enhance spatial coherence, facilitating the grouping of nearby pixels with similar visual features. Extensive experiments on skin lesion and lung segmentation datasets demonstrate the effectiveness of our method.

Introduction

Deep learning has revolutionized medical imaging, but its primary bottleneck remains the requirement for massive, pixel-perfect labeled datasets. In the medical field, generating these labels is exceptionally expensive and time-consuming, as it requires the expertise of trained specialists. While self-supervised learning (SSL) has emerged as a solution—learning from the data's inherent structure rather than human labels—existing methods often struggle to balance global context with local boundary details and maintain spatial consistency across neighboring pixels.

Methodology

FuseNet addresses these challenges through a dual-stream network that processes an original image and its augmented version (e.g., via ColorJitter and Gaussian blur) simultaneously.

The FuseNet architecture featuring dual-stream processing, CLIP-inspired cross-modal fusion, and edge refinement loss.

Key Technical Components

  • Dual-Path Architecture: Both images pass through a shared-weight encoder and projection head. This forces the model to learn features that are invariant to noise and lighting but consistent in anatomical structure.
  • Cross-Modal Fusion (CLIP-inspired): Taking inspiration from OpenAI’s CLIP, which aligns images with text, FuseNet aligns the features of the original image with its augmented counterpart. This "image-to-augmented-image" contrastive learning helps the model grasp complex visual representations without needing textual descriptions.
  • Cross-Attention Module: This module facilitates information exchange between the two pathways, allowing the model to attend to shared attributes and differences between the views.
  • Edge Refinement Loss ($L_{Boundary}$): To address blurry boundaries, the authors introduced a loss function that compares the segmentation map to a version of itself that has been downsampled and then upsampled. This discrepancy highlights where the model is losing edge detail, providing a direct signal to improve spatial coherence.

Experimental Setup and Results

The model was evaluated on two distinct medical imaging benchmarks: the PH2 Dataset (200 RGB dermoscopic images of skin lesions) and the Lung Analysis Dataset (2D and 3D CT scans).

FuseNet outperformed several state-of-the-art (SOTA) self-supervised methods, including MS-Former and SGSCN. On the PH2 dataset, it achieved a Dice Similarity Coefficient (DSC) of 88.7%, a significant improvement over the previous SOTA of 86.0%. For lung segmentation, it reached a DSC of 95.3%, approaching the performance of supervised models.

Qualitative comparison of FuseNet against other self-supervised methods and ablation results showing the impact of different loss components.

Ablation Insights

Ablation studies confirmed the importance of each component. Removing the CLIP-based fusion resulted in a 0.9% drop in DSC, while removing the edge refinement loss caused a 0.8% drop. These results underscore that both cross-modal alignment and boundary-specific supervision are vital for high-fidelity medical segmentation.

Conclusion

FuseNet demonstrates that the contrastive principles of CLIP can be successfully repurposed for pure vision tasks in specialized domains like medicine. By combining dual-path consistency with an elegant "down-up" edge refinement trick, the framework achieves superior spatial consistency and boundary alignment without manual labels. Future work will focus on automating cluster-to-class mapping and extending the architecture to multi-organ segmentation tasks.

Summary updated

Abstract

Semantic segmentation, a crucial task in computer vision, often relies on labor-intensive and costly annotated datasets for training. In response to this challenge, we introduce FuseNet, a dual-stream framework for self-supervised semantic segmentation that eliminates the need for manual annotation. FuseNet leverages the shared semantic dependencies between the original and augmented images to create a clustering space, effectively assigning pixels to semantically related clusters, and ultimately generating the segmentation map. Additionally, FuseNet incorporates a cross-modal fusion technique that extends the principles of CLIP by replacing textual data with augmented images. This approach enables the model to learn complex visual representations, enhancing robustness against variations similar to CLIP's text invariance. To further improve edge alignment and spatial consistency between neighboring pixels, we introduce an edge refinement loss. This loss function considers edge information to enhance spatial coherence, facilitating the grouping of nearby pixels with similar visual features. Extensive experiments on skin lesion and lung segmentation datasets demonstrate the effectiveness of our method. \href{https://github.com/xmindflow/FuseNet}{Codebase.}
BibTeX
@inproceedings{kazerouni2024fusenet,
  author = {Amirhossein Kazerouni and Sanaz Karimijafarbigloo and Reza Azad and Yury Velichko and Ulas Bagci and Dorit Merhof},
  title = {{FuseNet: Self-Supervised Dual-Path Network for Medical Image Segmentation}},
  booktitle = {ISBI},
  year = {2024},
  url = {https://openreview.net/forum?id=yxaXyUecOn},
}
Figure