Abstract
Medical image segmentation is essential for clinical applications such as disease diagnosis, treatment planning, and disease development monitoring because it provides precise morphological and spatial information on anatomical structures that directly influence treatment decisions. Convolutional neural networks significantly impact image segmentation; however, since convolution operations are local, capturing global contextual information and long-range dependencies is still challenging. Their capacity to precisely segment structures with complicated borders and a variety of sizes is impacted by this restriction. Since transformers use self-attention methods to capture global context and long-range dependencies efficiently, integrating transformer-based architecture with CNNs is a feasible approach to overcoming these challenges. To address these challenges, we propose the Focal Modulation and Bidirectional Feature Fusion Network for Medical Image Segmentation, referred to as FM-BFF-Net. The network combines convolutional and transformer components, employs a focal modulation attention mechanism to refine context awareness, and introduces a bidirectional feature fusion module that enables efficient interaction between encoder and decoder representations across scales. Extensive experiments on eight publicly available datasets show that FM-BFF-Net consistently surpasses recent state-of-the-art methods in Jaccard index and Dice coefficient.
Introduction
Accurate segmentation of anatomical structures and lesions is a cornerstone of modern medical imaging. While Convolutional Neural Networks (CNNs) have set high standards in this field, they often struggle with "nearsightedness" due to the local nature of convolution operations. This limitation makes it difficult to capture long-range spatial dependencies, which are crucial when dealing with lesions of varying sizes or those with irregular, low-contrast boundaries. Conversely, Vision Transformers (ViTs) excel at capturing global context but can be computationally intensive and may lose fine-grained local details. FM-BFF-Net addresses this trade-off by proposing a hybrid architecture that leverages the strengths of both paradigms.
Proposed Architecture
The FM-BFF-Net architecture is designed to harmonize local feature extraction with global contextual understanding through a sophisticated hybrid approach.
The network utilizes EfficientNetV2S as its primary encoder to maximize computational efficiency while maintaining high-quality feature extraction. Key innovations include:
- Focal Modulation-based Convformer Attention Block (FMCAB): Integrated into skip connections, this block uses focal modulation to adaptively refine features, ensuring the model focuses on salient regions while maintaining broad context awareness.
- Bidirectional Feature Fusion Module (BiFFM): Unlike traditional U-Net skip connections that simply concatenate features, BiFFM facilitates a bidirectional flow. It allows decoder features to interact back with encoder features, harmonizing semantic global information with precise local boundaries.
- Vision Transformer Module (ViTM): Positioned at the bottleneck, this module employs Global Self-Attention (GSA) and Token-based Self-Attention (TSA) to capture extensive dependencies before the decoding phase begins.
- Feature Refinement Modules (FRMs): These are used in the decoder to progressively reconstruct the image across multiple scales, ensuring boundary precision.
Experimental Results
The model was rigorously evaluated across eight datasets spanning polyp segmentation (Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB), skin lesion segmentation (ISIC 2016-2018), and ultrasound imaging (BUSI, DDTI).
FM-BFF-Net demonstrated superior performance compared to state-of-the-art (SOTA) models like Swin-Unet, TransUNet, and Duck-Net. For instance, on the Kvasir-SEG dataset, it achieved a Jaccard index of 92.96% and a Dice coefficient of 95.14%. In ultrasound imaging (DDTI), it showed a Jaccard index improvement of up to 11.67% over existing methods.
Qualitative analysis (as seen in Figure 5) reveals that FM-BFF-Net produces cleaner, more accurate segmentation masks, particularly in challenging scenarios with low contrast or multiple small lesions where other models often produce fragmented or noisy results.
Conclusion
FM-BFF-Net represents a significant advancement in medical image segmentation by effectively combining focal modulation with bidirectional feature fusion. By bridging the gap between local and global feature representation, the model achieves robust performance across diverse imaging modalities and lesion types. Future work will focus on extending this architecture to 3D volumetric data (e.g., CT or MRI scans) and integrating uncertainty quantification to further assist clinical decision-making.
Summary updated