Frequency-Domain Refinement of Vision Transformers for Robust Medical Image Segmentation Under Degradation

Sanaz Karimijafarbigloo, Sina Ghorbani Kolahi, Reza Azad, Ulas Bagci, Dorit MerhofWACV 2025 · DOI

Medical image segmentation—the process of identifying organs or lesions in scans—is a high-stakes task. While Convolutional Neural Networks (CNNs) have been the workhorse of the field, Vision Transformers (ViTs) recently took the lead by capturing long-range dependencies. However, ViTs have a known weakness: they often struggle with fine-grained local details and are notoriously sensitive to image degradation like noise or blur.

A new approach, the Enhanced Wave Vision Transformer (EW-ViT), moves the heavy lifting into the frequency domain to ensure accurate segmentation even when image quality is poor. By combining wavelet decomposition with learnable "prompts" and contrastive learning, EW-ViT achieves state-of-the-art performance and remarkable robustness against image degradation.

The Challenge of High-Frequency Noise

In clinical settings, images are rarely perfect. Scans can be noisy, blurry, or suffer from artifacts. Standard Transformers act like a "low-pass filter," meaning they are excellent at capturing the overall shape of an organ but often "smooth over" the sharp edges and fine textures (high-frequency components) essential for precise boundaries. Existing hybrid models often attempt to fix this by appending CNN layers, but this leads to bulky models that still struggle to distinguish between high-frequency anatomical details and high-frequency noise.

Proposed Method: EW-ViT

The EW-ViT architecture introduces three major innovations to address these limitations:

  1. Wavelet-Based Self-Attention: Instead of standard downsampling, the model uses the Discrete Wavelet Transform (DWT). This splits the image into low-frequency (basic structure) and high-frequency (edges/details) subbands without losing information.
  2. Prompt-Guided High-Frequency Refiner (PGHFR): This is the core "cleaning" mechanism. It uses learnable Component Prompt Generation (CPG) to adaptively predict weights for frequency components and Implicit Prompt Interaction (IPI) to recalibrate features, sharpening boundaries while mitigating noise.
  3. Contrastive Learning Strategy: To ensure robustness, the model employs a contrastive head. During training, it creates augmented, noisy versions of features and forces the model to ensure the representation of a pixel in a noisy image remains close to its representation in a clean image.
The EW-ViT architecture includes an encoder-decoder structure integrated with a contrastive head for feature refinement.

Experimental Results and Robustness

The model was evaluated on three diverse medical benchmarks: the Synapse abdominal CT dataset, the ISIC 2018 skin lesion dataset, and the ACDC cardiac MRI dataset. EW-ViT consistently outperformed previous state-of-the-art (SOTA) models like TransUNet, Swin-UNet, and HiFormer.

  • Synapse: Achieved a Dice Similarity Coefficient (DSC) of 83.51%, a significant improvement over HiFormer (80.39%).
  • ISIC 2018: Reached a DSC of 0.9164, outperforming the previous best, Laplacian-Former.
  • Efficiency: Despite its performance, EW-ViT maintains a competitive parameter count (33.35M) and lower FLOPs (6.03G) compared to many hybrid architectures.

One of the most significant findings was the model's resilience to noise. In tests with Gaussian noise (up to $\sigma = 0.2$), EW-ViT maintained significantly higher accuracy than its competitors. While other models' performance plummeted as noise increased, EW-ViT's frequency-domain refinement allowed it to preserve anatomical structures.

Quantitative comparison of DSC (%) under varying levels of Gaussian noise, showing EW-ViT's superior robustness compared to Swin-Unet and MT-Unet.

Conclusion

EW-ViT demonstrates that the frequency domain is a powerful tool for medical imaging. By treating "structure" and "detail" differently, the model can target refinement where noise is most prevalent. While the model shows high sensitivity, future work may focus on refining the decoder stages to improve specificity and exploring how this frequency-domain approach scales to larger, multi-modal foundation models.

Summary updated

BibTeX
@inproceedings{karimijafarbigloo2025frequency,
  author = {Sanaz Karimijafarbigloo and Sina Ghorbani Kolahi and Reza Azad and Ulas Bagci and Dorit Merhof},
  title = {{Frequency-Domain Refinement of Vision Transformers for Robust Medical Image Segmentation Under Degradation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00889},
}
Figure