Y-Net: A Spatiospectral Dual-Encoder Networkfor Medical Image Segmentation

Azade Farshad, Yousef Yeganeh, Peter Gehlbach, Nassir Navab

Abstract

Automated segmentation of retinal optical coherence tomography (OCT) images has become an important recent direction in machine learning for medical applications. We hypothesize that the anatomic structure of layers and their high-frequency variation in OCT images make retinal OCT a fitting choice for extracting spectral-domain features and combining them with spatial domain features. In this work, we present $Υ$-Net, an architecture that combines the frequency domain features with the image domain to improve the segmentation performance of OCT images. The results of this work demonstrate that the introduction of two branches, one for spectral and one for spatial domain features, brings a very significant improvement in fluid segmentation performance and allows outperformance as compared to the well-known U-Net model. Our improvement was 13% on the fluid segmentation dice score and 1.9% on the average dice score. Finally, removing selected frequency ranges in the spectral domain demonstrates the impact of these features on the fluid segmentation outperformance.

Introduction

Optical Coherence Tomography (OCT) is the gold standard for diagnosing retinal diseases such as age-related macular degeneration (AMD). A critical component of diagnosis is the precise segmentation of intraretinal fluid pockets and retinal layers. However, standard deep learning models like the U-Net often struggle in this domain due to two primary factors:

  1. Speckle Noise: OCT images are inherently prone to high-frequency noise that can interfere with traditional spatial convolutional filters.
  2. Global Context Requirements: While standard convolutions excel at local feature extraction, retinal structures and fluid pockets often possess global spectral signatures that local filters fail to capture.

Y-Net addresses these challenges by processing images in both the spatial and frequency domains simultaneously, allowing the model to disentangle meaningful features from noise and capture broader anatomical context.

Methods

The Y-Net architecture is an end-to-end autoencoder characterized by a dual-encoder structure that fuses spatial and spectral information.

The Y-Net architecture featuring dual spatial and spectral encoders.

Dual-Encoder Architecture

  • Spatial Encoder: A series of standard convolutional blocks (Conv + BN + ReLU + MaxPool) that extract local spatial features, similar to the contracting path of a U-Net.
  • Spectral Encoder: The core innovation of Y-Net. It utilizes Fast Fourier Convolutional (FFC) blocks. Within these blocks, a Fourier Unit performs a Fast Fourier Transform (FFT) on the feature maps, applies convolutions in the frequency domain (treating real and imaginary components as channels), and then uses an Inverse FFT (IFFT) to return the data to the spatial domain.
  • Spectral Norm: This component splits feature channels into "local" and "global" portions based on a hyperparameter $\alpha$. This allows the model to dynamically balance traditional spatial learning with spectral learning.

Fusion and Decoding

Features from both encoders are concatenated and passed through a spatial decoder. Skip connections from the spatial encoder help preserve high-resolution details during the reconstruction of the final segmentation map.

Key Results

Y-Net was evaluated on the Duke OCT dataset, demonstrating superior performance over established models like U-Net, RelayNet, and specialized language-based segmentors.

Qualitative comparison of segmentation results between U-Net and Y-Net against expert ground truth.

Performance Gains

  • Fluid Segmentation: Y-Net achieved a Dice score of 0.93, representing a 13% improvement over the standard U-Net (0.80).
  • Overall Accuracy: The mean Dice score across all retinal layers and fluid was 0.855, outperforming U-Net (0.836).
  • Efficiency: Despite the dual-encoder design, Y-Net is more parameter-efficient than U-Net. By leveraging 1x1 convolutions in the frequency domain, Y-Net utilizes 7.46M parameters compared to U-Net’s 7.76M.

Ablation Insights

Ablation studies revealed that the spectral features are particularly critical for fluid detection. Manually removing specific frequency ranges in the Fourier Unit caused fluid segmentation performance to drop significantly, while layer segmentation remained relatively stable. Furthermore, the optimal balance of features was found at $\alpha = 0.25$ or $0.5$, confirming that a hybrid spatiospectral approach is superior to either domain alone.

Conclusion

Y-Net demonstrates that incorporating frequency-domain information significantly enhances the segmentation of noisy medical images like retinal OCT. The spectral encoder's ability to capture global context and filter speckle noise leads to a substantial leap in fluid segmentation accuracy. Future research will explore the utility of this spatiospectral approach in other medical imaging modalities, such as Ultrasound and MRI, where noise and global structure are similarly prevalent.

Summary updated

Abstract

Automated segmentation of retinal optical coherence tomography (OCT) images has become an important recent direction in machine learning for medical applications. We hypothesize that the anatomic structure of layers and their high-frequency variation in OCT images make retinal OCT a fitting choice for extracting spectral-domain features and combining them with spatial domain features. In this work, we present $Υ$-Net, an architecture that combines the frequency domain features with the image domain to improve the segmentation performance of OCT images. The results of this work demonstrate that the introduction of two branches, one for spectral and one for spatial domain features, brings a very significant improvement in fluid segmentation performance and allows outperformance as compared to the well-known U-Net model. Our improvement was 13% on the fluid segmentation dice score and 1.9% on the average dice score. Finally, removing selected frequency ranges in the spectral domain demonstrates the impact of these features on the fluid segmentation outperformance.
BibTeX
@inproceedings{farshad2022y,
  author = {Azade Farshad and Yousef Yeganeh and Peter Gehlbach and Nassir Navab},
  title = {{Y-Net: A Spatiospectral Dual-Encoder Networkfor Medical Image Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2022},
  doi = {10.1007/978-3-031-16434-7\_56},
}
Figure