Abstract
3D medical image segmentation is often limited by the high computational cost of self-attention mechanisms and the sensitivity of models to input resolution. This paper presents HartleyMHA, a novel architecture that performs self-attention in the frequency domain using the Hartley transform. By focusing on essential low-frequency components and sharing parameters across frequencies, HartleyMHA achieves significant parameter efficiency—using less than 1% of the parameters of traditional models—while demonstrating superior robustness to resolution variations.
Introduction
The "curse of dimensionality" in 3D medical imaging, such as MRI, poses a significant challenge for deep learning models. While Transformers effectively capture long-range spatial dependencies, their quadratic complexity leads to memory exhaustion at high resolutions. Common workarounds, like downsampling or patch-based training, often result in performance degradation when models are deployed on full-resolution clinical data. There is a critical need for models that are both parameter-efficient and resolution-robust, capable of generalizing from low-resolution training data to high-resolution inference.
Methodology
The proposed approach builds on the Fourier Neural Operator (FNO) framework, introducing two primary components: the Hartley Neural Operator (HNOSeg) and Hartley Multi-Head Attention (HartleyMHA).
- Hartley Neural Operator (HNOSeg): Unlike the complex-valued Fourier Transform, the Hartley Transform operates entirely in the real-number domain. This reduces memory and computational overhead. By sharing model parameters across different frequencies, the authors drastically reduce the parameter count from millions to just a few thousand.
- Hartley Multi-Head Attention (HartleyMHA): Self-attention is applied directly in the frequency domain. Since global structure in medical images is primarily encoded in low frequencies, the model discards high-frequency noise and performs attention on a small subset of the spectrum. Efficiency is further enhanced by grouping frequency voxels into $2 \times 2 \times 2$ patches, reducing the sequence length for the attention mechanism by a factor of eight.
Experimental Results
The models were evaluated on the BraTS’19 (Brain Tumor Segmentation) dataset. To test resolution robustness, models were trained on images downsampled by factors of 1x to 4x but always evaluated on the original high-resolution volumes.
HartleyMHA demonstrated extreme parameter efficiency, requiring only 47.7k parameters compared to 5.7M for V-Net and 144.5M for a standard FNO. Crucially, it exhibited remarkable robustness: while standard models like V-Net and UTNet saw a drop in Dice scores of over 14.8% when training resolution was reduced by a factor of 3, HartleyMHA's performance dropped by less than 2.5%.
Conclusion
HartleyMHA offers a scalable and efficient solution for 3D medical image segmentation. By leveraging frequency-domain operations and the Hartley transform, it overcomes the memory limitations of spatial-domain Transformers and provides consistent performance across varying image resolutions. While the current model relies on a fixed frequency cutoff ($k_{max}$), future work may explore adaptive frequency selection to capture finer details for specific clinical tasks.
Summary updated