Spectroformer: Multi-Domain Query Cascaded Transformer Network For Underwater Image Enhancement

Md Raqib Khan, Priyanka Mishra, Nancy Mehta, Shruti S. Phutke, Santosh Kumar Vipparthi, Sukumar Nandi, Subrahmanyam MuralaWACV 2024 · OpenReview

Abstract

Underwater images often suffer from color distortion, haze, and limited visibility due to light refraction and absorption in water. These challenges significantly impact autonomous underwater vehicle (AUV) applications, necessitating efficient image enhancement techniques. To address these challenges, we propose a Multi-Domain Query Cascaded Transformer Network for underwater image enhancement. Our approach includes a novel Multi-Domain Query Cascaded Attention mechanism that integrates localized transmission features and global illumination features. To improve feature propagation from the encoder to the decoder, we propose a Spatio-Spectro Fusion-Based Attention Block. Additionally, we introduce a Hybrid Fourier-Spatial Up-sampling Block, which uniquely combines Fourier and spatial upsampling techniques to enhance feature resolution effectively. We evaluate our method on benchmark synthetic and real-world underwater image datasets, demonstrating its superiority through extensive ablation studies and comparative analysis.

Introduction

Capturing clear images underwater is a significant challenge for computer vision. Due to the physical properties of water, light is absorbed and scattered unevenly across different wavelengths. This results in three major issues: color distortion (where red light is absorbed first, leading to heavy green/blue tints), haziness from light scattering, and low visibility due to refraction. While deep learning—and specifically Transformers—has revolutionized image restoration, most existing models focus purely on the spatial domain. Spectroformer addresses these limitations by bridging the gap between spatial and frequency (spectral) information to capture fine-grained textures and global illumination patterns.

Methodology

The Spectroformer architecture is designed to process data in both the spatial and frequency domains simultaneously through three core innovations:

  • Multi-Domain Query Cascaded Attention (MQCA): Unlike standard Transformers that compare pixels to pixels, MQCA processes queries in the frequency domain and keys in the spatial domain. This allows the model to integrate localized transmission features with scene-wide global illumination features.
  • Spatio-Spectro Fusion-Based Attention Block (SSFB): Replacing traditional direct skip connections, the SSFB filters redundant information using an Adaptive Kernel Convolution (AKC). The AKC automatically adjusts its kernel size based on input channels to prevent over-smoothing.
  • Hybrid Fourier-Spatial Upsampling: This block combines spatial "Pixel-shuffling" with "Deep Fourier Upsampling" to retrieve high-frequency details often lost in murky underwater environments.
The Spectroformer architecture featuring Multi-Domain Query Cascaded Attention (MQCA) and Hybrid Fourier-Spatial Upsampling blocks.

Experimental Results

The researchers evaluated Spectroformer on a mix of synthetic and real-world benchmarks, including the Underwater Image Enhancement Benchmark (UIEB), U45, UCCS, and SQUID datasets.

Quantitative Performance

On the UIEB dataset, Spectroformer achieved a Peak Signal-to-Noise Ratio (PSNR) of 24.96 and a Structural Similarity Index (SSIM) of 0.917, outperforming several state-of-the-art (SOTA) models such as WaterNet and U-shape Transformers. Ablation studies confirmed that the integration of Multi-Domain Attention and the Fusion Block provided consistent performance gains.

Qualitative Analysis

Visual results demonstrate a dramatic reduction in the "green veil" common in underwater shots. The model produces sharper edges and more natural color balance compared to previous methods across diverse marine environments.

Qualitative comparison across UCCS, U45, and SQUID datasets showing Spectroformer's superior color restoration and haze removal compared to other SOTA methods.

Beyond visual aesthetics, the SQUID dataset tests showed that Spectroformer significantly improves the accuracy of downstream tasks like depth estimation, which is critical for AUV navigation and object detection.

Conclusion

Spectroformer provides a robust dual-domain solution for underwater image enhancement. By effectively combining spatial and spectral features, it sets a new benchmark for restoration quality. Future work will focus on reducing computational complexity for real-time deployment on low-power AUV hardware and extending the model to ensure temporal consistency in video processing.

Summary updated

Abstract

Underwater images often suffer from color distortion, haze, and limited visibility due to light refraction and absorption in water. These challenges significantly impact autonomous underwater vehicle applications, necessitating efficient image enhancement techniques. To address these challenges, we propose a Multi-Domain Query Cascaded Transformer Network for underwater image enhancement. Our approach includes a novel Multi-Domain Query Cascaded Attention mechanism that integrates localized transmission features and global illumination features. To improve feature propagation from the encoder to the decoder, we propose a Spatio-Spectro Fusion-Based Attention Block. Additionally, we introduce a Hybrid Fourier-Spatial Up-sampling Block, which uniquely combines Fourier and spatial upsampling techniques to enhance feature resolution effectively. We evaluate our method on benchmark synthetic and real-world underwater image datasets, demonstrating its superiority through extensive ablation studies and comparative analysis. The testing code is available at: https://github.com/Mdraqibkhan/Spectroformer.
BibTeX
@inproceedings{khan2024spectroformer,
  author = {Md Raqib Khan and Priyanka Mishra and Nancy Mehta and Shruti S. Phutke and Santosh Kumar Vipparthi and Sukumar Nandi and Subrahmanyam Murala},
  title = {{Spectroformer: Multi-Domain Query Cascaded Transformer Network For Underwater Image Enhancement}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=BBpaW2AI10},
}
Figure