TL;DR
Phaseformer is a lightweight transformer network (1.77M parameters) designed for efficient underwater image restoration by exploiting a key physical insight: underwater degradation severely distorts image amplitude (color and contrast) but leaves structural phase information relatively intact. By introducing phase-based self-attention and optimized skip connections, Phaseformer achieves state-of-the-art restoration quality with a fraction of the computational cost of competing models.
What Problem Does This Paper Address?
Underwater imaging is notoriously difficult due to light refraction, absorption, and scattering by water molecules and suspended particles. This results in severe color casts (often heavy blue or green tints), haziness, blurriness, and low contrast. For autonomous underwater vehicles (AUVs) and marine robotics relying on computer vision, this degradation severely degrades performance in tasks like object detection and mapping.
While deep learning and transformer-based models have advanced underwater image enhancement (UIE), they are typically computationally heavy, making them impractical for real-time deployment on autonomous edge devices. Furthermore, standard architectures often inadvertently propagate noise and degraded features through direct skip connections. Phaseformer addresses this dual challenge by designing a highly efficient, structure-focused architecture.
Methods
The core innovation of Phaseformer is its reliance on frequency-domain phase information to guide spatial attention. The architecture is built as a U-Net variant utilizing two novel components:
- Phase-based Multi-head Self-Attention (PMSA): Inside the Phase-based Transformer Blocks (PBTB), the network uses a Phase Extraction Module (PEM). By applying a Fast Fourier Transform (FFT) to the feature maps, setting the amplitude to a constant, and computing the Inverse FFT (IFFT), the network isolates pure structural phase features. The queries ($Q$) and keys ($K$) are derived from these clean phase features, ensuring the attention mechanism calculates pixel interactions based on stable structural layouts rather than corrupted colors or noise.
- Optimized Phase Attention Block (OPAB): Instead of using standard, direct skip connections that pass degraded encoder features straight to the decoder, Phaseformer routes them through the OPAB. This block applies global average pooling and an adaptive 1D convolution whose kernel size dynamically scales with the channel depth, preventing feature over-smoothing or under-smoothing.
Additionally, the authors introduce an Adaptive Weight Assignment strategy. Instead of manually tuning the weights of the multiple loss functions used (Charbonnier, Gradient, MS-SSIM, and Perceptual losses), the network treats these weights as learnable tensors that optimize automatically during training.
Datasets
The model was trained and evaluated across a comprehensive suite of synthetic and real-world datasets:
- Synthetic: UIEB (Underwater Image Enhancement Benchmark—800 training pairs) and UFO-120 (1500 training pairs).
- Real-World: U45 (45 challenging real-world images), UCCS (Underwater Color Cast Set), and SQUID (Stereo Quantitative Underwater Image Dataset).
- Extensions: The authors also tested the model's versatility on the LOL (Low-Light) dataset for low-light image enhancement.
Key Insights and Ideas
- The Resilience of Phase: Using t-SNE dimensionality reduction, the authors demonstrate that clean and degraded underwater images form completely distinct clusters when analyzing amplitude, but heavily overlap when analyzing phase. This confirms that phase information acts as a reliable, degradation-resistant skeleton of the image.
- Phase-Guided Attention Filters Noise: By computing self-attention maps purely from phase data, the transformer avoids getting "distracted" by severe color shifts or localized haziness, allowing it to restore global color consistency more effectively.
- Smart Skip Connections: Filtering skip connections through a phase-aware attention block ensures that only prominent, clean structural boundaries are passed to the decoder, actively filtering out encoder-level noise.
Key Results
Phaseformer delivers state-of-the-art performance while remaining remarkably lightweight:
- Restoration Quality: On the synthetic UIEB dataset, Phaseformer achieved a PSNR of 25.98 dB and SSIM of 0.928, outperforming recent SOTA models like Spectroformer (24.96 dB / 0.917) and Semi-UIR (24.59 dB / 0.901).
- Computational Efficiency: The model contains only 1.77M parameters and requires 13.0 GFLOPs. In comparison, the popular U-shape transformer uses 65.6M parameters and 66.2 GFLOPs. Phaseformer processes images at an impressive 0.034 seconds per image, making it highly viable for real-time robotic applications.
- Downstream Boost: Using Phaseformer as a pre-processing step significantly improved accuracy for downstream tasks like object detection (via YOLO-v3) and monocular depth estimation.
Limitations and Future Work
While Phaseformer excels at correcting severe color casts and restoring contrast, the authors note a limitation in handling extremely muddy or heavily blurred water conditions. The current architecture focuses heavily on structural and color restoration, leaving room for improvement in dedicated image deblurring. The authors plan to target specialized deblurring mechanisms in future iterations of the network.
Summary updated