Training Transformer Models by Wavelet Losses Improves Quantitative and Visual Performance in Single Image Super-Resolution

Cansu Korkmaz, Ahmet Murat TekalpWorkshops @ CVPR 2024 · OpenReview

Abstract

Transformer-based models have achieved remarkable results in low-level vision tasks including image super-resolution (SR). However, early Transformer-based approaches that rely on self-attention within non-overlapping windows encounter challenges in acquiring global information. To activate more input pixels globally, hybrid attention models have been proposed. Moreover, training by solely minimizing pixel-wise RGB losses, such as L1, have been found inadequate for capturing essential high-frequency details. This paper presents two contributions: i) We introduce convolutional non-local sparse attention (NLSA) blocks to extend the hybrid transformer architecture in order to further enhance its receptive field. ii) We employ wavelet losses to train Transformer models to improve quantitative and subjective performance. Our experimental results demonstrate that the proposed model provides state-of-the-art PSNR results as well as superior visual performance across various benchmark datasets.

Introduction

Single-image super-resolution (SISR) aims to reconstruct a high-resolution (HR) image from a single low-resolution (LR) input. While deep learning has revolutionized this field, two major hurdles remain. First, many Transformer-based models use window-based self-attention, which struggles to capture global information or long-range dependencies, such as repeating patterns in architectural images. Second, standard pixel-wise losses (like $L_1$ or $L_2$) tend to produce "blurry" results. These metrics often fail to distinguish between a slightly shifted edge and a blurry one, resulting in reconstructions that lack high-frequency (HF) details like textures and sharp edges.

Methodology

The Wavelettention Architecture

The proposed model, termed Wavelettention, builds upon the Hybrid Attention Transformer (HAT). To further expand the model's receptive field, the HAT structure is augmented with Convolutional Non-Local Sparse Attention (NLSA) blocks. NLSA allows the model to attend to related pixels across the entire image efficiently. This is particularly beneficial for reconstructing repetitive textures and global structures.

The proposed Wavelettention architecture featuring NLSA blocks and Hybrid Attention Groups.

Stationary Wavelet Transform (SWT) Loss

The core innovation is the introduction of a Wavelet Loss ($L_{SWT}$). During training, both the predicted SR image and the ground truth HR image are converted to the YCbCr color space. The Stationary Wavelet Transform (SWT) is then applied to the Y (luminance) channel, decomposing it into four subbands: LL (low-frequency), LH (horizontal), HL (vertical), and HH (diagonal).

Pipeline for calculating the Wavelet Loss using SWT on the Y-channel.

By calculating the $L_1$ loss in the wavelet domain, the model is forced to minimize errors in specific frequency components. This explicitly encourages the reconstruction of fine-grained details that are often smoothed over by standard RGB-domain losses. To avoid chroma artifacts (such as greenish tints), the wavelet loss is carefully weighted ($\lambda = 0.05$) alongside the standard RGB loss.

Results and Discussion

Wavelettention was evaluated on standard benchmarks including Set5, Set14, BSD100, and Urban100, after training on the LSDIR and DIV2K datasets.

Quantitative Performance

The model achieved state-of-the-art PSNR results across all tested benchmarks. Notably, it outperformed the previous state-of-the-art (HAT) by a significant 0.72 dB on the Urban100 dataset, which is known for its complex, structured patterns.

Visual Fidelity

Qualitatively, the model excels at restoring text and fine parallel lines. Where other models produced "hallucinated" wavy lines or blurry artifacts, Wavelettention reconstructed straight, distinct structures that closely matched the ground truth. The authors also demonstrated that the wavelet loss is a generic improvement that can be "plugged in" to other architectures like SwinIR to enhance their performance.

Conclusion

This work introduces a powerful combination of non-local attention and wavelet-domain training for SISR. By expanding the receptive field and providing a clearer mathematical signal for sharpness via wavelets, the Wavelettention model sets a new standard for both quantitative metrics and visual quality. Future work will focus on optimizing computational complexity as more NLSA blocks are added and exploring more robust ways to integrate color information into the wavelet domain to further mitigate color sensitivity.

Summary updated

Abstract

Transformer-based models have achieved remarkable results in low-level vision tasks including image super-resolution (SR). However, early Transformer-based approaches that rely on self-attention within non-overlapping windows encounter challenges in acquiring global information. To activate more input pixels globally, hybrid attention models have been proposed. Moreover, training by solely minimizing pixel-wise RGB losses, such as L1, have been found inadequate for capturing essential high-frequency details. This paper presents two contributions: i) We introduce convolutional non-local sparse attention (NLSA) blocks to extend the hybrid transformer architecture in order to further enhance its receptive field. ii) We employ wavelet losses to train Transformer models to improve quantitative and subjective performance. While wavelet losses have been explored previously, showing their power in training Transformer-based SR models is novel. Our experimental results demonstrate that the proposed model provides state-of-the-art PSNR results as well as superior visual performance across various benchmark datasets.
BibTeX
@inproceedings{korkmaz2024training,
  author = {Cansu Korkmaz and Ahmet Murat Tekalp},
  title = {{Training Transformer Models by Wavelet Losses Improves Quantitative and Visual Performance in Single Image Super-Resolution}},
  booktitle = {Workshops @ CVPR},
  year = {2024},
  url = {https://openreview.net/forum?id=rkNKyIflXy},
}
Figure