Local Implicit Wavelet Transformer for Arbitrary-Scale Super-Resolution

Minghong Duan, Linhao Qu, Shaolei Liu, Manning Wang

Abstract

Arbitrary-scale image super-resolution (SR) aims to reconstruct high-resolution images at any continuous scale factor using a single model. While Implicit Neural Representations (INRs) have advanced this field, existing methods often suffer from blurry textures and aliased edges due to their reliance on coordinate-based local ensembles that ignore high-frequency content. This paper presents the Local Implicit Wavelet Transformer (LIWT), a novel architecture that integrates Discrete Wavelet Transform (DWT) priors into a cross-attention mechanism. By leveraging frequency-aware features, LIWT effectively restores sharp details and outperforms state-of-the-art methods in both quantitative metrics and visual fidelity.

Introduction

Single Image Super-Resolution (SISR) has traditionally focused on fixed integer upscaling factors (e.g., 2x, 4x). However, real-world applications often require arbitrary-scale upscaling to accommodate diverse display resolutions. Recent approaches have adopted Implicit Neural Representations (INRs), treating images as continuous functions where pixel values are queried based on spatial coordinates.

Despite their flexibility, current INR-based models like LIIF and LTE frequently produce suboptimal results at large or non-integer scales. These models typically utilize a "local ensemble" technique, where the value of a target pixel is an interpolation of neighboring features weighted by spatial distance. Because this weighting is purely geometric and ignores the underlying image content, it often fails to capture high-frequency textures, leading to "hallucinated" artifacts or excessive blurring. The Local Implicit Wavelet Transformer (LIWT) addresses this by making the reconstruction process feature-aware rather than just coordinate-aware.

Methodology

LIWT introduces a specialized pipeline designed to inject high-frequency information into the continuous representation. The architecture is built around three primary innovations:

1. Wavelet Enhancement Residual Module (WERM)

After initial feature extraction via an encoder, the model employs a Discrete Wavelet Transform (DWT) to decompose features into four distinct frequency sub-bands: Low-Low (LL), Low-High (LH), High-Low (HL), and High-High (HH). The WERM specifically processes these sub-bands to isolate and amplify high-frequency texture information, ensuring that the model has a dedicated pathway for edge preservation.

2. Wavelet Mutual Projected Fusion (WMPF)

To integrate these enhanced priors back into the main feature stream, the WMPF module uses channel-wise attention. This fusion mechanism ensures that the most relevant high-frequency components are prioritized, allowing the deep features to benefit from the structural roadmap provided by the wavelet decomposition.

3. Wavelet-aware Implicit Attention (WIA)

Replacing the traditional distance-based ensemble, the WIA serves as the core upsampling engine. It utilizes a cross-attention mechanism where the query coordinate acts as the "query" and local features serve as "keys" and "values." Crucially, these interactions are weighted based on the high-frequency priors generated by the WERM. This allows the model to dynamically adjust its interpolation strategy based on the actual content—for instance, assigning higher weights to features that represent a sharp edge relevant to the target pixel.

Curriculum Learning Strategy

To stabilize training for complex arbitrary scales, the authors implement a curriculum learning approach. The model is initially trained on simpler upscaling tasks (x1–x4) before gradually progressing to more challenging factors (up to x8). This staged approach helps the network learn fundamental structures before attempting to reconstruct intricate high-frequency details.

Experimental Results

Quantitative Performance

The LIWT was evaluated against standard benchmarks, including Set5, Set14, B100, and Urban100, using DIV2K for training. Quantitative results demonstrate that LIWT consistently achieves superior Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) scores. Notably, on the Urban100 dataset—which is rich in repetitive geometric patterns—LIWT achieved a PSNR of 26.95 dB at x4 scale, significantly outperforming previous leaders like LTE (26.81 dB).

Qualitative Analysis

Visual comparisons highlight LIWT’s ability to reconstruct sharp, realistic textures where other models fail. In scenarios involving building grids or fine floral patterns, LIWT avoids the aliasing and blurring common in coordinate-only methods. Furthermore, the model demonstrates remarkable robustness at extreme scales (up to x30), maintaining structural integrity and edge sharpness far beyond its training distribution.

Discussion and Conclusion

While LIWT sets a new benchmark for arbitrary-scale SR, it involves a trade-off in computational efficiency. With 3.97M parameters compared to LTE’s 1.71M, and an inference time of approximately 0.50s per image, it is more resource-intensive than its predecessors. The inclusion of attention mechanisms and wavelet modules also increases the GPU memory footprint.

In conclusion, the Local Implicit Wavelet Transformer successfully bridges the gap between continuous coordinate representations and frequency-domain analysis. By making the upscaling process feature-aware through wavelet-guided attention, LIWT provides a robust solution for high-fidelity image reconstruction at any scale. Future research will focus on optimizing the architecture for real-time performance on mobile and edge devices.

Summary updated

BibTeX
@inproceedings{duan2024local,
  author = {Minghong Duan and Linhao Qu and Shaolei Liu and Manning Wang},
  title = {{Local Implicit Wavelet Transformer for Arbitrary-Scale Super-Resolution}},
  booktitle = {BMVC},
  year = {2024},
}
Figure