SRWarp: Generalized Image Super-Resolution under Arbitrary Transformation

Sanghyun Son, Kyoung Mu Lee

Abstract

Deep CNNs have achieved significant successes in single image super-resolution (SR). However, conventional methods still resort to predetermined integer scaling factors, making them difficult to apply when arbitrary target resolutions or geometric transformations are required. We propose the SRWarp framework to generalize SR tasks toward arbitrary image transformations. By interpreting image warping as a spatially-varying SR problem, we introduce an adaptive warping layer and multiscale blending to reconstruct visually favorable results. Extensive experiments demonstrate the advantage of SRWarp over traditional interpolation and fixed-scale SR methods.

Introduction

Standard Super-Resolution (SR) research typically focuses on uniform upscaling, where a low-resolution (LR) image is mapped to a high-resolution (HR) grid by a fixed integer factor. However, real-world applications like lens distortion correction, perspective shifts, or homographic alignments involve non-uniform stretching and compression across the image surface.

Traditional tools, such as OpenCV’s warpPerspective, rely on bicubic interpolation. While computationally efficient, these methods often produce blurry edges and lose high-frequency textures because they lack the capacity to reconstruct missing details. Conversely, existing SR models are constrained by regular grids and fixed scales. SRWarp addresses this gap by treating arbitrary transformations as a spatially-varying SR task where the magnification factor changes at every pixel.

Methodology

The core conceptual shift in SRWarp is viewing an image warp as a "curved slice" through a scale pyramid. In this model, the LR image resides at the top ($z=1$), and a high-magnification SR image resides at the bottom (e.g., $z=4$). A warped image is reconstructed by sampling from this continuous scale space.

Conceptual diagram of SRWarp viewing warping as a slice through a scale pyramid compared to conventional fixed-scale SR.

Adaptive Warping Layer (AWL)

To handle local deformations, the AWL utilizes the Jacobian matrix of the transformation to linearize the geometry at each pixel. Instead of a static kernel, the AWL dynamically predicts a resampling kernel that accounts for local rotation and stretching. This ensures that the sampling process is geometrically consistent with the intended transformation.

Multiscale Blending and Partial Convolutions

The framework extracts features at multiple scales (x1, x2, and x4) and blends them based on a "scale feature" derived from the Jacobian determinant. This determinant represents local magnification, allowing the model to prioritize low-resolution features in "shrunk" areas to avoid aliasing and high-resolution features in "stretched" areas to provide detail. Additionally, partial convolutions are employed to handle "void" pixels at image boundaries, preventing artifacts from leaking into the reconstructed regions.

Experiments and Results

The authors introduced the DIV2K-Warping (DIV2KW) dataset, consisting of high-quality images subjected to random projective transformations. SRWarp was evaluated against traditional interpolation and state-of-the-art SR models combined with standard warping.

Quantitative Performance

On the DIV2KW test set, SRWarp achieved a 31.04 dB mPSNR, significantly outperforming the combination of a state-of-the-art SR model (RDN) and OpenCV warping (30.22 dB). Furthermore, SRWarp proved efficient, running approximately 2.5x faster and using 40% less memory than specialized arbitrary-scale models like Meta-RDN.

Qualitative Comparison

Visual results highlight SRWarp's ability to eliminate "staircase" artifacts and blurriness common in traditional perspective warping.

Qualitative comparison showing (a) Input with distortion, (b) OpenCV's bicubic interpolation, and (c) SRWarp's sharp reconstruction.

Conclusion

SRWarp successfully generalizes the super-resolution task to arbitrary geometric transformations. By integrating the transformation's Jacobian into the network architecture via the Adaptive Warping Layer and multiscale blending, the model produces high-quality, sharp images that surpass traditional interpolation techniques. While currently focused on homographies, the framework provides a robust foundation for future work in complex, non-rigid image deformations.

Summary updated

Abstract

Deep CNNs have achieved significant successes in image processing and its applications, including single image super-resolution (SR). However, conventional methods still resort to some predetermined integer scaling factors, e.g., x2 or x4. Thus, they are difficult to be applied when arbitrary target resolutions are required. Recent approaches extend the scope to real-valued upsampling factors, even with varying aspect ratios to handle the limitation. In this paper, we propose the SRWarp framework to further generalize the SR tasks toward an arbitrary image transformation. We interpret the traditional image warping task, specifically when the input is enlarged, as a spatially-varying SR problem. We also propose several novel formulations, including the adaptive warping layer and multiscale blending, to reconstruct visually favorable results in the transformation process. Compared with previous methods, we do not constrain the SR model on a regular grid but allow numerous possible deformations for flexible and diverse image editing. Extensive experiments and ablation studies justify the necessity and demonstrate the advantage of the proposed SRWarp method under various transformations.
BibTeX
@inproceedings{son2021srwarp,
  author = {Sanghyun Son and Kyoung Mu Lee},
  title = {{SRWarp: Generalized Image Super-Resolution under Arbitrary Transformation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=uNxS8w7tnI},
}
Figure