Abstract
Deep CNNs have achieved significant successes in image processing and its applications, including single image super-resolution (SR). However, conventional methods still resort to some predetermined integer scaling factors, e.g., x2 or x4. Thus, they are difficult to be applied when arbitrary target resolutions are required. Recent approaches extend the scope to real-valued upsampling factors, even with varying aspect ratios to handle the limitation. In this paper, we propose the SRWarp framework to further generalize the SR tasks toward an arbitrary image transformation. We interpret the traditional image warping task, specifically when the input is enlarged, as a spatially-varying SR problem. We also propose several novel formulations, including the adaptive warping layer and multiscale blending, to reconstruct visually favorable results in the transformation process. Compared with previous methods, we do not constrain the SR model on a regular grid but allow numerous possible deformations for flexible and diverse image editing. Extensive experiments and ablation studies justify the necessity and demonstrate the advantage of the proposed SRWarp method under various transformations.
Introduction: Beyond Fixed Grids
Traditional Super-Resolution (SR) methods are typically "grid-locked," designed to upscale images by fixed integer factors (like x2 or x4) onto a regular rectangular grid. While recent "arbitrary-scale" SR methods have introduced flexibility for real-valued scaling, they still struggle with complex geometric deformations like rotation, shearing, or lens distortion correction.
In real-world image editing, warping is a fundamental operation. When a local region of an image is stretched using standard interpolation (e.g., OpenCV's bicubic functions), the result is often blurry because the algorithm lacks the ability to synthesize missing high-frequency details. Conversely, applying a standard SR model before warping is computationally expensive and suboptimal, as the SR model is unaware of the specific downstream deformation.
Methodology: Warping as Spatially-Varying SR
The SRWarp framework treats every pixel in a warped image as having its own unique, local scaling factor. The architecture introduces three main innovations:
1. Adaptive Warping Layer (AWL)
Instead of using a fixed interpolation kernel, the AWL utilizes the Jacobian of the transformation to calculate local distortion. A learnable kernel estimator—a small neural network—dynamically predicts resampling weights based on this local geometry, allowing the model to adapt to shearing and rotation.
2. Multiscale Blending
The model extracts features at multiple resolutions (e.g., x1, x2, and x4) using a shared backbone. These features are blended using a "scale feature" derived from the determinant of the Jacobian, which represents local magnification. This allows the model to use low-resolution features to prevent aliasing in smooth areas while leveraging high-resolution features to maintain crisp edges in textured regions.
3. Partial Convolution
Warping often creates "void" pixels at image boundaries. Standard CNNs treat these as zeros, which can "leak" into valid regions and cause artifacts. SRWarp employs partial convolutions and a binary mask to ensure the network only processes valid pixel data.
Experiments and Results
To evaluate the framework, the authors introduced the DIV2K-Warping (DIV2KW) dataset, consisting of 700 random warping matrices (upscaling, shearing, rotation, and perspective projection) applied to high-quality images.
Quantitative Performance
On the DIV2KW test set, SRWarp achieved a modified PSNR (mPSNR) of 31.04 dB, significantly outperforming the OpenCV bicubic baseline (27.85 dB). It also surpassed "SR then Warp" sequential pipelines; for instance, using the state-of-the-art RRDB model followed by OpenCV warping reached only 30.76 dB.
Visual Quality
Qualitative results demonstrate that SRWarp effectively removes the "jaggies" and blurriness common in traditional lens distortion correction and perspective warping.
Conclusion
SRWarp redefines image warping as a spatially-varying super-resolution task. By integrating local geometric information via the Jacobian and employing multiscale feature blending, the framework provides a flexible and high-quality solution for arbitrary image transformations. Future work may involve expanding the transformation scope to non-linear deformations and optimizing the architecture for real-time performance on mobile devices.
Summary updated