Learning Steerable Function for Efficient Image Resampling

Jiacheng Li, Chang Chen, Wei Huang, Zhiqiang Lang, Fenglong Song, Youliang Yan, Zhiwei XiongCVPR 2023 · OpenReview

Abstract

Image resampling is a basic technique that is widely employed in daily applications. Existing deep neural networks (DNNs) have made impressive progress in resampling performance. Yet these methods are still not the perfect substitute for interpolation, due to the issues of efficiency and continuous resampling. In this work, we propose a novel method of Learning Resampling Function (termed LeRF), which takes advantage of both the structural priors learned by DNNs and the locally continuous assumption of interpolation methods. Specifically, LeRF assigns spatially-varying steerable resampling functions to input image pixels and learns to predict the hyper-parameters that determine the orientations of these resampling functions with a neural network. To achieve highly efficient inference, we adopt look-up tables (LUTs) to accelerate the inference of the learned neural network. Furthermore, we design a directional ensemble strategy and edge-sensitive indexing patterns to better capture local structures. Extensive experiments show that our method runs as fast as interpolation, generalizes well to arbitrary transformations, and outperforms interpolation significantly, e.g., up to 3dB PSNR gain over bicubic for x2 upsampling on Manga109.

Introduction

Image resampling—the process of resizing, rotating, or warping an image—is a cornerstone of digital imaging. Traditionally, we rely on interpolation (like bilinear or bicubic), which is incredibly fast and works for any transformation but often results in blurry edges and lost detail. In recent years, Deep Neural Networks (DNNs) have revolutionized image quality through Super-Resolution, but they face two major hurdles: efficiency and flexibility. Most DNNs are computationally expensive for mobile devices and are often trained for specific scales, failing to generalize to arbitrary warping or rotation.

LeRF (Learning Resampling Function) bridges this gap by finding a "middle way": a method that possesses the structural intelligence of a DNN but the speed and versatility of traditional interpolation.

Methods

The core of LeRF is the use of Steerable Resampling Functions. Instead of using a fixed, symmetric kernel (like a standard Gaussian or Bicubic curve), LeRF uses an anisotropic Gaussian. This function can be stretched and rotated (steered) to align perfectly with local image structures, such as sharp edges or fine lines.

Figure 1: Comparison between fixed bicubic resampling and the proposed learned steerable resampling functions across various transformations.

The architecture consists of three main components:

  1. Hyper-parameter Prediction: The "steering" is controlled by parameters ($\rho, \sigma_X, \sigma_Y$). A small DNN is trained to predict these parameters for every pixel location based on its local neighborhood.
  2. LUT Acceleration: To achieve interpolation-level speed, the DNN is not run at inference time. Instead, its outputs are pre-calculated for all possible local pixel combinations and stored in Look-Up Tables (LUTs). At runtime, the system simply "looks up" the best kernel parameters, turning complex math into simple memory access.
  3. Directional Ensemble and Indexing: To ensure the LUTs capture edge orientation accurately, the authors introduced a Directional Ensemble (DE) strategy and edge-sensitive indexing patterns (termed "C" and "X" patterns) to preserve directional information.

Evaluation and Results

The model was trained on the DIV2K dataset and evaluated across five classic benchmarks: Set5, Set14, BSDS100, Urban100, and Manga109. These datasets cover a wide range of textures, from natural landscapes to sharp man-made structures and fine-lined illustrations.

Figure 2: Performance comparison (PSNR vs. Running Time) on a mobile device. LeRF achieves a superior quality-speed trade-off compared to traditional interpolation and heavy DNN methods.

Key results include:

  • Speed: On a mobile CPU, LeRF processes a 720p image in roughly 110ms, which is comparable to or even faster than standard Bicubic interpolation (~126ms) in their tests, and orders of magnitude faster than DNN-based methods like LIIF (which took over 60 seconds).
  • Quality: LeRF significantly outperforms interpolation. For example, on the Manga109 dataset, LeRF achieved a 3dB PSNR gain over bicubic interpolation for 2x upsampling.
  • Versatility: Unlike many SR models, LeRF handles rotation, shearing, and arbitrary warping (like barrel distortion) with ease, maintaining sharp edges throughout the transformation.

Conclusion and Future Work

LeRF demonstrates that sophisticated learned priors can be deployed on low-power devices by baking intelligence into LUTs. While the method excels at preserving sharpness, the authors note potential aliasing in extremely dense textures, suggesting that future work could integrate adaptive anti-aliasing filters. Additionally, the steerable function framework could be extended to other modalities, such as video resampling or 3D point clouds.

Summary updated

Abstract

Image resampling is a basic technique that is widely employed in daily applications. Existing deep neural networks (DNNs) have made impressive progress in resampling performance. Yet these methods are still not the perfect substitute for interpolation, due to the issues of efficiency and continuous resampling. In this work, we propose a novel method of Learning Resampling Function (termed LeRF), which takes advantage of both the structural priors learned by DNNs and the locally continuous assumption of interpolation methods. Specifically, LeRF assigns spatially-varying steerable resampling functions to input image pixels and learns to predict the hyper-parameters that determine the orientations of these resampling functions with a neural network. To achieve highly efficient inference, we adopt look-up tables (LUTs) to accelerate the inference of the learned neural network. Furthermore, we design a directional ensemble strategy and edge-sensitive indexing patterns to better capture local structures. Extensive experiments show that our method runs as fast as interpolation, generalizes well to arbitrary transformations, and outperforms interpolation significantly, e.g., up to 3dB PSNR gain over bicubic for x 2 upsampling on Manga109.
BibTeX
@inproceedings{li2023learning,
  author = {Jiacheng Li and Chang Chen and Wei Huang and Zhiqiang Lang and Fenglong Song and Youliang Yan and Zhiwei Xiong},
  title = {{Learning Steerable Function for Efficient Image Resampling}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=TlpScjUhY7},
}
Figure