Block-Based Multi-Scale Image Rescaling

Jian Li, Siwang ZhouAAAI 2025 · DOI

TL;DR

This paper introduces a novel Block-Based Multi-Scale Image Rescaling (BBMR) framework that dynamically adjusts scaling rates for different image regions to improve super-resolution quality. By treating image blocks with varying information density differently during downscaling, BBMR significantly enhances image reconstruction performance, especially for high-resolution images.

What Problem Does This Paper Address?

The increasing demand for high-resolution images (2K and above) presents a significant challenge for storage and transmission bandwidth. Traditional image rescaling (IR) methods, which aim to create a low-resolution (LR) representation for efficient storage and later reconstruct a high-resolution (HR) image, often fall short. These methods typically apply a uniform scaling rate across the entire image, neglecting the fact that different image regions contain varying amounts of information. This uniform approach can lead to a loss of detail and artifacts, particularly in complex areas. The paper addresses this by proposing a method that intelligently downscales different parts of an image at different rates, optimizing the LR representation for better SR reconstruction.

Methods

The proposed BBMR framework consists of two main modules:

  • Downscaling Module: This module is the core innovation. Instead of a uniform scaling rate, it divides the HR image into fixed-size sub-blocks (128x128). For each sub-block, a dynamic scaling rate is allocated based on its perceived information content or difficulty for super-resolution. This is achieved through a "Block-Based Multi-Scale Downscaling Strategy." The paper identifies "simple," "medium," and "hard" blocks based on PSNR changes after downscaling. The overall scaling rate is maintained constant by balancing the allocation of blocks with higher and lower scaling rates. To further reduce computational load, a lightweight super-resolution model can be used to assist in this scaling rate allocation.
  • Upscaling Module: This module utilizes a "Joint Super-Resolution" (JointSR) method. JointSR performs super-resolution on the downscaled sub-blocks. A key feature of JointSR is its integrated "deblock branch," which addresses blocking artifacts at the feature level before upscaling. This is more computationally efficient than post-processing the high-resolution SR image. The SRNet Feature Extraction layers in JointSR are derived from pre-trained SR models, with the final upscaling layer removed.

Datasets

  • Training Data: DIV2K and Flickr2K datasets were used for training.
  • Testing Data: Test2K and Test4K datasets, derived from the DIV8K dataset and downscaled using bicubic interpolation, were used for evaluation.

Key Insights and Ideas

  • Information Density Varies: The paper empirically demonstrates that different image regions have varying "difficulty" for super-resolution. This is a crucial insight that traditional uniform scaling methods overlook.
  • Adaptive Downscaling is Key: By intelligently allocating different scaling rates to different blocks, the quality of the LR representation can be significantly improved, leading to better SR reconstruction.
  • Deblocking at LR Stage is Efficient: The JointSR method's deblocking branch operating on LR features is a clever way to combat artifacts without incurring the high computational cost of deblocking on HR images.
  • Lightweight Models for Allocation: Using a lightweight SR model for scaling rate allocation in the Downscaling Module offers a significant computational advantage without substantial quality degradation.

Key Results

  • PSNR Improvement: BBMR achieves significant PSNR improvements, ranging from 1.06 dB to 1.96 dB on the Test2K and Test4K datasets compared to original methods. For instance, with the neural network downscaling method, PSNR can be enhanced by up to 1.97 dB.
  • Visual Quality: Qualitative comparisons show that BBMR produces clearer images, especially for primary subjects, by assigning them lower scaling rates. Blocking artifacts are significantly reduced, particularly when using the JointSR method with its deblock branch.
  • Computational Efficiency: While the Upscaling Module sees a negligible increase in computational load (around 3%), the use of a lightweight model for scaling rate allocation in the Downscaling Module can reduce computation by approximately three-quarters.

Limitations and Future Work

  • Computational Overhead: While BBMR improves quality, it does introduce some computational overhead, particularly in the Downscaling Module due to block processing and scaling rate allocation. The paper suggests using lightweight SR models to mitigate this.
  • Block Size Optimization: The paper uses a fixed block size (128x128). Future work could explore adaptive block sizes or different partitioning strategies.
  • Generalizability of Allocation Strategy: While the paper demonstrates the effectiveness of the block-based multi-scale strategy, further research could explore more sophisticated methods for determining block difficulty and allocating scaling rates.
  • Real-time Applications: While the framework aims to improve efficiency, achieving real-time performance for very high-resolution images might still be a challenge and a direction for future work.

Summary updated

BibTeX
@inproceedings{li2025block,
  author = {Jian Li and Siwang Zhou},
  title = {{Block-Based Multi-Scale Image Rescaling}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i5.32503},
}
Figure