Deep Generative Model based Rate-Distortion for Image Downscaling Assessment

Yuanbang Liang, Bhavesh Garg, Paul L. Rosin, Yipeng Qin

Abstract

Evaluating image downscaling algorithms is notoriously difficult because there is no "ground truth" low-resolution image to compare against. This paper introduces IDA-RD, a process-based metric that treats downscaling as an encoding step and uses deep generative models (GANs and Normalizing Flows) as "blind" decoders to measure how much information was lost during the shrink.

Introduction

When we shrink an image (downscaling), we inevitably lose information. While many algorithms exist to do this—from simple bicubic interpolation to complex perceptual models—we lack a reliable quantitative way to rank them. Standard metrics like PSNR or SSIM require a reference image, but when testing a new downscaling method, the "correct" low-res reference is often unknown. Furthermore, "No-Reference" metrics (like NIQE) are designed to judge the "naturalness" of high-resolution images and fail when applied to the tiny, often aliased or blurred outputs of downscaling. This lack of a quantitative "yardstick" has significantly slowed research in the field.

Methodology: The IDA-RD Framework

The authors turn to Claude Shannon’s Rate-Distortion theory. They propose that the best way to measure the quality of a downscaling algorithm ($f_{ds}$) is to see how well a super-resolution (SR) algorithm ($f_{us}$) can reconstruct the original high-resolution (HR) image from the downscaled version.

The IDA-RD pipeline: Downscaling an HR image $X$, upscaling the result using a generative model, and measuring the perceptual distortion between the original and the reconstruction.

The core of their method, IDA-RD (Image Downscaling Assessment by Rate-Distortion), follows this pipeline:

  1. Downscale: Take an HR image ($X$) and shrink it using the method being tested ($f_{ds}$).
  2. Upscale: Use a deep generative model to upscale it back to the original size ($\hat{X}$).
  3. Measure: Calculate the distance (distortion) between the original $X$ and the reconstruction $\hat{X}$ using a perceptual metric like LPIPS.

To be a fair judge, the upscaler must be blind (it shouldn't know which downscaling method was used) and stochastic (it must be able to explore the "manifold" of all possible HR images that could have produced that specific low-res input). The authors implement this using two types of generative models: StyleGAN (via GAN inversion) and SRFlow (a Normalizing Flow model). SRFlow proved to be much faster and more practical for large-scale evaluation.

Experiments and Datasets

The primary evaluation was conducted on FFHQ (Flickr-Faces-HQ), using a balanced subset of 900 images. To prove generalization, the authors also tested on DIV2K & Flickr2K (general scenes), RealSR (hardware-captured images), and specialized sets like NPRportrait 1.0 and AFHQ-Cat.

Key Results

  • Monotonicity: In synthetic tests, IDA-RD scores perfectly tracked with degradation levels. As more blur or noise was added to the downscaling process, the IDA-RD distortion score increased accordingly.
  • Ranking Accuracy: The metric correctly ranked traditional methods (Bilinear > Bicubic > Nearest Neighbor) across multiple datasets.
  • Real-world Validation: When tested on the RealSR dataset, IDA-RD correctly identified that hardware-captured low-res images often preserve more information than algorithmic downscaling.
  • The SOTA Paradox: The authors found that some "State-of-the-Art" (SOTA) downscalers designed for human visual appeal actually discard more underlying information than simpler methods like bilinear interpolation. IDA-RD is the first metric to quantitatively expose this trade-off.

Discussion and Limitations

IDA-RD shifts the paradigm from image-based to process-based evaluation. However, the accuracy of the metric is tied to the generative model used; the upscaler must "know" the domain of the images being tested (e.g., a face-trained model cannot evaluate landscape downscaling). Currently, IDA-RD still requires the original high-resolution image to calculate distortion. Future work aims to move toward "meta-measures" that can quantify performance even when the original high-resolution source is unavailable.

Summary updated

BibTeX
@inproceedings{liang2024deep,
  author = {Yuanbang Liang and Bhavesh Garg and Paul L. Rosin and Yipeng Qin},
  title = {{Deep Generative Model based Rate-Distortion for Image Downscaling Assessment}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01832},
}
Figure