Image Quality Assessment for Realistic Zoom Photos

Zongxi Han, Yutao Liu, Rong Xie, Guangtao Zhai

TL;DR

This paper introduces a novel no-reference image quality assessment (IQA) metric specifically designed for realistic zoom photos from smartphones. By combining a new sharpness metric derived from free-energy theory with a naturalness measure based on natural scene statistics, their approach significantly outperforms existing methods in predicting human perception of zoom photo quality.

What Problem Does This Paper Address?

Modern smartphones, with their advanced CMOS imaging sensors, have made user-generated content ubiquitous. Zoom photos, in particular, are a popular category. However, the small sensor sizes and digital zoom capabilities of smartphones often lead to image degradation, such as graininess, zigzag textures, and over-sharpening artifacts. Traditional IQA metrics, which are often designed for simpler distortions like Gaussian blur, tend to overestimate the quality of these realistically distorted zoom photos. This paper aims to bridge this gap by developing an IQA metric that accurately reflects human perception of zoom photo quality, accounting for the unique artifacts introduced by smartphone zoom technology.

Methods

The core of this work lies in a novel no-reference IQA metric that combines two key components: sharpness and naturalness.

  • Sharpness Measurement: The authors propose a new sharpness metric rooted in the free-energy principle. This principle, inspired by brain perception, models an image as a combination of a "predicted" and a "residual" component. They leverage sparse representation to model the brain's generative process. The sharpness is then quantified by combining the total energy of the predicted gradient image (related to acutance) with the entropy of the residual term (related to resolution). This is a novel application of free-energy theory to IQA, specifically for sharpness estimation.
  • Naturalness Measurement: To address artifacts like over-sharpening and unnatural textures, the paper incorporates a naturalness component. This is achieved by analyzing Mean Subtracted Contrast Normalized (MSCN) coefficients and their products. These coefficients are derived from natural scene statistics (NSS) and are known to capture deviations from natural image distributions. By comparing the NSS features of a test image to those of pristine images, they derive an "unnaturalness" score.

These two components are then linearly combined, with a learned weighting factor, to produce the final zoom quality score.

Datasets

The authors introduce a new benchmark dataset called the Zoom PHoto Database (ZPHD).

  • ZPHD: This dataset comprises 900 tele-photos captured from 20 different mobile cameras across 45 diverse, texture-complex scenes. The photos were taken without external lenses or post-processing, and each scene was captured by all cameras to ensure comparability. This dataset is notable for its focus on realistic zoom distortions, including those arising from digital zoom and post-processing, and its high resolution (mostly 4000x3000 pixels). It aims to be more representative of real-world zoom photo quality issues than existing datasets that often rely on simulated distortions like Gaussian blur.

Key Insights and Ideas

  • Free-Energy for Sharpness: Applying the free-energy principle, which models how the brain interprets visual information, to quantify image sharpness is a novel approach. The decomposition into predicted and residual components, and the subsequent analysis of their energy and entropy, provides a more nuanced understanding of sharpness than traditional methods.
  • Sharpness vs. Naturalness Trade-off: The paper highlights that sharpness alone is insufficient for zoom photos. Over-sharpening, while increasing sharpness scores, degrades perceptual quality. Conversely, focusing solely on naturalness might penalize necessary sharpening that improves perceived detail. A balanced combination is crucial.
  • MSCN Coefficients as Artifact Detectors: The use of MSCN coefficients and their products proves effective in capturing the subtle artifacts introduced by smartphone image processing, such as over-sharpening and unnatural textures.
  • Realistic Distortions Matter: The creation of the ZPHD dataset underscores the need for IQA research to move beyond simulated distortions and address the complex artifacts present in real-world images, particularly from mobile devices.

Key Results

  • The proposed zoom quality metric achieved SROCC (Spearman Rank Order Correlation Coefficient) and PLCC (Pearson Linear Correlation Coefficient) scores over 0.91 on the ZPHD database.
  • Individual sharpness or naturalness components achieved scores around 0.85, demonstrating the benefit of their combined approach.
  • The proposed metric outperformed the best general-purpose and sharpness models by 0.072 and 0.064 in SROCC, respectively.
  • Qualitative results in Figure 8 demonstrate the metric's ability to correctly rank image quality in scenarios where individual sharpness or naturalness metrics fail.

Limitations and Future Work

  • Color and Exposure: The current metric does not explicitly consider color differences or exposure variations, which can also impact overall image quality.
  • Content-Dependent Weighting: The weighting factor w for combining sharpness and naturalness is currently a constant. The authors suggest that this could be generalized to a function w(c, q) that adapts based on image content (e.g., characters vs. textures) and quality level.
  • Machine Learning Integration: The authors propose exploring machine learning approaches to further improve the metric's performance.
  • AI-Generated Textures: The metric currently struggles to effectively handle quality degradation from AI-restoration techniques used in advanced zoom photos, which can introduce fake, unnatural textures. This is identified as an area for future research.

Summary updated

Abstract

New CMOS imaging sensor (CIS) techniques in smartphones have helped user-generated content dominate our lives over traditional DSLRs. However, tiny sensor sizes and fixed focal lengths also lead to more grainy details, especially for zoom photos. Moreover, multi-frame stacking and post-sharpening algorithms would produce zigzag textures and over-sharpened appearances, for which traditional image-quality metrics may over-estimate. To solve this problem, a real-world zoom photo database is first constructed in this paper, which includes 900 tele-photos from 20 different mobile sensors and ISPs. Then we propose a novel no-reference zoom quality metric which incorporates the traditional estimation of sharpness and the concept of image naturalness. More specifically, for the measurement of image sharpness, we are the first to combine the total energy of the predicted gradient image with the entropy of the residual term under the framework of free-energy theory. To further compensate for the influence of over-sharpening effect and other artifacts, a set of model parameters of mean subtracted contrast normalized (MSCN) coefficients are utilized as the natural statistics representatives. Finally, these two measures are combined linearly. Experimental results on the zoom photo database demonstrate that our quality metric can achieve SROCC and PLCC over 0.91, while the performance of single sharpness or naturalness index is around 0.85. Moreover, compared with the best tested general-purpose and sharpness models, our zoom metric outperforms them by 0.072 and 0.064 in SROCC, respectively.
BibTeX
@inproceedings{han2023image,
  author = {Zongxi Han and Yutao Liu and Rong Xie and Guangtao Zhai},
  title = {{Image Quality Assessment for Realistic Zoom Photos}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=ISVlW3F4o2},
}
Figure