<title>Color image quality on the Internet</title>

Sabine Süsstrunk, Stefan WinklerProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2003 · DOI

TL;DR

This paper tackles the challenge of assessing color image and video quality on the internet, focusing on artifacts introduced by compression and transmission errors. The authors propose and evaluate no-reference metrics for blockiness, blurriness, and colorfulness, demonstrating their strong correlation with human perception and their effectiveness in predicting video quality in real-world streaming scenarios.

What Problem Does This Paper Address?

Delivering high-quality images and videos over the internet is a complex task. Images and videos are often compressed to reduce data size for storage and transmission, and during this process, or during transmission itself (due to packet loss or bit errors), visual artifacts can be introduced. These artifacts degrade the perceived quality of the content. Traditional methods for assessing image quality, like Mean Squared Error (MSE) or Peak Signal-to-Noise Ratio (PSNR), often fail to correlate well with human perception and require an original, uncompressed reference image, which is not always available, especially in streaming applications. This paper addresses the need for reliable, no-reference quality assessment methods that can predict perceived quality directly from the distorted image or video, mimicking how a human observer would judge its quality.

Methods

The paper focuses on developing and evaluating no-reference (NR) artifact metrics. Unlike full-reference metrics that compare a distorted image to an original, NR metrics analyze the distorted image alone. The authors propose specific NR metrics for:

  • Blockiness: This metric leverages the Fourier domain to detect the characteristic grid pattern of block-based compression artifacts. It analyzes the power spectrum of horizontal and vertical differences to identify peaks indicative of block structures.
  • Blurriness: This metric assumes that significant edges in an image should be sharp. It uses an edge detector (like Sobel) to find edges and then measures the distance between local intensity extrema around these edges, quantifying the degree of smearing.
  • Colorfulness: While not directly used for quality prediction in this paper, a metric for colorfulness is presented. It's based on the mean and standard deviations of color components in an opponent color space and correlates highly with perceived colorfulness.

These individual artifact metrics are then combined to predict overall image and video quality. The paper also discusses two general approaches to quality assessment: the "psychophysical approach" (modeling the human visual system) and the "engineering approach" (making assumptions about specific compression artifacts). The authors lean towards the latter for their NR metrics.

Datasets

The paper utilizes the following datasets for evaluation:

  • LIVE Image Quality Assessment Database: This database contains JPEG2000-compressed versions of 29 color images with varying compression ratios. Subjective experiments were conducted on this dataset to obtain Mean Opinion Scores (MOS).
  • Custom datasets for video quality assessment: The authors created specific test sequences for video streaming over packet networks and wireless channels. These sequences were compressed using various codecs (WMV8, Real Video 8, MPEG-4, Motion JPEG2000) under different network conditions (packet loss, bit errors). Subjective experiments using the Single Stimulus Continuous Quality Evaluation (SSCQE) method were performed on these sequences.

Key Insights and Ideas

  • No-reference metrics are crucial for internet applications: The inability to access original reference images in streaming scenarios makes NR metrics indispensable.
  • Artifact-specific metrics are effective: By focusing on specific, common artifacts like blockiness and blurriness, NR metrics can achieve high prediction accuracy.
  • Human perception is complex and context-dependent: The paper highlights that viewing conditions (ambient light, monitor characteristics) significantly influence perceived quality, and metrics need to be evaluated under relevant conditions.
  • "Visually lossless" is a moving target: Compression standards aim for visually lossless compression by suppressing high-frequency components, but what is "invisible" depends heavily on the output device and viewing conditions.
  • Simple metrics can be powerful: The proposed NR metrics are computationally light, making them suitable for real-time applications.
  • Outliers reveal metric limitations: The blurriness metric's performance was affected by images with natural shallow depth of field, demonstrating that NR metrics can struggle when artifacts are indistinguishable from content characteristics.

Key Results

  • The proposed NR blurriness metric achieved correlations of around 85% with MOS on the LIVE Image Quality Assessment Database (after removing specific outlier images, it reached 90%).
  • For video streaming over packet networks, the NR metrics achieved an overall 78% linear correlation with MOS, outperforming what would be expected from simple fidelity measures.
  • In video over wireless simulations, the NR metrics showed even stronger performance, with an overall 93% linear correlation with MOS, significantly outperforming PSNR (around 39%).
  • The paper demonstrates that NR metrics can be effectively "tuned" to specific codecs and transmission conditions to improve prediction accuracy.

Limitations and Future Work

  • Distinguishing artifacts from content: A key limitation of NR metrics is their difficulty in distinguishing compression artifacts from natural image content (e.g., blurriness due to shallow depth of field). The authors suggest incorporating detectors for specific content types to mitigate this.
  • Temporal artifacts in video: While the paper mentions a jerkiness metric for video, the primary focus is on spatial metrics. Further work could explore more sophisticated temporal artifact detection.
  • Subjective experiment analysis: The authors acknowledge that analyzing time-series data from subjective experiments is complex and not fully addressed by existing standards, suggesting this as an area for future research.
  • Reduced-reference metrics: While NR metrics are versatile, the paper notes that incorporating some reference information (reduced-reference metrics) could further improve prediction accuracy, especially in scenarios where some metadata or features from the original are available.
  • Viewing conditions: The paper emphasizes the importance of viewing conditions but primarily discusses them in the context of evaluation. Future work could explore how to incorporate viewing condition awareness directly into quality prediction models.

Summary updated

Abstract

Color image quality depends on many factors, such as the initial capture system and its color image processing, compression, transmission, the output device, media and associated viewing conditions. In this paper, we are primarily concerned with color image quality in relation to compression and transmission. We review the typical visual artifacts that occur due to high compression ratios and/or transmission errors. We discuss color image quality metrics and present no-reference artifact metrics for blockiness, blurriness, and colorfulness. We show that these metrics are highly correlated with experimental data collected through subjective experiments. We use them for no-reference video quality assessment in different compression and transmission scenarios and again obtain very good results. We conclude by discussing the important effects viewing conditions can have on image quality.
BibTeX
@inproceedings{susstrunk2003lt,
  author = {Sabine Süsstrunk and Stefan Winkler},
  title = {{\<title\>Color image quality on the Internet\</title\>}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2003},
  doi = {10.1117/12.537804},
}
Figure