Perceptual Video Quality Prediction Emphasizing Chroma Distortions

Li-Heng Chen, Christos G. Bampis, Zhi Li, Joel Sole, Alan C. BovikIEEE TIP 2021 · OpenReview

TL;DR

This paper introduces a new dataset and an improved video quality prediction model (VMAFc) that explicitly accounts for chroma distortions, which are often overlooked by existing models. By considering both luma and chroma quality, they demonstrate potential for significant bitrate savings in video compression without sacrificing perceived quality.

What Problem Does This Paper Address?

Measuring and predicting the perceived quality of digital videos is crucial for applications like streaming and video conferencing. While many objective video quality assessment (VQA) models exist, they predominantly focus on the luminance (brightness) component of video, largely ignoring the color (chroma) information. This is problematic because chroma distortions, such as color bleeding or desaturation, can significantly impact perceived quality, and current models fail to capture these effects accurately. Furthermore, the decoupling of luma and chroma compression in modern codecs presents an opportunity for bitrate reduction that is missed by luma-only VQA models. The paper aims to bridge this gap by investigating the perceptual impact of chroma distortions and developing VQA models that incorporate this information.

Methods

The core of this work involves a two-pronged approach:

  1. Subjective Study and Dataset Creation: The authors designed and conducted a subjective experiment with 34 participants to gather perceptual quality scores for videos with varying levels of luma and chroma distortions. They introduced these distortions by manipulating quantization parameters in the HEVC video codec. Crucially, they developed a method to generate videos where luma and chroma distortions could be independently controlled, moving beyond the typical coupling in standard codecs. This resulted in the creation of the NFLX-Color (NFLXc) dataset, a benchmark specifically designed to study chroma distortions.
  2. Objective VQA Model Development: Leveraging the insights from their subjective study and the NFLXc dataset, the authors proposed an enhanced VQA model, termed VMAFc. This model builds upon the existing VMAF framework by incorporating new features derived from the chroma channels. They explored various chromatic features and found that multiscale Additive Distortion Metric (ADM) features performed particularly well. These features were then integrated into a Support Vector Regressor (SVR) to create the VMAFc model.

Datasets

  • NFLX-Color (NFLXc) Dataset: This is the primary new dataset introduced in the paper. It comprises 210 distorted videos derived from 21 pristine Full High Definition (FHD) source sequences. The distortions were generated by varying luma and chroma quantization parameters, allowing for independent control and a wide range of quality levels.
  • Existing VQA Datasets: For evaluation and comparison, the authors utilized several publicly available VQA datasets, including LIVE-VQA, LIVE-MBL, CSIQ-VQA, BVI-HD, VQEG, SHVC, EPFL, and VMAF+.

Key Insights and Ideas

  • Chroma Matters: Existing VQA models that ignore chroma information are fundamentally limited and can lead to inaccurate quality predictions, especially when chroma distortions are present.
  • Bitrate Savings Potential: By decoupling luma and chroma compression, there's a significant opportunity to reduce bitrate consumption. The paper demonstrates that aggressive chroma quantization can yield substantial bitrate reductions with minimal perceived quality loss, particularly at higher bitrates.
  • Multiscale Chroma Features: Human visual perception of color is more sensitive to lower frequencies. Multiscale ADM features, which capture these lower-frequency chromatic distortions, proved highly effective for predicting perceived quality.
  • Monotonicity is Key: An ideal VQA model should exhibit monotonicity – as distortion increases (e.g., chroma QP offset increases), the predicted quality score should decrease or stay the same. The proposed VMAFc model demonstrates this desirable property, unlike some baseline models.

Key Results

  • The newly developed VMAFc model achieved state-of-the-art performance on the NFLXc dataset, outperforming existing VQA models, including VMAF 0.6.1, in predicting subjective chroma distortion quality.
  • The paper demonstrates that by increasing the chroma QP offset, bitrate reductions of up to 36% were achieved in specific examples without significant perceived quality degradation.
  • The VMAFc model showed improved performance across various datasets compared to VMAF 0.6.1, particularly on datasets where chroma distortions were more prominent.
  • Statistical significance tests confirmed that VMAFc and iCID statistically surpassed most other models, with VMAFc showing strong overall performance.

Limitations and Future Work

  • Dataset Limitations: The NFLXc dataset, while comprehensive for chroma distortions, is still limited by the number of subjects and distortion types. It does not include distortions like scaling artifacts or simulated transmission errors, which are common in real-world scenarios.
  • Real-world Compression: The study focused on HEVC-compressed videos. Future work could explore other codecs and compression techniques.
  • Feature Exploration: While promising, the authors suggest further research into new chromatic features and more sophisticated machine learning models for even better chroma distortion prediction.
  • Joint Optimization: The ultimate goal is to use these insights to jointly optimize luma and chroma compression in video encoders, leading to more efficient and perceptually pleasing video delivery. The paper lays the groundwork for this by highlighting the potential for bitrate savings through creative chroma compression.

Summary updated

Abstract

Measuring the quality of digital videos viewed by human observers has become a common practice in numerous multimedia applications, such as adaptive video streaming, quality monitoring, and other digital TV applications. Here we explore a significant, yet relatively unexplored problem: measuring perceptual quality on videos arising from both luma and chroma distortions from compression. Toward investigating this problem, it is important to understand the kinds of chroma distortions that arise, how they relate to luma compression distortions, and how they can affect perceived quality. We designed and carried out a subjective experiment to measure subjective video quality on both luma and chroma distortions, introduced both in isolation as well as together. Specifically, the new subjective dataset comprises a total of 210 videos afflicted by distortions caused by varying levels of luma quantization commingled with different amounts of chroma quantization. The subjective scores were evaluated by 34 subjects in a controlled environmental setting. Using the newly collected subjective data, we were able to demonstrate important shortcomings of existing video quality models, especially in regards to chroma distortions. Further, we designed an objective video quality model which builds on existing video quality algorithms, by considering the fidelity of chroma channels in a principled way. We also found that this quality analysis implies that there is room for reducing bitrate consumption in modern video codecs by creatively increasing the compression factor on chroma channels. We believe that this work will both encourage further research in this direction, as well as advance progress on the ultimate goal of jointly optimizing luma and chroma compression in modern video encoders.
BibTeX
@article{chen2021perceptual,
  author = {Li-Heng Chen and Christos G. Bampis and Zhi Li and Joel Sole and Alan C. Bovik},
  title = {{Perceptual Video Quality Prediction Emphasizing Chroma Distortions}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=53RYyJMGz-D},
}
Figure