PEA265: Perceptual Assessment of Video Compression Artifacts

Liqun Lin, Shiqi Yu, Liping Zhou, Weiling Chen, Tiesong Zhao, Zhou Wang

TL;DR

The PEA265 paper introduces the first large-scale, subjectively labeled database specifically designed to identify and quantify six distinct types of video compression artifacts in H.265/HEVC streams. By training deep learning models (specifically ResNeXt) on over 360,000 image and video patches, the researchers achieved over 80% accuracy in automatically detecting these "Perceivable Encoding Artifacts" (PEAs), paving the way for more perceptually-aware video encoders.

What Problem Does This Paper Address?

Despite the high efficiency of modern codecs like H.265/HEVC, compressed videos often suffer from visual distortions that frustrate viewers. Researchers categorize these as Perceivable Encoding Artifacts (PEAs).

The industry currently relies on objective metrics like PSNR (Peak Signal-to-Noise Ratio) or SSIM (Structural Similarity). While useful, these are "blind" to the type of distortion occurring. They might tell you a video is "low quality," but they can't distinguish between a blurry texture and a blocky edge. Furthermore, these metrics often correlate poorly with actual human Quality-of-Experience (QoE). To build better encoders, we need systems that can identify specific artifacts—like ringing or flickering—and quantify their severity just as a human viewer would.

Methods

The authors approached this as a supervised classification problem. Their methodology involved three main pillars:

  1. Subjective Labeling: They recruited 30 participants to watch 320 encoded sequences and manually circle regions containing specific artifacts.
  2. CNN-Based Recognition: They compared two Convolutional Neural Network (CNN) architectures—the classic LeNet-5 and the more modern ResNeXt. The goal was to determine if deep learning could learn the complex, high-dimensional features that define different PEAs.
  3. New Metrics: They proposed two new measures:
  4. PEA Pattern: A 6-bit binary vector (e.g., 111000) indicating which of the six artifact types are present in a specific patch.
  5. PEA Intensity: The percentage of "positive" artifact detections across a frame or sequence, providing a holistic view of degradation.

Datasets: The PEA265 Database

The core contribution is the PEA265 database. While other quality databases exist, they often focus on overall scores rather than specific artifact labeling.

  • Source: 320 video sequences compressed via HEVC with varying Quantization Parameters (QP 22, 27, 32, and 37).
  • Scale: Contains at least 60,000 patches (positive and negative labels) for each of the six artifact types.
  • Artifact Categories:
    • Spatial: Blurring, Blocking, Ringing, and Color Bleeding.
  • Temporal: Flickering and Floating (illusory movement in textured areas).

Key Insights and Ideas

  • The Architecture Matters: ResNeXt significantly outperformed LeNet-5. The authors attribute this to ResNeXt’s multi-branch architecture and bottleneck layers, which are better at capturing the subtle, complex textures of artifacts like "ringing" or "floating" compared to simpler stacked convolutions.
  • Artifacts are Content-Dependent: The research highlights that PEA intensity isn't just about the bitrate; it's highly dependent on the video content. For example, "SlideShow" sequences (static text/graphics) showed much lower blocking and blurring than high-motion sequences like "Kimono."
  • Quantization Correlation: As expected, artifact intensity generally increases with the Quantization Parameter (QP). However, the PEA265 data allows researchers to see which artifacts emerge first as the QP increases, providing a roadmap for "perceptual optimization" in encoders.

Key Results

  • High Detection Accuracy: The ResNeXt model achieved a testing accuracy of over 80% for all six PEA types.
  • Top Performer: The model was most effective at identifying Blocking artifacts, reaching 92.81% accuracy.
  • Efficiency: Despite being a deeper network, ResNeXt was faster to train than LeNet-5 (655 minutes vs 1966 minutes) due to its efficient use of bottleneck layers and group convolutions.
  • Benchmark Superiority: When compared to previous low-level feature extraction methods (like those used in [17] for floating artifacts), the proposed CNN approach showed much better generalization and higher accuracy on diverse test images.

Limitations and Future Work

The authors acknowledge that while they can now detect these artifacts with high accuracy, they haven't yet fully solved how to weight them. For instance, is a small amount of "ringing" more annoying to a viewer than a large amount of "blurring"?

Future directions include: * Developing a weighted formula that combines individual PEA detections into a single, unified QoE score. * Integrating these real-time PEA detectors directly into the "in-loop" filtering process of video encoders to prevent artifacts from forming in the first place. * Expanding the database to include newer codecs like VVC (Versatile Video Coding).

Summary updated

Abstract

The most widely used video encoders share a common hybrid coding framework that includes block-based motion estimation/compensation and block-based transform coding. Despite their high coding efficiency, the encoded videos often exhibit visually annoying artifacts, denoted as Perceivable Encoding Artifacts (PEAs), which significantly degrade the visual Quality-of-Experience (QoE) of end users. To monitor and improve visual QoE, it is crucial to develop subjective and objective measures that can identify and quantify various types of PEAs. In this work, we make the first attempt to build a large-scale subject-labeled database composed of H.265/HEVC compressed videos containing various PEAs. The database, namely the PEA265, includes 4 types of spatial PEAs (i.e. blurring, blocking, ringing and color bleeding) and 2 types of temporal PEAs (i.e. flickering and floating). Each containing at least 60,000 image or video patches with positive and negative labels. Based on the PEA265 database, we develop and optimize Convolutional Neural Networks (CNNs) to objectively recognize different types of PEAs. Experiments show that our architecture is capable of identifying the 6 types of PEAs with an accuracy over 86%. To further demonstrate its application, we explore the relationship between collected PEA intensities and subjective quality scores of compressed videos. A quality metric is consequently proposed with superior performance in terms of correlation to Mean Opinion Score (MOS) values. We believe that the PEA265 database and our findings will benefit the future development of video quality assessment methods and perceptually motivated video encoders.
BibTeX
@article{lin2020pea,
  author = {Liqun Lin and Shiqi Yu and Liping Zhou and Weiling Chen and Tiesong Zhao and Zhou Wang},
  title = {{PEA265: Perceptual Assessment of Video Compression Artifacts}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2020},
  url = {https://openreview.net/forum?id=3DhJWJOlNyp},
}
Figure