Convolutional Neural Networks for Video Quality Assessment

Michalis Giannopoulos, Grigorios Tsagkatakis, Saverio G. Blasi, Farzad Toutounchi, Athanasios Mouchtaris, Panagiotis Tsakalides, Marta Mrak, Ebroul IzquierdoarXiv 2018 · OpenReview

Abstract

Predicting human perception of video quality under compression and network instability is a complex challenge. This study proposes a No-Reference (NR) Video Quality Assessment (VQA) framework that leverages 3D Convolutional Neural Networks (3D-CNNs) to extract spatial and temporal features directly from raw video data. By integrating network parameters into the preprocessing stage, the model achieves significant accuracy in classifying perceived quality levels for user-generated content.

Introduction

Assessing video quality is inherently subjective. While traditional metrics like Peak Signal-to-Noise Ratio (PSNR) are computationally efficient, they often fail to correlate with human visual perception, particularly for modern standards like HEVC (H.265). In these cases, compression artifacts are unpredictable, and mobile streaming introduces further degradation through network congestion, latency, and packet loss.

Most existing VQA methods are "Full-Reference," requiring the original uncompressed video for comparison. However, in real-world streaming scenarios, the receiver only has access to the degraded signal. This paper addresses the "No-Reference" (NR) challenge: predicting quality using only the degraded video and known network conditions.

Methodology

The core of the proposed system is a 3D Convolutional Neural Network (3D-CNN) designed to capture both spatial and temporal features. Unlike 2D-CNNs, 3D kernels extract features across consecutive frames, which is essential for identifying motion-related artifacts.

Proposed NR-VQA framework architecture.

Key technical components include:

  • Network-Aware Input: The model estimates throughput ($R$) based on Round-Trip Time (RTT) and packet loss. A derived "delay" ($\delta$) is manually appended to the start of the raw pixel data, embedding network performance directly into the visual signal.
  • Patch-Based Learning: To manage the memory constraints of high-resolution video ($1920 \times 1080$), the input is divided into $16 \times 16 \times 16$ cubic patches.
  • Aggregation Strategies: Patch-level predictions are aggregated into a final video-level score using either Majority Voting or a Pre-trained 1D-CNN.

Experimental Evaluation

Due to the lack of suitable existing datasets, the authors developed a custom VQA dataset using three User-Generated Content (UGC) sequences from the Edinburgh Festival.

Sample frames from the Edinburgh Festival UGC dataset.

The videos were encoded using the Turing HEVC codec at five different Quantisation Parameters (QP 18 to 40) and subjected to four simulated network conditions (RTT: 5ms–50ms; loss: 0.1%–3%). Ground truth was established via Mean Opinion Scores (MOS) from eight human subjects, which were discretized into quality classes.

Results and Discussion

The end-to-end learning approach outperformed traditional methods relying on hand-crafted features. Key findings include:

  • Classification Accuracy: The system achieved 80% accuracy in classifying videos into three quality categories (Low, Mid, High). Individual patches were classified with 74% to 77% accuracy.
  • Aggregation Performance: Interestingly, the simpler Majority Voting strategy slightly outperformed the 1D-CNN approach in the 3-class scenario.
  • Optimization: The Adagrad optimizer proved more stable than standard SGD, likely due to its ability to handle sparse gradients in patch-based data.
  • Perceptual Sensitivity: The model was most accurate at the extremes of the quality spectrum ("Excellent" or "Bad"), mirroring human perceptual tendencies.

Conclusion

This research demonstrates the efficacy of 3D-CNNs for NR-VQA, particularly when augmented with network-layer information. Future work will focus on expanding the dataset diversity, transitioning from classification to continuous regression scores, and incorporating chrominance channels to account for color distortions.

Summary updated

Abstract

Video Quality Assessment (VQA) is a very challenging task due to its highly subjective nature. Moreover, many factors influence VQA. Compression of video content, while necessary for minimising transmission and storage requirements, introduces distortions which can have detrimental effects on the perceived quality. Especially when dealing with modern video coding standards, it is extremely difficult to model the effects of compression due to the unpredictability of encoding on different content types. Moreover, transmission also introduces delays and other distortion types which affect the perceived quality. Therefore, it would be highly beneficial to accurately predict the perceived quality of video to be distributed over modern content distribution platforms, so that specific actions could be undertaken to maximise the Quality of Experience (QoE) of the users. Traditional VQA techniques based on feature extraction and modelling may not be sufficiently accurate. In this paper, a novel Deep Learning (DL) framework is introduced for effectively predicting VQA of video content delivery mechanisms based on end-to-end feature learning. The proposed framework is based on Convolutional Neural Networks, taking into account compression distortion as well as transmission delays. Training and evaluation of the proposed framework are performed on a user annotated VQA dataset specifically created to undertake this work. The experiments show that the proposed methods can lead to high accuracy of the quality estimation, showcasing the potential of using DL in complex VQA scenarios.
BibTeX
@misc{giannopoulos2018convolutional,
  author = {Michalis Giannopoulos and Grigorios Tsagkatakis and Saverio G. Blasi and Farzad Toutounchi and Athanasios Mouchtaris and Panagiotis Tsakalides and Marta Mrak and Ebroul Izquierdo},
  title = {{Convolutional Neural Networks for Video Quality Assessment}},
  year = {2018},
  eprint = {1809.10117},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1809.10117},
}
Figure