Analysis of Video Quality Datasets via Design of Minimalistic Video Quality Models

Wei Sun, Wen Wen, Xiongkuo Min, Long Lan, Guangtao Zhai, Kede Ma

TL;DR

This paper introduces a novel approach to analyzing existing video quality assessment (VQA) datasets by designing "minimalistic" VQA models. The authors reveal that many current VQA datasets suffer from an "easy dataset problem," meaning they are too simple to adequately challenge VQA models and may even be solvable by image quality assessment techniques. This work highlights the need for more robust VQA dataset construction and model design for reliable progress in the field.

What Problem Does This Paper Address?

Blind Video Quality Assessment (BVQA) is crucial for ensuring good user experience in the ever-growing world of video content. However, the progress in developing effective BVQA models has largely been measured on a limited number of human-rated VQA datasets. This raises a critical question: are these datasets truly challenging enough to drive meaningful advancements in BVQA, or are they inadvertently creating a false sense of progress? The authors aim to address this by systematically analyzing the inherent difficulty and characteristics of existing VQA datasets, identifying potential flaws in their design that might hinder the development of truly robust BVQA models.

Methods

The core of this paper lies in the design of "minimalistic" BVQA models. These models are intentionally kept simple, built upon four fundamental building blocks:

  • Video Preprocessor: Aggressively downsamples videos spatially and temporally to extract key frames.
  • Spatial Quality Analyzer: Acts as a blind image quality assessment (BIQA) model to extract spatial features from key frames.
  • Temporal Quality Analyzer (Optional): Extracts temporal features from video chunks around key frames.
  • Quality Regressor: Combines spatial and temporal features to predict a global video quality score.

By using these basic building blocks with various instantiations and training them on existing datasets, the authors can effectively probe the datasets' difficulty. They also explore different initializations (e.g., pre-trained on ImageNet, IQA datasets, or VQA datasets) and network architectures (e.g., ResNet, Swin Transformer) for the spatial quality analyzer.

Datasets

The study evaluates its minimalistic BVQA models on eight widely used VQA datasets:

  • CVD2014
  • LIVE-Qualcomm
  • KoNViD-1k
  • LIVE-VQC
  • YouTube-UGC
  • LBVD
  • LSVQ
  • LIVE-YT-Gaming

The paper does not introduce new datasets but focuses on analyzing the existing ones.

Key Insights and Ideas

  • The "Easy Dataset Problem": The most significant finding is that many VQA datasets are too easy, allowing even simplified models (or BIQA models) to achieve high performance. This suggests that the datasets may not be effectively capturing the complexities of real-world video distortions.
  • Spatial Dominance: In many datasets, spatial distortions are so dominant that temporal distortions play a minimal role in quality prediction. This means that models primarily focused on spatial quality can perform well, undermining the need for sophisticated temporal analysis.
  • Dataset Diversity vs. Difficulty: While dataset diversity (in terms of content and distortion types) is important for generalization, it doesn't automatically guarantee difficulty. Datasets need to be challenging enough to differentiate between models with varying levels of complexity.
  • Importance of Initialization: For the spatial quality analyzer, pre-training on relevant datasets (IQA or VQA) significantly boosts performance, often more than architectural complexity or the addition of temporal features.
  • Temporal Analysis Limitations: The authors' minimalistic temporal analyzer, while simple, often provides little to no improvement, further supporting the idea that many datasets don't necessitate complex temporal modeling.

Key Results

  • The paper empirically ranks the eight VQA datasets based on the severity of the "easy dataset problem," from least to most severe: LBVD, LIVE-VQC, LIVE-Qualcomm, KoNViD-1k, YouTube-UGC, LSVQ, LIVE-YT-Gaming, and CVD2014.
  • Model (I) (ResNet-50 baseline with no temporal analyzer) achieved highly competitive results on CVD2014, indicating it's solvable by BIQA-like approaches.
  • For datasets like LBVD and LIVE-VQC, the temporal quality analyzer did bring noticeable improvements, suggesting these datasets are more challenging in terms of temporal distortions.
  • Ablation studies showed that while advanced backbones like Swin Transformer improved performance, the impact of quality-aware initializations was often more pronounced.

Limitations and Future Work

  • Minimalistic Temporal Analyzer: The authors acknowledge that their temporal quality analyzer is intentionally simple and does not model long-term temporal dependencies. This suggests that more sophisticated temporal modeling might be beneficial for datasets that truly require it.
  • Dataset Construction: The paper strongly advocates for the construction of next-generation VQA datasets that are more challenging. This involves carefully considering both sample selection (to ensure diversity and difficulty) and subjective testing methodologies (e.g., exploring continuous quality rating or forced-choice tasks to emphasize temporal aspects).
  • Model Generalization: While the study focuses on analyzing datasets, future work could explore how to build models that generalize better across diverse datasets and distortion types.
  • Computational Cost: The paper notes that while higher resolutions might improve performance, they are computationally prohibitive for end-to-end training, highlighting a trade-off between performance and efficiency.

Summary updated

Abstract

Blind video quality assessment (BVQA) plays an indispensable role in monitoring and improving the end-users' viewing experience in various real-world video-enabled media applications. As an experimental field, the improvements of BVQA models have been measured primarily on a few human-rated VQA datasets. Thus, it is crucial to gain a better understanding of existing VQA datasets in order to properly evaluate the current progress in BVQA. Towards this goal, we conduct a first-of-its-kind computational analysis of VQA datasets via designing minimalistic BVQA models. By minimalistic, we restrict our family of BVQA models to build only upon basic blocks: a video preprocessor (for aggressive spatiotemporal downsampling), a spatial quality analyzer, an optional temporal quality analyzer, and a quality regressor, all with the simplest possible instantiations. By comparing the quality prediction performance of different model variants on eight VQA datasets with realistic distortions, we find that nearly all datasets suffer from the easy dataset problem of varying severity, some of which even admit blind image quality assessment (BIQA) solutions. We additionally justify our claims by contrasting our model generalizability on these VQA datasets, and by ablating a dizzying set of BVQA design choices related to the basic building blocks. Our results cast doubt on the current progress in BVQA, and meanwhile shed light on good practices of constructing next-generation VQA datasets and models.
BibTeX
@article{sun2023analysis,
  author = {Wei Sun and Wen Wen and Xiongkuo Min and Long Lan and Guangtao Zhai and Kede Ma},
  title = {{Analysis of Video Quality Datasets via Design of Minimalistic Video Quality Models}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  doi = {10.1109/tpami.2024.3385364},
}
Figure