DCVQE: A Hierarchical Transformer for Video Quality Assessment

Zutong Li, Lei Yang

Introduction

The proliferation of User-Generated Content (UGC) on platforms like TikTok, YouTube, and Weibo has created a massive demand for automated quality assessment. Unlike professional cinematography, UGC is often plagued by "in-the-wild" distortions, including camera shake, poor lighting, aggressive compression, and amateur editing. Assessing these videos without a pristine reference—known as No-Reference Video Quality Assessment (NR-VQA)—is a significant challenge for computer vision.

Existing NR-VQA methods often suffer from two primary limitations. First, many models exhibit "temporal blindness," treating videos as a sequence of static images and failing to capture how motion affects human perception. Second, models frequently struggle with the "middle quality" problem; while they can easily identify high-definition masterpieces or low-quality noise, they fail to distinguish between videos that are both moderately distorted. To address these issues, the DCVQE (Divide and Conquer Video Quality Estimator) introduces a hierarchical Transformer architecture that mimics human observation patterns and utilizes a novel ranking-based loss function.

Methodology

The Divide and Conquer Transformer (DCTr)

The core of the DCVQE is the Divide and Conquer Transformer (DCTr) layer. This architecture is inspired by the way human annotators evaluate video: alternating between a "global" impression of the entire clip and "local" reviews of specific segments. The process is broken down into three stages:

  1. Feature Extraction: The model utilizes a ResNet-50 backbone. To ensure the features are sensitive to quality-specific distortions rather than just semantic content, the backbone is first fine-tuned on Image Quality Assessment (IQA) tasks.
  2. The Divide Process: The video sequence is partitioned into several clips. A dedicated Transformer module, TransformerD, processes these clips to learn clip-level quality embeddings while simultaneously updating individual frame embeddings. This localized attention allows the model to capture short-term temporal dynamics.
  3. The Conquer Process: A second Transformer, TransformerC, aggregates the clip-level embeddings. By looking across the entire sequence of clips, it generates a final, video-level quality score that reflects the global viewing experience.

Correlation Loss ($L_c$)

Standard VQA models typically rely on L1 or L2 loss functions, which measure the absolute distance between predicted scores and ground-truth Mean Opinion Scores (MOS). However, MOS can be subjective and noisy. DCVQE introduces a Correlation Loss ($L_c$) that prioritizes the relative ranking of videos within a training batch. If the ground truth indicates that Video A is better than Video B, the model is penalized if it fails to maintain that order, even if the absolute predicted values are close to the target. This approach makes the training process more robust and better aligned with how humans perceive quality differences.

Computational Efficiency

By employing a hierarchical structure and sequence masks, DCVQE avoids the $O(N^2)$ complexity associated with standard Transformers. Instead, the complexity is reduced to $O(D \cdot N/C)$, where $N$ is the number of frames and $C$ is the number of clips. This makes the model significantly more efficient and scalable for real-world applications.

Experimental Results

The model was evaluated on four major "in-the-wild" datasets: KoNViD-1k, LIVE-VQC, YouTube-UGC, and LSVQ (the largest public UGC dataset with over 39,000 videos).

DCVQE consistently outperformed existing state-of-the-art (SOTA) methods, such as RAPIQUE and VIDEVAL. On the LSVQ dataset, it achieved a Spearman Rank-Order Correlation Coefficient (SRCC) of 0.836, surpassing the previous best of 0.827. On KoNViD-1k, it reached an SRCC of 0.8382, a substantial improvement over vanilla Transformer baselines (0.7988). These results demonstrate that the hierarchical approach is more stable and accurate across varying temporal ranges than traditional self-attention mechanisms.

Conclusion and Future Work

DCVQE represents a significant step forward in NR-VQA by combining a human-centric hierarchical architecture with a ranking-aware loss function. While the model currently caps input at 600 frames to manage GPU memory, it provides a robust framework for assessing UGC at scale.

Future research may explore deeper backbones or alternative feature extraction methods, as the authors noted a "complexity ceiling" where ResNet-101 performed worse than ResNet-50 in this specific task. Additionally, the hierarchical "Divide and Conquer" Transformer architecture holds promise for other video-based tasks, such as action recognition and automated video summarization.

Summary updated

BibTeX
@inproceedings{li2022dcvqe,
  author = {Zutong Li and Lei Yang},
  title = {{DCVQE: A Hierarchical Transformer for Video Quality Assessment}},
  booktitle = {ACCV},
  year = {2022},
  doi = {10.1007/978-3-031-26316-3\_24},
}
Figure