TL;DR
HiVE-VQA achieves state-of-the-art performance in no-reference video quality assessment by leveraging a frozen V-JEPA foundation model. By fusing multi-depth features into a hierarchical spatio-temporal volume, it captures both low-level distortions and high-level semantics with high efficiency and resolution-invariant inference.
What Problem Does This Paper Address?
With the explosion of user-generated content (UGC) on social media, automated Video Quality Assessment (VQA) has become a critical bottleneck. Specifically, researchers focus on No-Reference (NR-VQA), where the system must judge a video's quality without having a "perfect" original version for comparison.
Existing state-of-the-art (SOTA) methods typically fall into two camps: architecturally complex multi-branch pipelines that are hard to scale, or single-branch models that require computationally intensive end-to-end fine-tuning. The authors of HiVE-VQA ask a provocative question: Can we build a superior VQA system using a single, frozen video foundation model?
Methods
The core of the paper is the HiVE (Hierarchical Interaction via Volumetric Embedding) framework. Instead of training a massive model from scratch, the authors repurpose a pre-trained V-JEPA 2.1 encoder (a self-supervised video foundation model) and keep it entirely frozen.
The process follows three main stages:
- Fragment Sampling: To handle high-resolution videos efficiently, the model uses a sampling strategy that extracts small "fragments" from the video. This preserves local textures and distortions while making the computational cost invariant to the original video's resolution.
- Hierarchical Feature Extraction: Rather than just using the final output of the encoder, HiVE-VQA extracts token grids from four different depths of the transformer. Shallow layers capture low-level cues like blur and noise, while deeper layers provide semantic and temporal context.
- Volumetric Quality Prediction: These multi-depth features are stacked into a 4D "Hierarchical Feature Volume." A lightweight 3D convolutional head (only 5.2M parameters) then fuses this volume across its layer, spatial, and temporal axes to predict a final quality score.
Datasets
The model was trained exclusively on the LSVQ (train) dataset. Evaluation was conducted across several major UGC benchmarks to test both intra-dataset and cross-dataset generalization:
- Intra-dataset: LSVQ_test and LSVQ_1080p.
- Cross-dataset: KoNViD-1k and LIVE-VQC.
- Fine-tuning: YouTube-UGC.
Key Insights and Ideas
- Frozen Backbones Work: The most significant "aha moment" is that a frozen foundation model, if properly "probed" at multiple depths, provides a stronger representation for quality assessment than task-specific fine-tuned models.
- JEPA > Standard ViT: The authors found that features from a JEPA-based (Joint-Embedding Predictive Architecture) video encoder are significantly more sensitive to distortions than those from a standard image-pretrained Vision Transformer (ViT).
- The Power of Depth: Quality isn't just about pixels. By fusing features from different depths, the model can distinguish between a high-quality video of a blurry subject and a low-quality video with motion blur.
Key Results
HiVE-VQA outperformed 17 SOTA methods, including both single-branch and complex multi-branch models, without requiring any fine-tuning on the test datasets.
- Accuracy: It achieved a mean SROCC of 0.877 and PLCC of 0.890 across the four main test sets, representing a ~2.5% improvement over the best multi-branch models.
- Efficiency: Despite its high accuracy, the model is highly efficient. It is roughly 11x to 29x less computationally expensive than older deep learning models like VSFA or BVQA. At 1080p, it can process a video in under a quarter of a second on a modern GPU.
Limitations and Future Work
While the trainable parameters are minimal (5.2M), the total parameter count including the frozen backbone is still relatively large (~91M). The authors acknowledge this and suggest that future work could involve knowledge distillation or model pruning to compress the backbone for deployment on mobile or edge devices.
Additionally, while the frozen backbone generalizes exceptionally well, it may not perfectly capture the specific nuances of very niche datasets compared to models that are fully fine-tuned on those specific distributions. Future iterations might explore "parameter-efficient fine-tuning" (PEFT) techniques like LoRA to bridge this gap without losing the efficiency of the frozen approach.
Summary updated