HiVE-VQA: Hierarchical Interaction via Volumetric Embedding for Video Quality Assessment

Robbie Hamilton, Fan Zhang, David BullarXiv 2026 · arXiv

TL;DR

HiVE-VQA achieves state-of-the-art performance in no-reference video quality assessment by leveraging a frozen V-JEPA foundation model. By fusing multi-depth features into a hierarchical spatio-temporal volume, it captures both low-level distortions and high-level semantics with high efficiency and resolution-invariant inference.

What Problem Does This Paper Address?

With the explosion of user-generated content (UGC) on social media, automated Video Quality Assessment (VQA) has become a critical bottleneck. Specifically, researchers focus on No-Reference (NR-VQA), where the system must judge a video's quality without having a "perfect" original version for comparison.

Existing state-of-the-art (SOTA) methods typically fall into two camps: architecturally complex multi-branch pipelines that are hard to scale, or single-branch models that require computationally intensive end-to-end fine-tuning. The authors of HiVE-VQA ask a provocative question: Can we build a superior VQA system using a single, frozen video foundation model?

Methods

The core of the paper is the HiVE (Hierarchical Interaction via Volumetric Embedding) framework. Instead of training a massive model from scratch, the authors repurpose a pre-trained V-JEPA 2.1 encoder (a self-supervised video foundation model) and keep it entirely frozen.

The process follows three main stages:

  1. Fragment Sampling: To handle high-resolution videos efficiently, the model uses a sampling strategy that extracts small "fragments" from the video. This preserves local textures and distortions while making the computational cost invariant to the original video's resolution.
  2. Hierarchical Feature Extraction: Rather than just using the final output of the encoder, HiVE-VQA extracts token grids from four different depths of the transformer. Shallow layers capture low-level cues like blur and noise, while deeper layers provide semantic and temporal context.
  3. Volumetric Quality Prediction: These multi-depth features are stacked into a 4D "Hierarchical Feature Volume." A lightweight 3D convolutional head (only 5.2M parameters) then fuses this volume across its layer, spatial, and temporal axes to predict a final quality score.
Figure 2. The framework of the proposed HiVE-VQA method.

Datasets

The model was trained exclusively on the LSVQ (train) dataset. Evaluation was conducted across several major UGC benchmarks to test both intra-dataset and cross-dataset generalization:

  • Intra-dataset: LSVQ_test and LSVQ_1080p.
  • Cross-dataset: KoNViD-1k and LIVE-VQC.
  • Fine-tuning: YouTube-UGC.

Key Insights and Ideas

  • Frozen Backbones Work: The most significant "aha moment" is that a frozen foundation model, if properly "probed" at multiple depths, provides a stronger representation for quality assessment than task-specific fine-tuned models.
  • JEPA > Standard ViT: The authors found that features from a JEPA-based (Joint-Embedding Predictive Architecture) video encoder are significantly more sensitive to distortions than those from a standard image-pretrained Vision Transformer (ViT).
  • The Power of Depth: Quality isn't just about pixels. By fusing features from different depths, the model can distinguish between a high-quality video of a blurry subject and a low-quality video with motion blur.
Figure 3. Representation of artifacts. We apply 4 synthetic distortions (blur, compression, darkening, super-resolution) to a source sequence, in order to determine if the encoder is able to see artifacts. We perform principal component analysis (PCA) on output features from the V-JEPA 2.1 encoder; the first three principal components are then mapped to the RGB channels.

Key Results

HiVE-VQA outperformed 17 SOTA methods, including both single-branch and complex multi-branch models, without requiring any fine-tuning on the test datasets.

  • Accuracy: It achieved a mean SROCC of 0.877 and PLCC of 0.890 across the four main test sets, representing a ~2.5% improvement over the best multi-branch models.
  • Efficiency: Despite its high accuracy, the model is highly efficient. It is roughly 11x to 29x less computationally expensive than older deep learning models like VSFA or BVQA. At 1080p, it can process a video in under a quarter of a second on a modern GPU.
Figure 1. The performance of the proposed HiVE-VQA model, and selected benchmark NR-VQA methods. Here, SROCC and PLCC are used to measure each model’s correlation with human- based mean opinion scores, and the results are averaged across the four test datasets. HiVE-VQA is the top performing model compared to 17 classical, deep (single-branch) and deep (multi- branch) models. The results shown here are based on frozen model parameters, without performing any fine-tuning on these datasets.

Limitations and Future Work

While the trainable parameters are minimal (5.2M), the total parameter count including the frozen backbone is still relatively large (~91M). The authors acknowledge this and suggest that future work could involve knowledge distillation or model pruning to compress the backbone for deployment on mobile or edge devices.

Additionally, while the frozen backbone generalizes exceptionally well, it may not perfectly capture the specific nuances of very niche datasets compared to models that are fully fine-tuned on those specific distributions. Future iterations might explore "parameter-efficient fine-tuning" (PEFT) techniques like LoRA to bridge this gap without losing the efficiency of the frozen approach.

Summary updated

Abstract

The proliferation of user-generated video content has made accurate and robust no-reference video quality assessment (NR-VQA) a highly important challenge. Existing research increasingly relies on architecturally complex multi-branch pipelines or computationally intensive fine-tuning of single-branch backbones. In this paper, we investigate whether a single frozen video foundation model can serve as an effective NR-VQA backbone, and propose HiVE-VQA, the first framework (to our knowledge) to apply a JEPA-based video encoder to this task. HiVE-VQA extracts token grids from multiple depths of a pre-trained V-JEPA 2.1 encoder and stacks them into a hierarchical spatio-temporal volume, which a compact 3D convolutional head fuses across its layer, spatial, and temporal axes, combining low-level distortion cues from shallow layers with the semantic and temporal context captured at greater depths. The encoder remains frozen throughout, so only the head is trained: 5.2M parameters (5.7\% of the total), with no auxiliary multi-modal models and an inference cost that is invariant to video resolution. Trained solely on LSVQ and evaluated without any fine-tuning, HiVE-VQA outperforms 17 state-of-the-art (SOTA) methods in SROCC and PLCC on both intra-dataset (LSVQ-test, LSVQ-1080p) and cross-dataset (KoNViD-1k, LIVE-VQC) benchmarks. These results indicate that hierarchical, depth-wise fusion of features from a single frozen video foundation model provides a more effective and efficient basis for NR-VQA than task-specific multi-network fusion. Code and model checkpoints are available at https://github.com/RobHam99/HiVE-VQA.
BibTeX
@misc{hamilton2026hive,
  author = {Robbie Hamilton and Fan Zhang and David Bull},
  title = {{HiVE-VQA: Hierarchical Interaction via Volumetric Embedding for Video Quality Assessment}},
  year = {2026},
  eprint = {2611.17146},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17146},
}
Figure