One Transform to Compute Them All: Efficient Fusion-Based Full-Reference Video Quality Assessment

Abhinau Kumar Venkataramanan, Cosmin Stejerean, Ioannis Katsavounidis, Alan C. Bovik

TL;DR

The FUNQUE (Fusion of Unified Quality Evaluators) framework introduces a "unified transform" that allows multiple video quality metrics to share computations, drastically reducing the overhead of full-reference quality assessment. By integrating human vision models directly into this shared wavelet transform, the authors created a suite of models (FUNQUE+) that are not only 3.8 to 11 times faster than the industry-standard VMAF but also 4-5% more accurate.

What Problem Does This Paper Address?

In the world of streaming (Netflix, YouTube, TikTok), "bitrate ladders" are the gold standard for delivery. To build these, providers must encode a single video dozens of times at different resolutions and bitrates to find the "perceptual sweet spot." This requires a Video Quality Assessment (VQA) tool to tell the encoder which version looks best to a human.

The current industry leader is VMAF (Visual Multimethod Assessment Fusion). While accurate, VMAF is a "fusion" model that runs several different algorithms (like VIF and DLM) and combines their scores. Because these sub-models (called "atoms") are heterogeneous—meaning they each require their own specific mathematical transformations of the video data—VMAF has become a massive computational bottleneck in modern hardware-accelerated encoding pipelines.

Methods

The authors propose a "unified transform" approach. Instead of each atom model doing its own math, FUNQUE processes the reference and distorted video frames through a single, perceptually-tuned transform:

  1. Unified Transform: They use a Haar wavelet transform modulated by a Contrast Sensitivity Function (CSF). The CSF models how the human eye is less sensitive to very high and very low spatial frequencies. By applying this filter once at the start, every subsequent feature extracted is "pre-sensitized" to human perception.
  2. Computation Sharing: They redesigned classic quality metrics (like SSIM, VIF, and DLM) to work directly on these wavelet coefficients. This allows the system to reuse the same intermediate calculations across multiple features.
  3. Self-Adaptive Scale Transform (SAST): This pre-processing step rescales videos based on expected viewing conditions (e.g., watching a 1080p screen from 3H distance), which further improves both speed and accuracy.
  4. Constrained Greedy Feature Selection (CGFS): To keep the final model lean, the authors developed a new selection algorithm. It picks the best features from "buckets" (SSIM-type, Info-type, Sharpness-type, etc.) to ensure the final fusion model is diverse but compact.

Datasets

The models were trained and evaluated using a massive cross-database methodology involving eight major Full HD video datasets:

  • BVI-HD & BVI-CC: High-definition and codec-comparison sets.
  • CC-HDDO: Specifically includes videos optimized by the Dynamic Optimizer.
  • NFLX-P: The Netflix Public dataset.
  • MCL-V, IVP, SHVC, and VQEG: Standard benchmarks covering various codecs (HEVC, AV1, x264) and distortion types.

Key Insights and Ideas

  • The Wavelet "Sweet Spot": The authors realized that the wavelet domain is uniquely suited for fusion. It provides a natural way to compute both "structural" features (like SSIM) and "information-theoretic" features (like VIF) without leaving the transform domain.
  • Chroma Matters: While VMAF and most SOTA models only look at the luma (brightness) channel, FUNQUE+ includes features from the Cb and Cr (color) channels. This "3C" (three-channel) approach significantly boosts accuracy for modern high-quality content.
  • Monotonicity is King: For an encoder to use a quality metric for optimization, the metric must be "monotonic"—if you increase the bitrate, the score must go up. The authors found that VMAF v0.6.1 occasionally fails this test in low-quality settings, whereas FUNQUE+ remains robustly monotonic.

Key Results

  • Accuracy: FUNQUE+ models achieved a Spearman Rank Order Correlation (SROCC) of 0.875, outperforming VMAF (0.831) and even complex deep-learning-based metrics like LPIPS and DISTS.
  • Efficiency:
    • Y-FUNQUE+ (luma only) is 11x faster than VMAF.
  • 3C-FUNQUE+ (including color) is 3.8x faster than VMAF.
  • GFLOPs: In terms of raw operations, Y-FUNQUE+ requires only ~2.2 GFLOPs to process a 6-second clip, compared to VMAF’s ~201 GFLOPs.

Limitations and Future Work

  • Metric Hacking: The authors admit that using the SAST (downscaling) step creates a vulnerability. A "sneaky" encoder could process video at half-resolution to get a perfect score from the metric, even though the actual video would be blurry. To counter this, they also provide "Full-Scale" (FS) versions of the models.
  • Future Directions: The current work focuses on 1080p SDR video. The authors aim to extend the FUNQUE framework to more demanding domains, including 4K/8K resolution, High Dynamic Range (HDR), and 360° VR video, where the computational savings of a unified transform will be even more critical.

Summary updated

Abstract

The Visual Multimethod Assessment Fusion (VMAF) algorithm has recently emerged as a state-of-the-art approach to video quality prediction, that now pervades the streaming and social media industry. However, since VMAF requires the evaluation of a heterogeneous set of quality models, it is computationally expensive. Given other advances in hardware-accelerated encoding, quality assessment is emerging as a significant bottleneck in video compression pipelines. Towards alleviating this burden, we propose a novel Fusion of Unified Quality Evaluators (FUNQUE) framework, by enabling computation sharing and by using a transform that is sensitive to visual perception to boost accuracy. Further, we expand the FUNQUE framework to define a collection of improved low-complexity fused-feature models that advance the state-of-the-art of video quality performance with respect to both accuracy, by 4.2% to 5.3%, and computational efficiency, by factors of 3.8 to 11 times!.
BibTeX
@article{venkataramanan2024one,
  author = {Abhinau Kumar Venkataramanan and Cosmin Stejerean and Ioannis Katsavounidis and Alan C. Bovik},
  title = {{One Transform to Compute Them All: Efficient Fusion-Based Full-Reference Video Quality Assessment}},
  journal = {IEEE Trans. Image Process.},
  year = {2024},
  doi = {10.1109/tip.2023.3345227},
}

Loading related content...

Figure