Abstract
Video quality measurement takes an important role in many applications. Full-reference quality metrics which are usually used in video codecs comparisons are expected to reflect any changes in videos. In this article, we consider different color corrections of compressed videos which increase the values of full-reference metric VMAF and almost don't decrease other widely-used metric SSIM. The proposed video contrast enhancement approach shows the metric inapplicability in some cases for video codecs comparisons, as it may be used for cheating in the comparisons via tuning to improve this metric values.
Introduction
In the domain of video compression, objective metrics are essential for evaluating quality loss during encoding. While traditional metrics like Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) are industry standards, they often fail to align with human visual perception. Netflix’s Video Multimethod Fusion Approach (VMAF) was developed to address this by using a machine-learning model to fuse multiple spatial and temporal features into a single quality score.
However, as a learning-based regressor, VMAF may possess "blind spots" that can be exploited. This study investigates whether specific image-processing transformations can artificially inflate VMAF scores without improving—or while even slightly distorting—the actual visual quality. This is a critical issue for industry benchmarks, where developers might "tune" their codecs specifically to win on VMAF rather than to provide better visual quality.
Methods
The research focuses on two common image processing techniques: Unsharp Masking and Histogram Equalization. Unsharp Masking enhances edge contrast using a Gaussian blur mask, controlled by radius and amount parameters. Histogram Equalization adjusts global contrast, specifically through the clipping limit parameter.
To identify optimal parameters for maximizing VMAF while maintaining SSIM stability, the researchers employed the NSGA-II multi-objective genetic algorithm. The VMAF architecture, which fuses features via a Support Vector Machine (SVM) regressor, is shown below.
The study utilized four high-definition (FullHD) video sequences: "Bay time-lapse," "Crowd run," "Red kayak," and "Speed bag." These sequences represent a range of spatial and temporal complexities.
Results and Discussion
The core finding of this study is the existence of a "second quadrant" hack—parameter configurations where VMAF scores increase significantly while SSIM remains constant or improves. This indicates a successful manipulation of the metric without a corresponding improvement in structural integrity.
VMAF demonstrates a high sensitivity to contrast. By artificially sharpening edges or stretching the histogram, the underlying SVM is misled into predicting higher quality. For the "Bay time-lapse" sequence, an unsharp mask (radius 2.843, amount 0.179) increased the VMAF score from 68.16 to 72.01, while SSIM remained nearly unchanged (0.879 to 0.878).
Visually, these "hacked" frames are nearly indistinguishable from the originals, yet the metric reports a substantial quality jump. Similar results were observed for the "Crowd run" sequence using histogram equalization, where VMAF increased from 51.0 to 53.1 with negligible changes to SSIM. The fact that a simple pre-processing step can lead to a higher final score suggests that VMAF is not yet a "universal" metric and can be fooled by "tuning" that doesn't necessarily reflect a better codec.
Conclusion
The ability to inflate VMAF scores through simple pre-processing steps highlights a vulnerability in objective metrics. While VMAF has become a gold standard, it is not unhackable. These results serve as a call to action for metric developers to enhance robustness against contrast-based tuning. Future research will focus on extensive subjective testing to determine if human viewers actually prefer these "hacked" versions or find the added contrast distracting, ensuring that automated quality assessment remains a reliable tool for the industry.
Summary updated