Subjective and Objective Quality Assessment of High Frame Rate Videos

Pavan C. Madhusudana, Xiangxu Yu, Neil Birkbeck, Yilin Wang, Balu Adsumilli, Alan C. Bovik

Abstract

High frame rate (HFR) videos are becoming increasingly common with the tremendous popularity of live, high-action streaming content such as sports. Although HFR contents are generally of very high quality, high bandwidth requirements make them challenging to deliver efficiently, while simultaneously maintaining their quality. To optimize trade-offs between bandwidth requirements and video quality, in terms of frame rate adaptation, it is imperative to understand the intricate relationship between frame rate and perceptual video quality. Towards advancing progression in this direction we designed a new subjective resource, called the LIVE-YouTube-HFR (LIVE-YT-HFR) dataset, which is comprised of 480 videos having 6 different frame rates, obtained from 16 diverse contents. In order to understand the combined effects of compression and frame rate adjustment, we also processed videos at 5 compression levels at each frame rate. To obtain subjective labels on the videos, we conducted a human study yielding 19,000 human quality ratings obtained from a pool of 85 human subjects. We also conducted a holistic evaluation of existing state-of-the-art Full and No-Reference video quality algorithms, and statistically benchmarked their performance on the new database.

Introduction: The Challenge of High Frame Rate Video

While the digital media industry has aggressively pursued higher spatial resolutions (4K and 8K) and High Dynamic Range (HDR), the impact of High Frame Rate (HFR) on user experience has remained less explored. Delivering HFR content (60–120 fps) is significantly more expensive in terms of bandwidth. To optimize the trade-off between file size and fluid motion, engineers need to understand the "perceptual math" of frame rates—specifically, how much quality is gained by increasing temporal resolution versus the cost of compression.

Existing Video Quality Assessment (VQA) datasets often lack HFR content, are not publicly available, or fail to consider the complex interaction between frame rate and compression. This research fills that gap by providing a massive, labeled dataset and benchmarking how well current algorithms can predict the benefits of smoother motion.

Methodology: Building the LIVE-YT-HFR Dataset

The researchers followed a rigorous pipeline to create a reliable "ground truth" for HFR quality assessment. The process involved four key stages:

  1. Temporal Downsampling: Starting with 120 fps source videos, the team used frame dropping (rather than frame averaging) to create versions at 24, 30, 60, 82, and 98 fps. This method was chosen to avoid artificial motion blur and better simulate native capture at lower rates.
  2. Compression: Each version was compressed using the VP9 codec at five different levels, ranging from lossless to high compression (CRF 63), to study the joint effects of temporal and spatial degradation.
  3. Subjective Study: 85 human subjects provided over 19,000 individual quality ratings. To ensure accuracy, the researchers used a professional 4K monitor capable of native 120Hz playback, avoiding any device-side interpolation that could skew results.
  4. Objective Benchmarking: A suite of Full-Reference (FR) and No-Reference (NR) algorithms, including popular metrics like VMAF and SSIM, were tested to see if they could accurately predict human scores.

The LIVE-YT-HFR Dataset

The primary contribution of this work is the LIVE-YT-HFR Database, a large-scale resource for the research community. It contains 480 videos derived from 16 diverse source sequences, covering a wide range of motion and spatial complexity.

Representative frames from the 16 source sequences used to construct the LIVE-YT-HFR dataset.

The dataset includes six frame rates (24–120 fps) and five compression levels at both 1080p and 4K resolutions. By providing these diverse sequences, the researchers allow for a nuanced analysis of how different types of content respond to HFR. The dataset has been made publicly available for research purposes.

Perceptual Insights: The 60 fps Threshold

The study revealed several critical insights into how humans perceive motion smoothness:

  • The 60 fps Inflection Point: Perceived quality increases sharply as the frame rate moves from 24 to 60 fps. However, the "returns" on quality start to diminish between 60 and 120 fps. For many viewers, the jump from 30 to 60 fps is far more noticeable than the jump from 60 to 120 fps.
  • Content Dependency: The benefits of HFR are highly content-dependent. High-action sports sequences showed a much wider quality gap between 24 and 120 fps compared to more static scenes.
High-action content, such as this hurdles sequence, shows the most significant quality gains from high frame rates.
  • The "Spatial Bias" Trap: Standard metrics like PSNR actually performed better at lower frame rates. This is because these metrics only evaluate spatial pixels. When a video has high spatial quality but suffers from severe temporal "judder" (common at 24 fps), these metrics give it a high score, ignoring the motion artifacts that significantly degrade the human viewing experience.

Benchmarking Objective Metrics

The benchmarking results highlighted a significant gap in current VQA technology:

  • Top Performer: The Generalized Spatio-Temporal Index (GSTI) was the most accurate Full-Reference metric, achieving a Spearman Rank Order Correlation Coefficient (SROCC) of 0.79.
  • VMAF Performance: Netflix's VMAF performed reasonably well (SROCC 0.73) but struggled to capture specific motion-related artifacts like "stutter" in cinematic content.
  • Blind VQA Gap: No-Reference (blind) metrics performed poorly, with SROCCs hovering around 0.40. This proves that predicting HFR quality without a reference video remains a major open challenge in the field.

Conclusion and Future Work

The LIVE-YT-HFR dataset provides a vital foundation for improving video delivery. However, limitations remain. The study focused exclusively on the VP9 codec; future work should explore interactions with HEVC or AV1. Furthermore, the poor performance of blind VQA metrics suggests a need for new features that specifically target temporal smoothness and motion aliasing. As streaming services move toward Variable Frame Rate (VFR) delivery, understanding these perceptual thresholds will be essential for maintaining high user satisfaction while managing bandwidth costs.

Summary updated

Abstract

High frame rate (HFR) videos are becoming increasingly common with the tremendous popularity of live, high-action streaming content such as sports. Although HFR contents are generally of very high quality, high bandwidth requirements make them challenging to deliver efficiently, while simultaneously maintaining their quality. To optimize trade-offs between bandwidth requirements and video quality, in terms of frame rate adaptation, it is imperative to understand the intricate relationship between frame rate and perceptual video quality. Towards advancing progression in this direction we designed a new subjective resource, called the LIVE-YouTube-HFR (LIVE-YT-HFR) dataset, which is comprised of 480 videos having 6 different frame rates, obtained from 16 diverse contents. In order to understand the combined effects of compression and frame rate adjustment, we also processed videos at 5 compression levels at each frame rate. To obtain subjective labels on the videos, we conducted a human study yielding 19,000 human quality ratings obtained from a pool of 85 human subjects. We also conducted a holistic evaluation of existing state-of-the-art Full and No-Reference video quality algorithms, and statistically benchmarked their performance on the new database. The LIVE-YT-HFR database has been made available online for public use and evaluation purposes, with hopes that it will help advance research in this exciting video technology direction. It may be obtained at \url{https://live.ece.utexas.edu/research/LIVE_YT_HFR/LIVE_YT_HFR/index.html}
BibTeX
@inproceedings{madhusudana2021subjective,
  author = {Pavan C. Madhusudana and Xiangxu Yu and Neil Birkbeck and Yilin Wang and Balu Adsumilli and Alan C. Bovik},
  title = {{Subjective and Objective Quality Assessment of High Frame Rate Videos}},
  booktitle = {IEEE Access},
  year = {2021},
  url = {https://openreview.net/forum?id=J-71v0SpA16},
}
Figure