Efficient Bitrate Ladder Construction for Content-Optimized Adaptive Video Streaming

Angeliki Katsenou, Joel Solé, David BullIEEE Open Journal of Signal Processing 2021 · DOI

TL;DR

This paper introduces a machine learning-driven approach to automatically generate optimized bitrate ladders for adaptive video streaming. By analyzing content features, it significantly reduces the computational cost of creating these ladders, achieving substantial savings in encoding time and resources while maintaining high video quality.

What Problem Does This Paper Address?

The explosive growth of video streaming has created a demand for high-quality, seamless viewing experiences across a diverse range of devices and network conditions. Traditional "one-size-fits-all" bitrate ladders, which offer a fixed set of video qualities, often fail to meet these diverse needs, leading to buffering or visual artifacts. While content-tailored solutions exist, they are prohibitively expensive due to the massive number of video encodes required to determine optimal compression parameters for every video. This paper addresses the challenge of creating content-specific bitrate ladders efficiently, balancing quality and computational cost.

Methods

The core of the proposed method lies in predicting the optimal bitrate ladder without exhaustive encoding. The authors extract spatio-temporal features from uncompressed video content. These features are then used to train machine learning models to predict key parameters of the Rate-Quality (RQ) curves, specifically the "cross-over QPs" (Quantization Parameters) where the optimal resolution switches.

The process involves two main stages:

  1. Training: Uncompressed videos are downscaled to various resolutions. For each resolution, a wide range of QPs are used to generate RQ curves. The "Pareto Front" (PF), representing the optimal trade-off between rate and quality, is constructed. Crucially, the QPs at the intersection points of these curves across resolutions (cross-over QPs) are identified as ground truth. Spatio-temporal features are extracted from the original videos, and ML models are trained to predict these cross-over QPs.
  2. Testing: For a new video, spatio-temporal features are extracted. The trained ML models predict the cross-over QPs. A small number of actual encodes are then performed at these predicted QPs to define the bitrate ladder.

A key innovation is the development of a "Feature-based Ladder" (FL) method that leverages these predicted cross-over QPs and a QP-log(Rate) model to estimate the bitrate ladder. Additionally, a "Hybrid Ladder" (HL) approach is introduced, which intelligently selects between the FL method and a simpler interpolation-based method based on predicted performance for each sequence.

Datasets

The authors utilized a dataset of 100 publicly available UHD video sequences from various sources, including Netflix Chimera, Ultra Video Group, Harmonic, SJTU, and AWS Elemental. These sequences were chosen for their diversity in scenes, camera motion, and spatial activity. The dataset was pre-processed to a native resolution of 3840x2160 and temporally cropped to 64 frames. For evaluation, the authors compared their methods against a "Reference Ladder" (RL) constructed via exhaustive search and an "Interpolation-based Ladder" (IL).

Key Insights and Ideas

  • Content is King (for optimization): The paper strongly emphasizes that video content characteristics are the primary drivers of optimal compression parameters, making content-specific ladders far superior to generic ones.
  • Predicting the "Switching Points": The core idea of predicting the cross-over QPs is a clever way to bypass extensive encoding. These points are critical because they dictate when a streaming service should switch to a different resolution.
  • Feature Engineering Matters: The selection and use of spatio-temporal features (like GLCM descriptors and Temporal Coherence) are crucial for the ML models to accurately predict the cross-over QPs.
  • The Power of Hybridization: The hybrid approach (HL) demonstrates that a combination of methods, intelligently chosen based on sequence characteristics, can offer a better balance of accuracy and efficiency than any single method alone.
  • QP-log(Rate) as a Predictor: The linear relationship between QP and log(Rate) is a robust model for estimating bitrate ladder points once the cross-over QPs are known.

Key Results

The proposed Feature-based Ladder (FL) method achieved significant reductions in encoding requirements:

  • 89.06% fewer encodes compared to the exhaustive search (RL) method.
  • 61.46% fewer encodes compared to an interpolation-based method (IL).
  • The FL method resulted in an average Bjøntegaard Delta Rate (BD-Rate) difference of 1.78% compared to the exhaustive approach.

The Hybrid Ladder (HL) method further improved efficiency:

  • An overall 83.83% reduction in required encodings compared to RL.
  • An average BD-Rate difference of 1.26% compared to RL.
  • Both FL and HL methods produced bitrate ladders where, on average, over 80% of points belonged to the Pareto Front, indicating high optimality.

Limitations and Future Work

  • Codec Dependency: The current implementation is based on the HEVC codec. While the authors suggest the method is extendable to other codecs, retraining of the ML models would be necessary.
  • Quality Metric: The study primarily uses PSNR. Future work will explore the effectiveness of the method with other quality metrics like VMAF and SSIM.
  • Classification Accuracy: The hybrid method's performance relies on a classifier that selects between FL and IL. The current classification accuracy of 68% is noted as an area for potential improvement.
  • Generalization: While tested on a diverse dataset, further validation across a wider range of content types and codecs would be beneficial.

The authors propose future work to investigate the method's performance with different video codecs and quality metrics, aiming for even broader applicability and efficiency gains in adaptive video streaming.

Summary updated

Abstract

One of the challenges faced by many video providers is the heterogeneity of network specifications, user requirements, and content compression performance. The universal solution of a fixed bitrate ladder is inadequate in ensuring a high quality of user experience without re-buffering or introducing annoying compression artifacts. However, a content-tailored solution, based on extensively encoding across all resolutions and over a wide quality range is highly expensive in terms of computational, financial, and energy costs. Inspired by this, we propose an approach that exploits machine learning to predict a content-optimized bitrate ladder for on-demand video services. The method extracts spatio-temporal features from the uncompressed content, trains machine-learning models to predict the Pareto front parameters and, based on that, builds the ladder within a defined bitrate range. The method has the benefit of significantly reducing the number of encodes required per sequence. The presented results, based on 100 HEVC-encoded sequences, demonstrate a reduction in the number of encodes required when compared to an exhaustive search and an interpolation-based method, by 89.06% and 61.46%, respectively, at the cost of an average Bjøntegaard Delta Rate difference of 1.78% compared to the exhaustive approach. Finally, a hybrid method is introduced that selects either the proposed or the interpolation-based method depending on the sequence features. This results in an overall 83.83% reduction of required encodings at the cost of an average Bjøntegaard Delta Rate difference of 1.26%.
BibTeX
@article{katsenou2021efficient,
  author = {Angeliki Katsenou and Joel Solé and David Bull},
  title = {{Efficient Bitrate Ladder Construction for Content-Optimized Adaptive Video Streaming}},
  journal = {IEEE Open Journal of Signal Processing},
  year = {2021},
  doi = {10.1109/ojsp.2021.3086691},
}
Figure