Referenceless Rate-Distortion Modeling with Learning from Bitstream and Pixel Features

Yangfan Sun, Li Li, Shan LiuACM Multimedia 2020 · OpenReview

Abstract

Generally, adaptive bitrates for variable Internet bandwidths can be obtained through multi-pass coding. Referenceless prediction-based methods show practical benefits compared with multi-pass coding to avoid excessive computational resource consumption, especially in low-latency circumstances. However, most of them fail to predict precisely due to the complex inner structure of modern codecs. Therefore, to improve the fidelity of prediction, we propose a referenceless prediction-based R-QP modeling (PmR-QP) method to estimate bitrate by leveraging a deep learning algorithm with only one-pass coding. It refines the global rate-control paradigm in modern codecs on flexibility and applicability with few adjustments as possible. By exploring the potentials of bitstream and pixel features from the prerequisite of one-pass coding, it can reach the expectation of bitrate estimation in terms of precision. Extensive experiments demonstrate the proposed method significantly decreases the proportion of samples' bitrate estimation error exceeding 10% by 24.60% on average over the state-of-the-art.

Introduction

In the realm of video streaming, rate control is the essential process of balancing visual quality with available bandwidth. This requires an accurate understanding of the Rate-Quantization Parameter (R-QP) relationship—how many bits (Rate) are required for a specific quality level (QP). While traditional multi-pass encoding provides high accuracy by mapping this relationship through repeated trials, its computational cost and latency are prohibitive for live-streaming applications. Existing referenceless methods, which attempt to estimate the bitrate without multiple encodes, often fall short because they rely on overly simplistic linear models or ignore the complex internal logic of modern codecs like HEVC.

Methodology

The proposed PmR-QP method addresses these challenges by integrating deep learning with internal codec metadata, moving beyond raw pixel analysis to a more holistic "hybrid" feature set.

Quadratic R-QP Modeling

Traditional encoders often use linear approximations that fail to capture the non-linear relationship between bitrate and quality, especially as video resolution and content complexity increase. We derive a robust quadratic modeling function based on the observation that Discrete Cosine Transform (DCT) coefficients follow a Cauchy-based distribution. As illustrated in the log-log plot below, the quadratic approximation closely follows the Cauchy-based function across various content complexities (represented by $\gamma$), significantly outperforming standard linear models.

Comparison of the Cauchy-based R-QP function with linear and quadratic approximations across various content complexities.

Hybrid Referenceless Features and Framework

Instead of relying solely on image texture, PmR-QP extracts "hybrid" features from the initial mandatory encoding pass. These features treat the codec itself as a high-level feature extractor:

  • Reconstructed Mappings ($M_{rec}$): The actual pixel data after reconstruction.
  • Segmentation Mappings ($M_{seg}$): A visualization of the hierarchical quad-tree architecture used for block partitioning (CUs).
  • Intra-Mode Mappings ($M_{intra}$): A map of spatial prediction directions chosen for each block.

These disparate data types are unified into an image-like structure and processed by a lightweight 5-layer Convolutional Neural Network (CNN) for regression. To further boost accuracy, the model uses the data from the first encoding pass as a "fixed point" on the R-QP curve, anchoring the quadratic function and reducing the parameters the network must learn.

The PmR-QP framework integrates pixel-level texture data with internal codec decisions like block partitioning and prediction modes.

Experimental Results

The model was trained and validated using the DIV2K dataset, with features extracted via the HEVC test model (HM 16.9). Extensive experiments demonstrate that PmR-QP significantly improves bitrate estimation accuracy over state-of-the-art linear modeling solutions. Specifically, it decreased the proportion of samples with a bitrate estimation error exceeding 10% by an average of 24.60%. Notably, the model maintained high performance across different resolutions (512x512 and 768x768), whereas traditional linear models saw their performance deteriorate as frame size increased.

Conclusion

PmR-QP demonstrates that leveraging internal codec decisions—such as partitioning and prediction modes—as high-level features significantly enhances referenceless bitrate prediction. By combining these bitstream features with a robust quadratic model, the system achieves high-precision rate control suitable for low-latency environments. Future work will focus on extending this methodology to inter-predicted frames (P and B frames) by incorporating motion estimation data to handle entire video sequences.

Summary updated

Abstract

Generally, adaptive bitrates for variable Internet bandwidths can be obtained through multi-pass coding. Referenceless prediction-based methods show practical benefits compared with multi-pass coding to avoid excessive computational resource consumption, especially in low-latency circumstances. However, most of them fail to predict precisely due to the complex inner structure of modern codecs. Therefore, to improve the fidelity of prediction, we propose a referenceless prediction-based R-QP modeling (PmR-QP) method to estimate bitrate by leveraging a deep learning algorithm with only one-pass coding. It refines the global rate-control paradigm in modern codecs on flexibility and applicability with few adjustments as possible. By exploring the potentials of bitstream and pixel features from the prerequisite of one-pass coding, it can reach the expectation of bitrate estimation in terms of precision. To be more specific, we first describe the R-QP relationship curve as a robust quadratic R-QP modeling function derived from the Cauchy-based distribution. Second, we simplify the modeling function by fastening one operational point of the relationship curve received from the coding process. Third, we learn the model parameters from bitstream and pixel features, named them hybrid referenceless features, comprising texture information, hierarchical coding structure, and selected modes in intra-prediction. Extensive experiments demonstrate the proposed method significantly decreases the proportion of samples' bitrate estimation error within 10% by 24.60% on average over the state-of-the-art.
BibTeX
@inproceedings{sun2020referenceless,
  author = {Yangfan Sun and Li Li and Zhu Li and Shan Liu},
  title = {{Referenceless Rate-Distortion Modeling with Learning from Bitstream and Pixel Features}},
  booktitle = {ACM Multimedia},
  year = {2020},
  url = {https://openreview.net/forum?id=Oi-hp30DCQn},
}
Figure