Complexity-Oriented Per-Shot Video Coding Optimization

Hongcheng Zhong, Jun Xu, Chen Zhu, Donghui Feng, Li SongICME 2022 · OpenReview

TL;DR

This paper introduces a novel approach to video encoding that optimizes not just for bitrate and distortion, but also for computational complexity. By extending per-shot encoding to include encoder presets as a tunable parameter, they enable flexible complexity control, making advanced video compression more practical for user-generated content.

What Problem Does This Paper Address?

The increasing demand for video content, especially user-generated content (UGC), puts a strain on storage, bandwidth, and encoding resources. While per-shot video encoding has shown promise in improving compression efficiency by optimizing each shot individually, it often comes at the cost of significant pre-processing complexity. Current methods primarily adjust spatial resolution and quantization parameters (QP/CRF), lacking flexibility in controlling the actual encoding effort. This limits their applicability to UGC platforms where computational resources are more constrained compared to professionally generated content (PGC). The authors aim to bridge this gap by introducing a method that allows for fine-grained control over encoding complexity, making efficient video compression accessible for a wider range of applications.

Methods

The core of this work lies in extending the per-shot encoding framework to incorporate encoder presets as a tunable parameter, alongside resolution and QP/CRF. This creates a richer parameter space for optimizing each shot. The authors propose a hyperbolic Rate-Distortion-Time (RDT) model to capture the relationship between bitrate, distortion, and encoding time.

Their methodology involves:

  1. Convex Hull Encoding with Extended Parameters: For each shot, they encode it using a wide range of presets, resolutions, and QP/CRF values, generating numerous "operating points" (combinations of parameters and their resulting metrics). These points are then filtered using a 3D convex hull to remove sub-optimal ones, similar to conventional per-shot encoding but with the added complexity dimension.
  2. RDT Model Fitting: They fit a hyperbolic model ($d = c \cdot r^{k_1} \cdot t^{k_2}$) to the filtered operating points for each shot. This model allows them to estimate the rate-distortion-complexity trade-offs.
  3. Lagrange Multiplier Optimization: Using the RDT model, they derive Lagrange multipliers ($\lambda$ and $\mu$) that represent the trade-offs between rate, distortion, and time. These multipliers are crucial for finding optimal operating points.
  4. Representation Assembly: An algorithm is proposed to assemble these optimal operating points from individual shots into complete video representations. This assembly is guided by the derived Lagrange multipliers, allowing for the generation of representations with specific target bitrates and complexity constraints. Essentially, they solve the problem in reverse, by finding the optimal representation for a given complexity and bitrate.

Datasets

The experiments were conducted using the BVI-1004K dataset. This dataset comprises 100 UHD video sequences (3840x2160 resolution) with a 4:2:0 chroma format, 10-bit depth, and 60 fps frame rate. Notably, each sequence in this dataset represents a single scene, meaning no scene cuts are present within a sequence. This simplifies the per-shot analysis as each sequence can be treated as a single "shot" for the purpose of this experiment.

Key Insights and Ideas

  • Complexity as a First-Class Parameter: The most significant insight is treating encoding complexity as a tunable parameter on par with bitrate and distortion. This moves beyond simply selecting a fixed preset and allows for dynamic adaptation.
  • Hyperbolic RDT Model Efficacy: The hyperbolic RDT model ($d = c \cdot r^{k_1} \cdot t^{k_2}$) proved to be a surprisingly accurate fit for describing the rate-distortion-complexity relationship across different encoder presets, with high R-squared scores.
  • Dense Complexity Control: The ability to achieve a dense range of complexity constraints (from 100% down to 3%) is a major practical advantage, offering much finer control than traditional methods.
  • Flexibility for UGC: By enabling complexity-aware encoding, the method directly addresses the needs of user-generated content platforms, where computational costs are a significant concern.

Key Results

  • The proposed method achieves complexity constraints ranging from 100% down to 3% in a dense form, offering significant flexibility.
  • When compared to conventional per-shot encoding with similar complexity levels (fixed presets), the proposed method achieves BDrate gains up to -19.17%. This indicates substantial improvements in compression efficiency for a given complexity.
  • The hyperbolic RDT model demonstrated high fitting accuracy, with average R² scores above 0.974 for x265 and above 0.948 for x264.

Limitations and Future Work

  • Brute-Force Convex Hull: The current method involves a brute-force convex hull computation across all presets, which can be computationally intensive (about twice the complexity of testing the slowest preset).
  • Future Work: The authors propose to simplify the brute-force method by developing techniques to predict the encoding results of slower presets. This would further reduce the computational overhead and make the approach even more practical.

Summary updated

Abstract

Current per-shot encoding schemes aim to improve the compression efficiency by shot-level optimization. It splits a source video sequence into shots and imposes optimal sets of encoding parameters on each shot. Per-shot encoding achieved approximately 20% bitrate savings over baseline fixed QP encoding at the expense of pre-processing complexity. However, the adjustable parameter space of the current per-shot encoding schemes only has spatial resolution and QP/CRF, resulting in a lack of encoding flexibility. In this paper, we extend the per-shot encoding framework in the complexity dimension. We believe that per-shot encoding with flexible complexity will help in deploying user-generated content. We propose a rate-distortion-complexity optimization process for encoders and a methodology to determine the coding parameters under the constraints of complexities and bitrate ladders. Experimental results show that our proposed method achieves complexity constraints ranging from 100% to 3% in a dense form compared to the slowest per-shot anchor. With similar complexities of the per-shot scheme fixed in specific presets, our proposed method achieves BDrate gain up to −19.17%.
BibTeX
@inproceedings{zhong2022complexity,
  author = {Hongcheng Zhong and Jun Xu and Chen Zhu and Donghui Feng and Li Song},
  title = {{Complexity-Oriented Per-Shot Video Coding Optimization}},
  booktitle = {ICME},
  year = {2022},
  url = {https://openreview.net/forum?id=Ljz_y8z0tCC},
}
Figure