Learned Fast HEVC Intra Coding

Zhibo Chen, Jun Shi, Weiping LiIEEE TIP 2020 · OpenReview

TL;DR

This paper introduces a novel framework, LFHI, for significantly speeding up HEVC intra-frame video compression. By employing specialized Convolutional Neural Networks (CNNs) and a new concept called Minimum Number of RDO Candidates (MNRC), they achieve up to 75.2% complexity reduction with minimal impact on video quality, outperforming existing methods.

What Problem Does This Paper Address?

High Efficiency Video Coding (HEVC) offers excellent video compression performance, largely due to its flexible coding unit (CU) partitioning and numerous intra-prediction modes. However, this flexibility comes at a steep computational cost, making real-time applications challenging. Existing methods for speeding up HEVC intra coding often focus on either CU partitioning or intra-mode decision, but rarely address both comprehensively while also considering factors like varying quantization parameters (QPs) and hardware parallelism. This paper aims to bridge this gap by proposing a unified framework that tackles both aspects efficiently and adaptively.

Methods

The proposed Learned Fast HEVC Intra Coding (LFHI) framework integrates several novel components:

  • Asymmetric-Kernel CNN (AK-CNN): A lightweight CNN designed to efficiently extract directional texture features. This is crucial for both predicting CU/PU partitioning decisions and for fast intra-mode selection. The asymmetric kernels are specifically tailored to capture near-horizontal and near-vertical textures, which are important for intra-prediction.
  • Minimum Number of RDO Candidates (MNRC): Instead of using a fixed number of candidate modes for Rate-Distortion Optimization (RDO), LFHI predicts the "minimum number of RDO candidates" (MNRC) required for each prediction unit (PU). This prediction is done using the AK-CNN, allowing for adaptive RDO candidate selection and reducing redundant computations.
  • Evolution Optimized Threshold Decision (EOTD): To achieve configurable tradeoffs between complexity reduction and coding performance, an evolutionary algorithm is used to optimize confidence thresholds for the CNN's predictions. This allows users to balance speed and quality based on their specific needs.
  • Interpolation-based QP Adaption: To generalize the framework across different quantization parameters (QPs) without retraining for each, LFHI uses an interpolation scheme. It trains models on anchor QPs and then interpolates predictions for intermediate QPs, ensuring consistent performance across a wide QP range.

Datasets

  • EHIC Dataset: The authors created a new dataset called the Extended HEVC Intra Coding (EHIC) dataset. This dataset is specifically designed for training and evaluating CNNs for both CU/PU size decision and intra-mode decision. It includes high-definition and low-definition raw images from various sources (UCID, RAISE, DIV2K), encoded using the HEVC reference software (HM 16.9) with four QPs ({22, 27, 32, 37}). Labels for CU splitting and the ranking of intra-prediction modes for RDO were collected.

Key Insights and Ideas

  • Texture Directionality Matters: The success of the AK-CNN highlights the importance of specialized feature extraction for intra-coding. Asymmetric kernels that focus on directional textures are more effective than standard square kernels for predicting partitioning and modes.
  • Adaptive RDO is Key: Moving away from fixed RDO candidate lists to an adaptive approach based on predicted MNRC significantly reduces redundant computations without sacrificing quality.
  • Configurable Tradeoffs are Essential: The EOTD scheme provides a powerful mechanism for users to fine-tune the balance between compression speed and quality, making the framework highly adaptable to different application requirements.
  • Interpolation for QP Generalization: The interpolation-based approach for QP adaption is a clever way to avoid the complexity and precision issues of training a single, massive CNN for all QPs.

Key Results

  • Complexity Reduction: The LFHI framework achieves up to 75.2% intra-mode encoding complexity reduction.
  • RD Performance: The framework incurs a negligible 2.09% Bjøntegaard delta bit-rate (BD-BR) increase on average, demonstrating excellent rate-distortion performance maintenance.
  • Outperformance: LFHI significantly outperforms existing state-of-the-art fast intra-coding schemes in terms of both complexity reduction and RD performance.
  • Parallelism: The framework is designed for high parallelism, as it does not require reconstructed pixels from neighboring blocks, making it suitable for hardware implementation.
  • QP Generalization: The interpolation scheme shows stable performance across a wide range of QPs, outperforming methods that use a single large CNN.

Limitations and Future Work

  • H.266/VVC Extension: The authors acknowledge that the partition pattern in the newer H.266/VVC standard is more complex. Directly applying their current separate prediction scheme might not be suitable, and they plan to explore extensions for VVC.
  • Mode Decision Comparison: While the paper compares the final chosen modes, they note that directly comparing modes might not be entirely effective due to the influence of reconstructed pixels. They propose using "cover rate" as a metric to measure prediction effectiveness, which shows high performance for their scheme.
  • Computational Overhead: While significantly reducing overall complexity, the CNN inference itself introduces some overhead. However, this overhead is shown to be minimal (less than 1% of the anchor's encoding time) and can be further accelerated with hardware like GPUs.

Summary updated

Abstract

In High Efficiency Video Coding (HEVC), excellent rate-distortion (RD) performance is achieved in part by having a flexible quadtree coding unit (CU) partition and a large number of intra-prediction modes. Such an excellent RD performance is achieved at the expense of much higher computational complexity. In this paper, we propose a learned fast HEVC intra coding (LFHI) framework taking into account the comprehensive factors of fast intra coding to reach an improved configurable tradeoff between coding performance and computational complexity. First, we design a low-complex shallow asymmetric-kernel CNN (AK-CNN) to efficiently extract the local directional texture features of each block for both fast CU partition and fast intra-mode decision. Second, we introduce the concept of the minimum number of RDO candidates (MNRC) into fast mode decision, which utilizes AK-CNN to predict the minimum number of best candidates for RDO calculation to further reduce the computation of intra-mode selection. Third, an evolution optimized threshold decision (EOTD) scheme is designed to achieve configurable complexity-efficiency tradeoffs. Finally, we propose an interpolation-based prediction scheme that allows for our framework to be generalized to all quantization parameters (QPs) without the need for training the network on each QP. The experimental results demonstrate that the LFHI framework has a high degree of parallelism and achieves a much better complexity-efficiency tradeoff, achieving up to 75.2% intra-mode encoding complexity reduction with negligible rate-distortion performance degradation, superior to the existing fast intra-coding schemes.
BibTeX
@article{chen2020learned,
  author = {Zhibo Chen and Jun Shi and Weiping Li},
  title = {{Learned Fast HEVC Intra Coding}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=218SYNDUV8k},
}
Figure