Enhancing Quality for HEVC Compressed Videos

Ren Yang, Mai Xu, Zulin Wang, Zhenyu GuanIEEE TCSVT 2017 · DOI · OpenReview

Abstract

The High Efficiency Video Coding (HEVC) standard is the current benchmark for video compression, yet it often introduces significant visual artifacts at low bit rates. To mitigate these distortions, we propose a Quality Enhancement Convolutional Neural Network (QE-CNN) framework designed for the decoder side. Our approach utilizes two specialized models: QE-CNN-I for intra-coded frames and QE-CNN-P for inter-coded (P/B) frames. By incorporating a Time-constrained Quality Enhancement Optimization (TQEO) scheme, we ensure that the computational overhead of these deep models does not impede real-time playback. Experimental results show that QE-CNN significantly improves visual quality, achieving up to 0.55 dB PSNR gains and an average BD-rate saving of 11.06%.

Introduction

As internet video traffic continues to grow, efficient compression standards like HEVC are vital. However, aggressive compression at low bit rates leads to artifacts such as blocking, ringing, and blurring, which degrade the user experience. While post-processing techniques have long been used to "clean" video, traditional filters often struggle with the complex, non-linear distortions produced by modern codecs. Deep learning offers a powerful alternative, but existing image-based CNNs are ill-equipped to handle the temporal dependencies and motion-compensated residuals unique to video.

The QE-CNN Architecture

The core innovation of our work is the separation of quality enhancement into two distinct tasks based on frame type.

QE-CNN-I: Intra-Frame Enhancement

Intra-coded frames (I-frames) serve as spatial anchors in a video stream. While they resemble static images, HEVC intra-coding produces artifacts that are more complex than those found in JPEG compression. To capture these features, we developed QE-CNN-I, a 5-layer convolutional network.

Comparison of learned filters ($W_1$) between AR-CNN (trained on JPEG, top) and the proposed QE-CNN-I (trained on HEVC, bottom).

Unlike previous 3-layer models, QE-CNN-I uses a larger number of filters (up to 128 in the first layer) and Parametric Rectified Linear Units (PReLU) to better model the distortion distribution. As seen in the learned filter weights above, QE-CNN-I captures much more complex spatial patterns compared to models trained on simpler JPEG artifacts, which is crucial for effective HEVC restoration.

QE-CNN-P: Addressing Inter-Frame Distortion

Inter-coded frames (P and B frames) introduce temporal complexity. These frames are reconstructed using motion vectors and residuals from reference frames. Consequently, their distortion is a hybrid of spatial artifacts and temporal propagation.

To address this, we propose QE-CNN-P, a 9-layer architecture that utilizes "feature concatenation." As shown in the diagram below, the model extracts features from both the current frame and the motion-compensated reference frame.

The 9-layer QE-CNN-P architecture, which fuses intra and inter coding features to handle temporal distortions.

By fusing these two feature streams, QE-CNN-P can effectively denoise motion-compensated blocks, a task that standard image-restoration CNNs cannot perform. This "secret sauce" allows the network to understand how motion-compensated blocks differ from purely spatial ones.

Real-Time Optimization: TQEO

A significant hurdle for deep learning in video is computational complexity. Enhancing every pixel in every frame is often impossible in real-time scenarios. Our Time-constrained Quality Enhancement Optimization (TQEO) scheme solves this by treating quality enhancement as a resource allocation problem.

TQEO predicts which Coding Tree Units (CTUs) will provide the greatest quality improvement if processed by the CNN. It uses computationally "cheap" metrics—bit allocation for I-frames and Mean Absolute Deviation (MAD) for P-frames—to rank blocks. The system then processes the highest-ranked blocks until the time budget for the current frame is exhausted, ensuring smooth playback without sacrificing the most impactful visual gains.

Key Results and Performance

Our evaluation on standard JCT-VC test sequences confirms the effectiveness of the QE-CNN framework:

  • Quality Gains: QE-CNN-I achieved a PSNR improvement of up to 0.55 dB, significantly outperforming previous state-of-the-art models like AR-CNN.
  • Efficiency: The framework provides an average BD-rate saving of 11.06%, allowing for significant bandwidth reduction without loss of perceived quality.
  • Real-Time Capability: Using a prototype equipped with four GTX 1080 GPUs, we demonstrated real-time enhancement of 720p video, proving that TQEO effectively manages the trade-off between quality and speed.

Conclusion

QE-CNN represents a significant step forward in decoder-side video enhancement. By tailoring neural architectures to the specific distortions of HEVC and implementing a smart time-management layer, we provide a practical path toward high-quality, low-bandwidth video streaming. Future research will focus on optimizing these models for mobile hardware and expanding the framework to handle color (chroma) distortions more effectively.

Summary updated

Abstract

The latest High Efficiency Video Coding (HEVC) standard has been increasingly applied to generate video streams over the Internet. However, HEVC compressed videos may incur severe quality degradation, particularly at low bit-rates. Thus, it is necessary to enhance the visual quality of HEVC videos at the decoder side. To this end, this paper proposes a Quality Enhancement Convolutional Neural Network (QE-CNN) method that does not require any modification of the encoder to achieve quality enhancement for HEVC. In particular, our QE-CNN method learns QE-CNN-I and QE-CNN-P models to reduce the distortion of HEVC I and P frames, respectively. The proposed method differs from the existing CNN-based quality enhancement approaches, which only handle intra-coding distortion and are thus not suitable for P frames. Our experimental results validate that our QE-CNN method is effective in enhancing quality for both I and P frames of HEVC videos. To apply our QE-CNN method in time-constrained scenarios, we further propose a Time-constrained Quality Enhancement Optimization (TQEO) scheme. Our TQEO scheme controls the computational time of QE-CNN to meet a target, meanwhile maximizing the quality enhancement. Next, the experimental results demonstrate the effectiveness of our TQEO scheme from the aspects of time control accuracy and quality enhancement under different time constraints. Finally, we design a prototype to implement our TQEO scheme in a real-time scenario.
BibTeX
@article{yang2017enhancing,
  author = {Ren Yang and Mai Xu and Zulin Wang and Zhenyu Guan},
  title = {{Enhancing Quality for HEVC Compressed Videos}},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
  year = {2017},
  doi = {10.1109/tcsvt.2018.2867568},
}
Figure