Abstract
The High Efficiency Video Coding (HEVC) standard is the current benchmark for video compression, yet it often introduces significant visual artifacts at low bit rates. To mitigate these distortions, we propose a Quality Enhancement Convolutional Neural Network (QE-CNN) framework designed for the decoder side. Our approach utilizes two specialized models: QE-CNN-I for intra-coded frames and QE-CNN-P for inter-coded (P/B) frames. By incorporating a Time-constrained Quality Enhancement Optimization (TQEO) scheme, we ensure that the computational overhead of these deep models does not impede real-time playback. Experimental results show that QE-CNN significantly improves visual quality, achieving up to 0.55 dB PSNR gains and an average BD-rate saving of 11.06%.
Introduction
As internet video traffic continues to grow, efficient compression standards like HEVC are vital. However, aggressive compression at low bit rates leads to artifacts such as blocking, ringing, and blurring, which degrade the user experience. While post-processing techniques have long been used to "clean" video, traditional filters often struggle with the complex, non-linear distortions produced by modern codecs. Deep learning offers a powerful alternative, but existing image-based CNNs are ill-equipped to handle the temporal dependencies and motion-compensated residuals unique to video.
The QE-CNN Architecture
The core innovation of our work is the separation of quality enhancement into two distinct tasks based on frame type.
QE-CNN-I: Intra-Frame Enhancement
Intra-coded frames (I-frames) serve as spatial anchors in a video stream. While they resemble static images, HEVC intra-coding produces artifacts that are more complex than those found in JPEG compression. To capture these features, we developed QE-CNN-I, a 5-layer convolutional network.
Unlike previous 3-layer models, QE-CNN-I uses a larger number of filters (up to 128 in the first layer) and Parametric Rectified Linear Units (PReLU) to better model the distortion distribution. As seen in the learned filter weights above, QE-CNN-I captures much more complex spatial patterns compared to models trained on simpler JPEG artifacts, which is crucial for effective HEVC restoration.
QE-CNN-P: Addressing Inter-Frame Distortion
Inter-coded frames (P and B frames) introduce temporal complexity. These frames are reconstructed using motion vectors and residuals from reference frames. Consequently, their distortion is a hybrid of spatial artifacts and temporal propagation.
To address this, we propose QE-CNN-P, a 9-layer architecture that utilizes "feature concatenation." As shown in the diagram below, the model extracts features from both the current frame and the motion-compensated reference frame.
By fusing these two feature streams, QE-CNN-P can effectively denoise motion-compensated blocks, a task that standard image-restoration CNNs cannot perform. This "secret sauce" allows the network to understand how motion-compensated blocks differ from purely spatial ones.
Real-Time Optimization: TQEO
A significant hurdle for deep learning in video is computational complexity. Enhancing every pixel in every frame is often impossible in real-time scenarios. Our Time-constrained Quality Enhancement Optimization (TQEO) scheme solves this by treating quality enhancement as a resource allocation problem.
TQEO predicts which Coding Tree Units (CTUs) will provide the greatest quality improvement if processed by the CNN. It uses computationally "cheap" metrics—bit allocation for I-frames and Mean Absolute Deviation (MAD) for P-frames—to rank blocks. The system then processes the highest-ranked blocks until the time budget for the current frame is exhausted, ensuring smooth playback without sacrificing the most impactful visual gains.
Key Results and Performance
Our evaluation on standard JCT-VC test sequences confirms the effectiveness of the QE-CNN framework:
- Quality Gains: QE-CNN-I achieved a PSNR improvement of up to 0.55 dB, significantly outperforming previous state-of-the-art models like AR-CNN.
- Efficiency: The framework provides an average BD-rate saving of 11.06%, allowing for significant bandwidth reduction without loss of perceived quality.
- Real-Time Capability: Using a prototype equipped with four GTX 1080 GPUs, we demonstrated real-time enhancement of 720p video, proving that TQEO effectively manages the trade-off between quality and speed.
Conclusion
QE-CNN represents a significant step forward in decoder-side video enhancement. By tailoring neural architectures to the specific distortions of HEVC and implementing a smart time-management layer, we provide a practical path toward high-quality, low-bandwidth video streaming. Future research will focus on optimizing these models for mobile hardware and expanding the framework to handle color (chroma) distortions more effectively.
Summary updated