TL;DR
This paper introduces a novel "Multi-Frame Deformable Look-Up Table" (MF-DLUT) approach for Compressed Video Quality Enhancement (CVQE). By efficiently leveraging temporal information and deformable convolutions, their method achieves state-of-the-art performance in low-latency CVQE, outperforming existing LUT-based methods and rivaling CNN-based approaches while maintaining real-time processing speeds.
What Problem Does This Paper Address?
The ever-increasing volume of video data necessitates efficient compression techniques like H.264/AVC and H.265/HEVC. However, these compression methods often introduce visual artifacts (blurring, ringing, blocking), degrading the Quality of Experience (QoE), especially in low bit-rate or low-latency scenarios. While deep learning methods have shown promise in Compressed Video Quality Enhancement (CVQE), many suffer from high computational complexity, hindering their real-time application. Recently, Look-Up Table (LUT)-based methods have emerged as a more efficient alternative, but existing LUT approaches often focus on single frames and struggle to effectively utilize temporal information for CVQE. This paper aims to bridge this gap by proposing an efficient, multi-frame LUT-based method that can achieve high-quality enhancement with low latency.
Methods
The proposed method, MF-DLUT, is a hybrid CNN-LUT structure designed for efficient CVQE. It comprises three main components:
- Motion Feature Modulation Network (MFMN): This lightweight CNN takes multiple low-quality frames and their corresponding motion vectors (MVs) as input. It extracts multi-frame information and uses MVs to predict multi-scale convolution offsets. These offsets are crucial for guiding precise temporal alignment and feature extraction.
- Temporal Feature Extraction Module (TFEM): This module utilizes the predicted offsets from MFMN to sample features from the input frames, effectively aligning them temporally. It then extracts multi-frame features.
- Multi-Scale Fusion Module (MSFM): This module fuses the multi-scale features extracted by TFEM to reconstruct high-quality video frames.
The key innovation lies in converting the TFEM and MSFM into LUTs during the inference phase. This conversion significantly reduces computational complexity, enabling real-time performance. The method leverages deformable convolution concepts to dynamically adjust the sampling locations based on learned offsets, allowing for more accurate feature extraction and alignment across frames. The LUT conversion process involves quantization and interpolation to manage memory and computational costs.
Datasets
- Training: CVCP dataset, consisting of 589 video clips with 32 frames each at 1080p resolution.
- Testing: 18 standard test sequences from JCT-VC, with frame counts ranging from 150 to 600. These sequences were compressed using H.265/HEVC reference software (HM16.20) at four different Quantization Parameters (QPs): 22, 27, 32, and 37, to evaluate performance across various compression levels. Motion vectors were extracted during the decoding process and stored.
Key Insights and Ideas
- Deformable Convolutions for Temporal Alignment: The paper effectively demonstrates how learning offsets for deformable convolutions can be used to precisely align features across multiple frames, a critical step for multi-frame video enhancement.
- LUT Conversion for Real-Time Performance: The core idea of converting complex CNN modules (TFEM and MSFM) into LUTs is a powerful strategy for achieving significant speedups without substantial performance degradation. This is particularly relevant for low-latency applications.
- Motion Vectors as a Modulator: Incorporating motion vectors into the feature modulation process allows the network to better understand inter-frame motion and adapt its feature extraction accordingly, leading to improved quality.
- Hybrid CNN-LUT Approach: The combination of a lightweight CNN for offset prediction and LUTs for feature extraction and fusion offers a compelling balance between model expressiveness and computational efficiency.
Key Results
- Performance: The proposed MF-DLUT method significantly outperforms existing LUT-based CVQE methods. It also achieves competitive performance compared to state-of-the-art CNN-based multi-frame methods.
- Speed: The method is capable of running in real-time (30fps) at 1080p resolution, a significant advantage over many existing CVQE solutions. For instance, at 1080p, the TFEM module takes only 7.4ms and the MSFM module takes 0.5ms, compared to a full CNN module taking 217.8ms.
- Quantitative Metrics: Table 1 shows that the proposed method achieves an average ∆PSNR of 0.44 dB and an average ∆SSIM of 1.01 × 10⁻², outperforming other LUT-based methods. Table 2 further highlights its real-time capabilities across different resolutions.
- Qualitative Results: Figure 5 demonstrates that the MF-DLUT method produces visually superior results, with fewer artifacts like blurring, ringing, and blocking, and better retrieval of details compared to other methods.
Limitations and Future Work
- LUT Size and Quantization: While the paper addresses LUT size through quantization and interpolation, managing the size of LUTs for very large kernel sizes or high-dimensional inputs remains a challenge. The paper mentions a 4-bit quantization for 4D LUTs in the MSFM.
- Generalizability: The paper focuses on CVQE. The authors suggest that their proposed method can be extended to other low-level video tasks, implying that further research could explore its applicability in areas like video super-resolution or deblurring.
- Training Strategy: The paper details a three-stage training strategy (CNN training, sampling finetuning, LUT-convert finetuning) to mitigate performance loss during LUT conversion. Further research could explore more streamlined or end-to-end training approaches for the LUT conversion process.
Summary updated