TL;DR
This paper introduces DiQP, a novel Transformer-Diffusion model designed to restore the visual quality of 8K videos degraded by codec compression. By treating video restoration as a denoising diffusion process and leveraging the Quantization Parameter (QP) as a conditioning signal, DiQP effectively reverses compression artifacts without introducing artificial noise, outperforming existing methods, especially for high-resolution content.
What Problem Does This Paper Address?
The increasing demand for high-resolution video, particularly 8K, presents significant challenges in terms of data transmission and storage. Video codecs are essential for compressing this data, but they inevitably introduce visual artifacts like blockiness, blurring, and ringing. These artifacts become more pronounced at higher resolutions, making restoration a difficult task. Existing video restoration methods often struggle with the complex, non-Gaussian nature of these compression artifacts, especially when dealing with the massive data volumes of 8K content. This paper aims to tackle this challenge by developing a model specifically for 8K video restoration that can effectively reverse codec-induced degradation.
Methods
The core of DiQP is a Transformer-Diffusion model. Unlike traditional diffusion models that start from pure noise, DiQP leverages the inherent "noise" introduced by compression artifacts. The model is conditioned on the Quantization Parameter (QP), which directly influences the compression level, allowing it to adapt its restoration strategy.
Key architectural components include:
- U-shaped Hierarchical Network: Inspired by U-Net architectures, this network combines an encoder and decoder with skip connections to preserve spatial-temporal context.
- Enhanced Windowed Self-Attention: Transformers are used to capture long-range dependencies, enhanced with a windowed mechanism to maintain local context.
- Look Around Module: This module uses Temporal Separable CNNs to process downsampled frames, providing surrounding context to the encoder.
- Look Ahead Module: This module processes future frames to anticipate changes and enhance temporal coherence, feeding information to the decoder.
- LOST Embedding: This novel embedding mechanism incorporates conditional information, including the diffusion step (QP), frame location, and window coordinates, to guide the restoration process.
The restoration process is framed as predicting the compression-induced noise, which is then added back to the degraded frame to reconstruct the original.
Datasets
- SEPE8K: Used for training, this dataset comprises 40 different 8K (8192 x 4320) video sequences, each 10 seconds long at 29.97 FPS.
- UVG 4K: Used for evaluation, this dataset contains 4K videos with a duration of 12 seconds.
The SEPE8K dataset was used to generate compressed videos using HEVC/H.265 and AV1 codecs with various QPs.
Key Insights and Ideas
- Restoration as Denoising Diffusion: The paper reframes video restoration not as adding noise and then denoising, but as directly reversing compression artifacts by treating them as the "noise" to be removed within a diffusion framework. This is a more direct approach to tackling compression artifacts.
- QP as a Diffusion Signal: Using the Quantization Parameter (QP) as a conditioning signal for the diffusion process is a clever way to inform the model about the severity of the compression, allowing for adaptive restoration.
- Auxiliary Modules for Context: The "Look Around" and "Look Ahead" modules are crucial for capturing broader spatial and temporal context, which is vital for high-resolution video restoration where local windows might miss important information.
- Efficiency of DDIMs: The paper draws inspiration from Denoising Diffusion Implicit Models (DDIMs) for their efficiency, suggesting that video restoration might not require the full, lengthy diffusion process of traditional DDPMs.
- 8K Specificity: The model is explicitly designed and trained for 8K resolution, addressing the unique challenges posed by the sheer volume of data at this resolution.
Key Results
- State-of-the-Art Performance: DiQP outperforms existing video restoration methods (VRT, RVRT, BasicVSR++, FTVSR) on both SEPE8K and UVG datasets.
- Significant PSNR Gains: DiQP improves PSNR by margins of 1.77 to 1.99dB on SEPE8K and 0.84 to 0.69dB on UVG compared to baselines.
- Visual Quality: Visual comparisons show DiQP generating smoother, clearer, and artifact-free high-quality frames, while other methods struggle to restore fine textures.
- Efficiency: Despite having more parameters, DiQP achieves a faster runtime compared to some baseline methods.
- Impact of Auxiliary Modules: An ablation study showed a significant 3dB PSNR gain when using the full DiQP model with the Look Ahead and Look Around modules compared to a version without them.
Limitations and Future Work
- LOST Embedding for 8K: The current design of the LOST embedding is specifically tailored for 8K resolution. Adapting it for other resolutions is identified as a challenging but important area for future work.
- Computational Cost: While DiQP is efficient, the computational demands of training and running such models, especially for 8K content, remain a consideration.
- Generalizability to Other Codecs/Artifacts: While tested on HEVC and AV1, further exploration into its performance on other codecs or different types of video degradation could be beneficial.
Summary updated