Enhancing Quality for VVC Compressed Videos by Jointly Exploiting Spatial Details and Temporal Structure

Xiandong Meng, Xuan Deng, Shuyuan Zhu, Bing Zeng

Abstract

Video compression is a fundamental trade-off between file size and visual fidelity. While the Versatile Video Coding (VVC) standard offers superior compression efficiency, it still introduces noticeable artifacts at low bitrates. This paper proposes the Spatial Details and Temporal Structure (SDTS) framework, a deep-learning-based post-processing tool designed to enhance VVC-compressed video quality. The framework consists of a temporal structure fusion subnet and a progressive spatial enhancement subnet. By leveraging motion compensation and hierarchical fusion, SDTS effectively utilizes information from adjacent frames. Furthermore, the introduction of Res_Slice_Blocks allows for adaptive feature delivery, improving spatial reconstruction. Experimental results demonstrate that SDTS significantly outperforms existing state-of-the-art methods in both quantitative metrics and visual stability.

Introduction

The transition from HEVC/H.265 to the Versatile Video Coding (VVC) standard has brought significant improvements in compression. However, at low bitrates, VVC still suffers from compression artifacts such as blockiness and ringing. Enhancing video quality is more challenging than image enhancement because it requires maintaining temporal consistency and accurately aligning moving objects across frames. Moreover, VVC often exhibits quality fluctuations between frames, which can lead to visual "flicker" if not properly addressed.

The SDTS Framework

The authors propose the SDTS-based network to jointly exploit spatial details and temporal structures. The overall architecture integrates motion compensation with spatial refinement to clean up artifacts while preserving temporal coherence.

Overall architecture of the SDTS framework, illustrating the flow from input frames through motion compensation and fusion to the enhancement network (ENet).

Temporal Structure Fusion Subnet

This component handles temporal dependencies using a Motion Compensation (MC) module. It employs a coarse-to-fine approach (scaling at x4 and x2) to estimate motion between the current frame and its neighbors. A specialized module for still scenes ensures accuracy across varying motion types. Following alignment, a "slow fusion" process merges temporal information hierarchically, avoiding the loss of detail that occurs when frames are merged too early in the network.

Spatial Detail Enhancement Subnet (ENet)

The ENet focuses on spatial reconstruction using a novel component called the Res_Slice_Block. Instead of processing all features simultaneously, this block "slices" features and delivers them progressively. This mechanism allows the network to adaptively determine which information from previous layers is most relevant for the current frame's reconstruction, leading to more refined spatial details.

Detailed architecture of the Fusion and ENet subnets. The ENet utilizes Res_Slice_Blocks to progressively refine spatial features.

Training Strategy

The model is trained in three distinct phases. First, the motion module is pre-trained using raw frames as ground truth for movement. Second, the enhancement subnet is trained. Finally, the entire system is fine-tuned end-to-end. To address quality fluctuations, the researchers trained separate models for High-Quality Frames (HQFs) and Low-Quality Frames (LQFs), allowing the network to use HQFs as reliable references for improving LQFs.

Experimental Results

The SDTS method was evaluated using standard video sequences compressed with VVC reference software (VTM3.0) in a Low-Delay P configuration at Quantization Parameters (QP) 32 and 37.

Quantitative analysis shows that SDTS consistently outperforms state-of-the-art methods like VRCNN, DCAD, and MFQE. On average, SDTS achieved a $\Delta$PSNR of 0.3262 dB at QP 37, a significant improvement over the 0.1939 dB achieved by the previous leading multi-frame method, MFQE. Notably, the framework demonstrated high stability, maintaining superior quality gains across every frame in test sequences like "BlowingBubbles," effectively reducing temporal jitter.

Conclusion

The SDTS framework provides a robust solution for enhancing VVC-compressed video by effectively combining temporal motion compensation with progressive spatial refinement. While the results are promising, future work could focus on reducing computational complexity for real-time applications and exploring the framework's performance under different VVC configurations, such as Random Access. Integrating the enhancement process directly into the VVC decoder loop also remains a potential avenue for further optimization.

Summary updated

BibTeX
@inproceedings{meng2019enhancing,
  author = {Xiandong Meng and Xuan Deng and Shuyuan Zhu and Bing Zeng},
  title = {{Enhancing Quality for VVC Compressed Videos by Jointly Exploiting Spatial Details and Temporal Structure}},
  booktitle = {ICIP},
  year = {2019},
  doi = {10.1109/icip.2019.8804469},
}
Figure