Introduction
Video compression standards like H.265/HEVC are indispensable for modern digital streaming, yet they inevitably introduce visual degradations. Common artifacts such as blocking, ringing, and blurring significantly diminish the viewing experience. Video Quality Enhancement (VQE) seeks to restore these degraded videos by leveraging information from adjacent frames. However, traditional VQE methods often struggle with motion estimation in the presence of heavy compression noise. This paper proposes a Spatio-Temporal Deformable Fusion (STDF) scheme that bypasses the limitations of traditional optical flow by using deformable convolutions to align and fuse temporal information effectively.
The Challenge of Compression Artifacts
The primary hurdle in VQE is the accurate utilization of temporal redundancy. While a detail might be obscured in the current frame, it may be clearly visible in a neighboring one. Most state-of-the-art methods rely on optical flow to align these frames. Unfortunately, compression artifacts "break" the assumptions of optical flow algorithms, leading to alignment errors that propagate through the enhancement process.
As shown in the figure above, compression introduces distinct patterns:
- Blurring: Loss of high-frequency details.
- Ringing: Oscillatory artifacts near sharp edges.
- Blocking: Discontinuities at the boundaries of transform blocks.
Methodology: Spatio-Temporal Deformable Fusion
The proposed STDF framework consists of two core components: a fusion module and a quality enhancement module.
1. Spatio-Temporal Deformable Fusion (STDF) Module
Instead of explicit motion estimation, the authors utilize Spatio-Temporal Deformable Convolution (STDC).
- Joint Offset Prediction: A U-Net-style "Offset Prediction Network" analyzes a window of frames (the target frame plus $R$ surrounding frames). Unlike previous methods that compute offsets pairwise, this network predicts sampling offsets for the entire stack simultaneously. This joint approach is not only more computationally efficient but also allows the model to capture complex motion trajectories across multiple frames.
- Deformable Sampling: The STDC operation uses these predicted offsets to sample pixels from all input frames. This allows the network to "look" at the most relevant spatial locations across time, effectively performing alignment and fusion in a single step.
2. Quality Enhancement (QE) Module
The fused feature map is passed to a Quality Enhancement module, a deep CNN designed to predict a residual map. This residual is added to the original compressed frame to produce the final, high-quality output. By focusing on learning the difference (residual) between the compressed and raw frames, the network can more easily target and remove specific artifacts.
Experimental Results
The model was trained on 106 uncompressed videos from the Xiph and VQEG databases, compressed at various Quantization Parameters (QPs). Evaluation was performed on the standard JCT-VC test set.
- Superior Accuracy: At high compression levels (QP 37), the STDF-R3 model achieved an average $\Delta$PSNR of 0.75 dB. This represents a 34% improvement over the previous state-of-the-art, MFQE 2.0, and a 70% improvement over the heavy RNAN model.
- Efficiency and Speed: By avoiding the heavy computation of optical flow and using joint prediction, the STDF framework is significantly faster than previous multi-frame approaches. A lighter version, STDF-R1, achieves 4.2 FPS at 720p resolution, making it more viable for practical applications.
- Visual Fidelity: Qualitative analysis indicates that STDF is particularly effective at restoring structural integrity and reducing ringing artifacts around edges, where traditional methods often over-smooth the image.
Conclusion and Future Directions
The STDF framework demonstrates that implicit alignment via deformable convolutions is a robust and efficient alternative to optical flow for video quality enhancement. While the current implementation focuses on the luminance channel and uses a standard CNN for the QE module, future work could involve integrating more sophisticated architectures like attention mechanisms or extending the method to handle chrominance channels for a complete end-to-end video restoration system.
Summary updated