Deep Learning for Video Compression Artifact Reduction

45 concepts44 papers1 proposal

To fit high-definition video through narrow network pipes, modern codecs like HEVC and VVC must discard data. This is primarily achieved through quantization, a process that maps a range of continuous transform coefficients to a smaller set of discrete values. While efficient, this lossy process introduces structured visual distortions: blocking (visible grid boundaries from block-based processing), ringing (ghosting artifacts near sharp edges), and blurring (the loss of high-frequency textures).

Recent research focuses on treating these artifacts not as permanent damage, but as a restoration problem that can be solved by leveraging spatial context, temporal redundancy, and metadata hidden within the video bitstream.

Spatial Restoration and Multi-Scale Context

The most immediate way to combat artifacts is through spatial post-processing. Traditional linear filters often struggle to distinguish between actual image edges and compression-induced ringing. Convolutional Neural Networks (CNNs) solve this by learning non-linear mappings from distorted pixels to clean ones.

Wu et al. demonstrated this with VRCNN, which uses variable filter sizes to capture multi-scale residues, effectively cleaning HEVC intra-coded frames more efficiently than previous deep models. As the field has evolved, researchers have moved toward hybrid architectures. For instance, Zhang proposed SC-HVPPNet, which combines the local feature extraction of CNNs with the global context-awareness of Transformers. This hybrid approach allows the model to fix local blocking while ensuring global structural consistency across the frame.

Temporal Fusion: Borrowing from Neighbors

Video has a unique advantage over static images: the "true" detail lost in one frame might be perfectly preserved in a neighboring one. Artifacts like flicker occur when quality fluctuates between frames, a problem Meng et al. addressed by jointly exploiting spatial details and temporal structures to stabilize video quality.

Effective temporal restoration requires alignment. Because objects move between frames, a model cannot simply average pixels. Zhu et al. introduced TGAFNet, which uses a hierarchical "align-then-fuse" strategy. By grouping frames based on temporal distance, the model can borrow sharp details from both near and distant frames to erase blurring. Similarly, Xu et al. utilized Reference Frame Proposals (RFP) to identify the best source of high-frequency information, significantly improving the recovery of textures and edges.

Exploiting Coding Priors

One of the most powerful recent insights is that the video decoder already possesses "hints" about where artifacts are likely to occur. When a video is decoded, the bitstream contains coding priors—metadata such as motion vectors, quantization parameters, and residual partitions.

Instead of treating the decoded video as a raw image, Zhou proposed the CPGA network, which uses these priors to guide the restoration process. By looking at motion vectors, the model knows exactly how the codec intended for pixels to move, allowing for more accurate temporal aggregation without the heavy computational cost of estimating optical flow from scratch. Pham et al. integrated this concept directly into the codec loop with STILF, using metadata to adaptively filter spatial artifacts during the decoding process itself.

Perceptual Realism and Generative Priors

Traditional restoration aims to minimize the Mean Squared Error (MSE), which often results in "safe" but blurry outputs. To achieve visual "crispness," researchers are shifting toward perceptual metrics and generative models.

Wang et al. developed MW-GAN, which operates in the wavelet domain to specifically target high-frequency details that humans perceive as sharpness. By using a Generative Adversarial Network (GAN), the model learns to "hallucinate" realistic textures rather than just smoothing out blocks.

For extreme scenarios, such as video conferencing at ultra-low bitrates (under 20 kbps), standard codecs fail entirely. In these cases, researchers use domain-specific generative priors:

  • Talking Heads: Zhang developed DAVD-Net, which uses audio signals to predict mouth movements, helping the model restore facial details that the video stream has completely lost.
  • Animation Codecs: Konuko et al. and the authors of RDAC proposed treating video as a series of animation instructions. By sending compact 3D facial features or motion tokens and using a diffusion model to "re-animate" the scene at the decoder, these systems maintain high fidelity where traditional pixels would dissolve into blocks.

Measuring the Unseen

Finally, understanding artifacts requires better metrics. Traditional scores like PSNR often fail to capture the "staircase" nature of human perception. Wang et al. introduced VideoSet, a dataset based on Just-Noticeable Difference (JND), which helps models understand the threshold at which compression artifacts actually become visible to a human viewer. This shift toward human-centric evaluation, including specialized datasets for HDR content like LIVE-TMHDR by Venkataramanan et al., ensures that restoration efforts prioritize the artifacts that matter most to the end user.

Key Foundational Work

Joint Video Summarization and Transmission Adaptation for Energy-Efficient Wireless Video Streaming (2008) — contributed 1 concept elements to this direction.
Sparse and redundant representations and motion-estimation-free algorithm for video denoising (2007) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_video_compression_artifacts_614b1d3d,
  title = {video compression artifacts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/video-compression-artifacts/rdr_614b1d3d60864d178ce2a84ab2395678},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_video_compression_artifacts_614b1d3d_v1,
  title = {video compression artifacts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/video-compression-artifacts/rdr_614b1d3d60864d178ce2a84ab2395678/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake