Learning to Remove Compression Artifacts

14 concepts15 papers

Lossy image compression is the backbone of modern digital media distribution. Codecs like JPEG and BPG allow us to store and transmit vast amounts of visual data by discarding information that the human eye is less likely to notice. However, when compression ratios are high, this information loss manifests as glaring visual degradations: grid-like blocking artifacts, halo-like ringing effects around sharp edges, and a general loss of fine texture.

To combat these degradations, researchers have developed post-processing techniques that treat artifact removal not merely as a filtering problem, but as an intelligent image restoration task. The field has evolved from rigid mathematical optimization to deep, domain-aware neural networks and generative models.

The Root of the Artifacts: Quantization

To understand why restoration methods work, we must first understand why compression artifacts occur. In a standard block-based codec like JPEG, an image is divided into $8 \times 8$ pixel blocks. Each block is transformed via the Discrete Cosine Transform (DCT) into frequency coefficients.

Compression happens during quantization, where these coefficients are divided by a quality-dependent step size and rounded to the nearest integer. High-frequency coefficients—which represent fine details—are often rounded straight to zero.

This process introduces two distinct mathematical problems:

  1. Blocking: Because each $8 \times 8$ block is quantized completely independently, the continuous gradients spanning across neighboring blocks are broken. The reconstructed image exhibits sharp, artificial step-discontinuities at block boundaries.
  2. Ringing: Truncating high-frequency coefficients creates a truncation error in the frequency domain. When transformed back into pixels, this manifests as artificial oscillations or "ghost ripples" near high-contrast edges (the Gibbs phenomenon).

Constrained Optimization: Smoothing with Total Variation

Early sophisticated approaches tackled this problem by framing decompression as a constrained optimization problem rather than a simple inverse transform.

In A Total Variation-Based JPEG Decompression Model, Kristian Bredies et al. introduced a framework based on Total Variation (TV) minimization. The intuition is elegant: find an image that minimizes the total absolute gradient (which penalizes the artificial, sharp jumps at block boundaries) while ensuring that if you re-compress this restored image, its DCT coefficients still round to the exact same quantized values as the received file.

By enforcing this "quantization constraint," the algorithm ensures it does not invent arbitrary data; it merely selects the smoothest possible image that matches the compressed data. To prevent the blocky "staircasing" effect that standard TV minimization can cause in smooth regions, Bredies et al. et al. extended this framework in A TGV-Based Framework for Variational Image Decompression, Zooming, and Reconstruction. Part II: Numerics to utilize Total Generalized Variation (TGV), allowing for higher-order polynomial smoothness and much more natural transitions.

Dual-Domain Networks: Bridging Pixels and Frequencies

With the advent of deep learning, convolutional neural networks (CNNs) like the one proposed by Lukas Cavigelli et al. in CAS-CNN: A Deep Convolutional Neural Network for Image Compression Artifact Suppression demonstrated that deep architectures could learn to smooth out artifacts far more efficiently than iterative optimization frameworks. However, a pure pixel-domain CNN has to implicitly "re-learn" the mathematics of the DCT to understand why certain artifacts occur.

To bridge this gap, researchers introduced dual-domain learning. The core idea is to process the image in both the spatial (pixel) domain and the frequency (DCT) domain simultaneously.

Compressed Image ───► [ Pixel-Domain CNN ] ───┐
       │                                       ▼
       └─────────────► [  DCT-Domain CNN  ] ───► [ Residual Fusion ] ──► Restored Image

In Building Dual-Domain Representations for Compression Artifacts Reduction, Jun Guo 0024 et al. designed a network that alternates operations between these two domains. Similarly, Zhangyang Wang developed the D3: Deep Dual-Domain Based Fast Restoration of JPEG-Compressed Images model, which leverages a sparse coding formulation directly inside a neural network.

This dual approach works because each domain excels at correcting a different type of error:

  • The pixel domain provides spatial context, making it easy for the network to identify and smooth out grid-like block boundaries.
  • The DCT domain explicitly exposes the quantization errors. It tells the network exactly which frequency bins were zeroed out, allowing it to predict and restore the missing high-frequency coefficients.

To further embed this domain knowledge, Patryk Najgebauer et al. demonstrated in Fully Convolutional Network for Removing DCT Artefacts From Images that initializing the convolutional layers of a network with fixed, predefined DCT filters—rather than random weights—gives the model an inductive bias that improves universal artifact removal.

Beyond Over-Smoothing: Generative and Perceptual Restoration

Traditional networks trained to maximize metrics like Peak Signal-to-Noise Ratio (PSNR) suffer from a fundamental flaw: they optimize for the Mean Squared Error (MSE). When a network is uncertain about the exact placement of a lost high-frequency texture, minimizing MSE forces it to output the mathematical average of all possible valid textures. This results in a blurry, over-smoothed image.

To recover true visual fidelity, researchers shifted toward generative models that prioritize perceptual realism over pixel-perfect matching.

Leonardo Galteri et al. pioneered this shift in Deep Generative Adversarial Compression Artifact Removal and Deep Universal Generative Adversarial Compression Artifact Removal by utilizing Generative Adversarial Networks (GANs). In a GAN framework, a generator network learns to remove artifacts not just to match the ground truth pixels, but to fool a discriminator network trained to distinguish between restored images and real, uncompressed photographs. This adversarial pressure forces the network to hallucinate sharp, plausible high-frequency textures instead of blurring them away. Zengshun Zhao et al. followed a similar philosophy with ARGAN in Compression artifacts reduction by improved generative adversarial networks, focusing on restoring structural clarity along sharp edges.

The latest evolution in this direction leverages diffusion models. In Compression-Aware One-Step Diffusion Model for JPEG Artifact Removal, the authors introduced a method that uses a compression-aware visual embedder to condition a one-step generative diffusion process. This allows the model to dynamically adjust its restoration strength based on the severity of the compression, generating highly realistic details in a single step.

Integrating Restoration into the Compression Pipeline

While post-processing can dramatically clean up standard JPEG or BPG files (as shown by Ogun Kirmemis et al. in Learned Compression Artifact Removal by Deep Residual Networks and Yuchen Fan et al. in Wide-activated Deep Residual Networks based Restoration for BPG-compressed Images), the ultimate realization of this research direction is to integrate deep learning directly into the compression loop.

Instead of treating the codec as a black box, Feng Jiang et al. proposed an end-to-end framework in An End-to-End Compression Framework Based on Convolutional Neural Networks. Their system uses a compacting CNN before the traditional codec to downscale or optimize the image representation, and a reconstruction CNN after the codec to decode it. By optimizing both sides of the pipeline simultaneously, the network learns to pre-compensate for the artifacts the codec is about to introduce, maximizing visual quality at ultra-low bitrates.

Key Foundational Work

A TGV-Based Framework for Variational Image Decompression, Zooming, and Reconstruction. Part II: Numerics (2015) — contributed 1 concept elements to this direction.
A Total Variation-Based JPEG Decompression Model (2012) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_lossy_compression_artifacts_0c4b3ad1,
  title = {lossy compression artifacts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lossy-compression-artifacts/rdr_0c4b3ad16c8c41e4acef3a1831f463fb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_lossy_compression_artifacts_0c4b3ad1_v1,
  title = {lossy compression artifacts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lossy-compression-artifacts/rdr_0c4b3ad16c8c41e4acef3a1831f463fb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake