JPEG Artifacts Reduction via Deep Convolutional Sparse Coding

Xueyang Fu, Zheng-Jun Zha, Feng Wu, Xinghao Ding, John W. PaisleyICCV 2019 · OpenReview

Abstract

To effectively reduce JPEG compression artifacts, we propose a deep convolutional sparse coding (DCSC) network architecture. We design our DCSC in the framework of the classic learned iterative shrinkage-threshold algorithm (LISTA). To focus on recognizing and separating artifacts only, we sparsely code the feature maps instead of the raw image. The final de-blocked image is directly reconstructed from the coded features. We use dilated convolution to extract multi-scale image features, which allows our single model to simultaneously handle multiple JPEG compression levels. Since our method integrates model-based convolutional sparse coding with a learning-based deep neural network, the entire network structure is compact and more explainable. The resulting lightweight model generates comparable or better de-blocking results when compared with state-of-the-art methods.

Introduction

JPEG compression is a standard for image storage, but it introduces significant visual degradation at high compression ratios. These artifacts, primarily "blocking" (visible $8 \times 8$ grid boundaries) and blurring, occur because the algorithm discards high-frequency data to save space. Traditional solutions have historically fallen into two categories: model-based methods, which are mathematically grounded but computationally slow, and learning-based methods (CNNs), which are fast but often act as opaque "black boxes" requiring massive parameter counts.

This paper introduces Deep Convolutional Sparse Coding (DCSC), a hybrid framework that bridges this gap. By "unrolling" a classical sparse coding optimization algorithm into a neural network structure, the authors create a model that is both mathematically explainable and highly efficient—achieving state-of-the-art performance with 86% fewer parameters than leading competitors.

Methodology: The DCSC Architecture

The DCSC architecture is designed to be compact and interpretable, consisting of three primary stages:

  1. Multi-scale Feature Extraction: Instead of processing raw pixels directly, the network employs dilated convolutions. By increasing the receptive field without adding parameters, the model captures both fine textures and large-scale blocking patterns simultaneously.
  2. Convolutional LISTA: This is the core innovation. The authors "unroll" the iterations of the Learned Iterative Shrinkage-Thresholding Algorithm (ISTA) into layers of a recurrent neural network. Each layer effectively performs a step of sparse coding, learning to separate artifacts from the underlying image structure.
  3. Image Reconstruction: The network predicts a residual map (the artifacts themselves) which is then subtracted from the original compressed image to produce the final output.
The DCSC network architecture, featuring multi-scale feature extraction, convolutional LISTA, and image reconstruction stages.

The model is trained using Mean Absolute Error (MAE) rather than Mean Squared Error (MSE), as MAE better preserves sharp edges and avoids the "plastic" over-smoothing often seen in MSE-optimized models.

Key Insights: Explainability and Efficiency

A major advantage of DCSC is its transparency. Because the architecture follows a sparse coding logic, the intermediate feature maps are physically meaningful.

Visualizing the de-blocking process: (a) input JPEG, (b) DCSC result, (c) estimated artifacts, and (d-f) multi-scale feature maps at different dilation factors.

As shown in the visualization above, the network identifies specific features at different scales. This interpretability allows researchers to verify that the model is indeed focusing on artifacts rather than hallucinating new details. Furthermore, the mathematical structure allows for extreme efficiency; DCSC achieves results comparable to the popular DnCNN while utilizing 86.49% fewer parameters. Additionally, the multi-scale approach allows a single model to handle multiple JPEG quality levels (e.g., 10, 20, and 30) without retraining.

Experimental Results

The authors evaluated DCSC on standard benchmarks including BSD500, Classic5, LIVE1, and a real-world Twitter dataset.

  • Quantitative Performance: DCSC consistently achieved top scores in PSNR-B, a metric specifically tuned for blocking artifacts.
  • Qualitative Quality: Visual inspections confirm that DCSC preserves sharper edges and more realistic textures than competing methods, which often leave "ghosting" artifacts or smudge fine details.
  • Speed: The model is highly practical, processing a $1024 \times 1024$ image in approximately 0.3 seconds on a standard GPU.
  • Downstream Utility: Pre-processing compressed images with DCSC significantly improved the accuracy of object detection models like YOLO, demonstrating its value for automated vision systems in the wild.

Conclusion

DCSC represents a successful fusion of classical optimization and modern deep learning. While it currently operates primarily in the pixel domain, future work could integrate JPEG-specific penalties in the DCT (frequency) domain to further enhance performance. By prioritizing mathematical structure over raw depth, DCSC provides a lightweight, explainable, and high-performance solution for real-world image restoration.

Summary updated

Abstract

To effectively reduce JPEG compression artifacts, we propose a deep convolutional sparse coding (DCSC) network architecture. We design our DCSC in the framework of classic learned iterative shrinkage-threshold algorithm. To focus on recognizing and separating artifacts only, we sparsely code the feature maps instead of the raw image. The final de-blocked image is directly reconstructed from the coded features. We use dilated convolution to extract multi-scale image features, which allows our single model to simultaneously handle multiple JPEG compression levels. Since our method integrates model-based convolutional sparse coding with a learning-based deep neural network, the entire network structure is compact and more explainable. The resulting lightweight model generates comparable or better de-blocking results when compared with state-of-the-art methods.
BibTeX
@inproceedings{fu2019jpeg,
  author = {Xueyang Fu and Zheng-Jun Zha and Feng Wu and Xinghao Ding and John W. Paisley},
  title = {{JPEG Artifacts Reduction via Deep Convolutional Sparse Coding}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=-Fvizq-KJv},
}
Figure