Fully Convolutional Network for Removing DCT Artefacts From Images

Patryk Najgebauer, Rafal Scherer, Leszek Rutkowski

TL;DR

This paper introduces three novel fully convolutional network (FCN) architectures designed to remove DCT-based compression artifacts, particularly from JPEG images. They explore the impact of initializing convolutional layers with DCT filters, finding that predefined, non-learned DCT filters can lead to slightly better and more universal artifact removal compared to fully learned networks.

What Problem Does This Paper Address?

Image compression, especially lossy compression like JPEG, is crucial for efficient storage and transmission of multimedia data. However, it inevitably leads to the loss of image details and the introduction of visual artifacts such as blockiness and ringing. These artifacts are particularly noticeable at high compression ratios and can degrade the perceived quality of images, especially in areas with high contrast. While various denoising and deblurring techniques exist, this paper focuses specifically on addressing the artifacts introduced by DCT-based compression, aiming to restore image quality without significant computational overhead.

Methods

The authors propose three Fully Convolutional Network (FCN) models for artifact reduction. All models are designed to be lightweight and efficient.

  • Model 1 & 2: These models incorporate a novel "DCT-Conv2D" layer. In Model 1, this layer is initialized with DCT filters and its weights are frozen, mimicking the JPEG compression process. Model 2 also uses this DCT-Conv2D layer but allows its weights to be trained. Both models then proceed with standard convolutional layers for artifact removal.
  • Model 3: This model is similar to Model 2 but uses standard convolutional layers throughout, without the specialized DCT-Conv2D layer.
  • ARCNN: For comparison, they use the ARCNN (Artifact Reduction Convolutional Neural Network) model, a well-established FCN for artifact removal.

The key novelty lies in the exploration of using predefined DCT filters in the initial layers, aiming to leverage the inherent structure of DCT compression for more effective artifact removal. They also investigate the trade-offs between learned filters and fixed, DCT-based filters.

Datasets

The models were trained and evaluated on the BSDS500 dataset, which contains approximately 500 natural photographs. For training, the ground truth images were compressed with a quality factor q=60 to create the input for the network.

Key Insights and Ideas

  • Predefined DCT Filters Offer Universality: The research suggests that initializing the first convolutional layer with DCT filters (as in Model 1) can lead to more universal artifact removal compared to networks where all filters are learned from scratch. This implies that the inherent structure of DCT compression can be effectively exploited by fixed filters.
  • Lightweight Networks Can Be Effective: The proposed FCN models are relatively small in terms of parameters, yet they achieve competitive results in artifact reduction, demonstrating that high performance doesn't always require massive networks.
  • PSNR vs. SSIM Sensitivity to Noise: The paper highlights that PSNR is more sensitive to noise, which is often lost during compression. While removing this noise can improve PSNR, it doesn't necessarily translate to better perceived image quality. SSIM, on the other hand, is less affected by noise and might better reflect visual similarity.
  • Artifacts are More Visible on Uniform Areas: Compression artifacts are most noticeable on smooth, uniform areas or edges with high contrast, rather than on highly textured regions.

Key Results

  • Model 1 (predefined DCT filters) achieved the best average PSNR results on the test set, outperforming Model 2 (learned DCT filters), Model 3, and ARCNN.
  • SSIM results were very close across all models, indicating that they all effectively removed block artifacts with similar efficiency.
  • The proposed models, particularly Model 1, demonstrated competitive performance with the ARCNN model, often with fewer parameters.
  • The paper shows qualitative examples where their models successfully reduce block artifacts, especially in images with uniform surfaces.

Limitations and Future Work

  • Noise Removal vs. Perceptual Quality: The paper notes that while PSNR can be improved by removing noise (which is often lost during compression), this doesn't always align with human perception. The PSNR measure can be misleading when noise is a significant factor.
  • Limited Scope of DCT: The current work focuses on DCT-based compression (JPEG). Future work could explore extending these methods to other compression techniques.
  • Dataset Limitations: While BSDS500 is a standard dataset, exploring performance on a wider variety of image types and compression levels could provide more comprehensive insights.
  • The authors suggest that further research could explore the optimal initialization strategies for DCT layers and investigate how these networks perform on different types of compression artifacts.

Summary updated

Abstract

Image compression is one of the essential methods of image processing. Its most prominent advantage is the significant reduction of image size allowing for more efficient storage and transfer. However, lossy compression is associated with the loss of some image details in favor of reducing its size. In compressed images, the deficiencies are manifested by noticeable defects in the form of artifacts; the most common are block artifacts, ringing effect, or blur. In this article, we propose three models of fully convolutional networks with different configurations and examine their abilities in reducing compression artifacts. In the experiments, we research the extent to which the results are improved for models that will process the image in a similar way to the compression algorithm, and whether the initialization with predefined filters would allow for better image reconstruction than developed solely during learning.
BibTeX
@inproceedings{najgebauer2020fully,
  author = {Patryk Najgebauer and Rafal Scherer and Leszek Rutkowski},
  title = {{Fully Convolutional Network for Removing DCT Artefacts From Images}},
  booktitle = {IJCNN},
  year = {2020},
  url = {https://openreview.net/forum?id=RZVbZhZ6WB},
}
Figure