TL;DR
This paper proposes a deep learning-based post-processing method to remove compression artifacts from images encoded using the BPG codec. Instead of aiming for end-to-end learned compression, they leverage existing high-performance codecs and enhance their output, achieving impressive visual quality improvements with a modest PSNR gain.
What Problem Does This Paper Address?
Lossy image compression techniques, like JPEG and its more modern successors such as BPG (Better Portable Graphics), achieve smaller file sizes by discarding information. This process inevitably introduces visual artifacts, such as blocking, ringing, and blurring, which degrade the perceived quality of the image. While end-to-end learned image compression methods are emerging, they haven't yet surpassed the performance of established signal-processing-based codecs. This paper addresses the gap by proposing a learned post-processing approach to clean up artifacts from already compressed images, aiming to improve visual quality without reinventing the entire compression pipeline.
Methods
The core of the proposed method is a deep neural network designed for artifact removal. The authors adapt the Enhanced Deep Super-Resolution (EDSR) network, which is itself based on the SRResNet architecture. Key modifications include:
- SELU Activation: Replacing ReLU with SELU (Scaled Exponential Linear Unit) activation functions, which are claimed to enable faster learning.
- Direct Shortcut Connection: A direct connection from the input RGB image to the output RGB image is added. This is motivated by the observation that the compressed image is already a good approximation of the original, and this shortcut helps the network learn the residual difference more effectively, especially at the beginning of training. The contribution of the network is scaled by 0.1 to keep the output closer to the input.
- Residual Scaling: Employing residual scaling, as suggested in [19], to improve training stability.
- Overlap-Save Method: To manage computational and memory constraints (specifically 8GB RAM on the evaluation server), the input image is divided into four blocks. These blocks are processed independently using an "Overlap-Save" method to reconstruct the final output image, effectively simulating processing the entire image at once.
The authors trained three networks of varying depths (number of residual blocks) and widths (number of feature maps), referred to as MVGL A, MVGL B, and MVGL C.
Datasets
- Training: The paper mentions training on a set of 1633 images. Specific details about the origin or nature of these images are not provided beyond their use for training the artifact removal networks.
- Evaluation: The evaluation was performed on a test set of 286 images. The paper also refers to validation sets for reporting results.
Key Insights and Ideas
- Post-processing is a viable strategy: The paper demonstrates that instead of building a completely new learned compression system, enhancing existing high-performance codecs like BPG with learned post-processing can yield significant visual quality improvements.
- Deeper networks generally perform better: Comparing MVGL B and MVGL C, the authors found that deeper networks (MVGL A, with more blocks) generally achieve higher PSNR gains, suggesting that more complex models can better learn to remove artifacts.
- Computational constraints are critical: The trade-off between network depth/complexity and computational cost (especially for testing) is a major practical consideration, as highlighted by their submission strategy for the challenge.
- Input image proximity helps training: The direct shortcut connection from the input to the output of the network, scaled by 0.1, helps stabilize training by keeping the network's initial predictions close to the original image.
Key Results
- PSNR Improvement: The proposed method achieved an average PSNR improvement of about 0.5 dB over BPG compressed images. Specifically, MVGL A provided the highest gains, with up to 0.575 dB improvement on the test set.
- MS-SSIM Improvement: Similar gains were observed in MS-SSIM scores, indicating improved structural similarity.
- File Size Constraint Management: The authors formulated an integer programming problem to select optimal QP values for each image in the test set to meet a file size constraint (0.15 bpp), demonstrating a practical approach to balancing quality and compression.
- Visual Quality: The paper emphasizes impressive visual improvements, with Figure 4 showing clear reductions in blocking and ringing artifacts after post-processing.
- Speed vs. Quality Trade-off: MVGL A offered the best quality but was computationally expensive. MVGL B and C were significantly faster but provided smaller PSNR gains. Their submitted result (MVGL) was a hybrid approach to balance these factors.
Limitations and Future Work
- Computational Cost: The primary limitation acknowledged is the computational demand of the deep networks, particularly for testing. This prevented them from submitting their best-performing model (MVGL A) for all images in the challenge.
- Training Data: The paper does not detail the specific training dataset, which could influence the generalizability of the model to diverse image content.
- Future Work: The authors suggest that further research could explore more efficient network architectures or training strategies to mitigate the computational burden. They also note that the success of these methods depends on the availability of sufficient processing power.
Summary updated