Compressing Weight-updates for Image Artifacts Removal Neural Networks

Yat Hong Lam, Alireza Zare, Çaglar Aytekin, Francesco Cricri, Jani Lainema, Emre Aksu, Miska M. HannukselaWorkshops @ CVPR 2019

TL;DR

This paper introduces a novel method to compress the "updates" to a neural network's weights, rather than the entire network, for image artifact removal. By fine-tuning a pre-trained network with a special loss function that encourages sparse and quantized weight updates, they achieve comparable or better image quality than traditional codecs at similar bitrates, while significantly reducing the overhead of transmitting network modifications.

What Problem Does This Paper Address?

Image compression, while essential for storage and transmission, often introduces visual artifacts like blocking and quantization. While neural networks can effectively remove these artifacts, integrating them into a traditional compression pipeline presents a challenge: how to efficiently transmit the necessary network modifications to the decoder without a prohibitive bitrate cost. Existing methods either compress the entire network (which is still too large) or don't adapt the network to the specific content being compressed. This paper tackles the problem of adapting a neural network for artifact removal at the decoder side by efficiently compressing only the changes (weight updates) needed for that specific image or set of images.

Methods

The core of the proposed method lies in a novel fine-tuning strategy for a pre-trained neural network artifact removal filter.

  1. Pre-training: A U-Net-like convolutional neural network is first pre-trained offline on a large dataset to remove compression artifacts. This network is trained using the Mean Squared Error (MSE) loss between the original and reconstructed images.
  2. Fine-tuning with Compression Objective: During the encoding process, the pre-trained network is further fine-tuned on the specific image (or a small set of similar images) to be compressed. This fine-tuning uses a combined loss function:
  3. Reconstruction Loss (MSE): To ensure the network still effectively removes artifacts.
  4. Weight-Update Compression Objective: This novel term encourages the weight-update (the difference between the fine-tuned weights and the original pre-trained weights) to be sparse (many values close to zero) and have smaller absolute values. This is achieved using a combination of a sparsity term and a term that penalizes large weight update values.
  5. Post-training Compression: The resulting fine-tuned weight-update is then compressed. This involves:
  6. Pruning: Setting weight-update values below a certain threshold to zero, further increasing sparsity.
  7. Non-uniform Quantization: Applying k-means clustering to the remaining non-zero weight-update values and storing cluster centroids and labels.
  8. Transmission: The compressed weight-update is then transmitted alongside the traditional codec's bitstream.
  9. Decoding: At the decoder, the compressed weight-update is decompressed and applied to the pre-trained network. The updated network is then used to filter the decoded image. Crucially, the network weights are reset to their pre-trained values after processing each image, ready for the next update.

Datasets

  • CLIC Dataset: Used for both training and testing. The training split contains 1632 images, and the test split contains 330 images.

Key Insights and Ideas

  • Compressing Updates, Not the Whole Network: The key insight is that for content-adaptive artifact removal, transmitting only the delta (weight-update) is far more efficient than transmitting the entire fine-tuned network. This significantly reduces the bitrate overhead.
  • Joint Fine-tuning and Compression: The paper demonstrates that by incorporating a compression objective directly into the fine-tuning loss, the resulting weight-updates are inherently more compressible. This is a more elegant solution than compressing an already-trained, dense weight-update.
  • Sparsity and Quantization for Compressibility: The specific compression objective, promoting sparsity and small values, directly targets properties that enable efficient compression techniques like pruning and quantization.
  • Adaptive Artifact Removal: The fine-tuning process allows the artifact removal network to adapt to the specific characteristics of the images being compressed, leading to better quality than a static, pre-trained network.

Key Results

The proposed method, when integrated with the VVC codec, achieved competitive results on the CLIC test dataset.

  • PSNR Improvement: For VTM encoded at 0.12 bpp, the PSNR improved by 0.21 dB with the pre-trained filter and 0.26 dB with the fine-tuned filter. For VTM encoded at 0.14 bpp, improvements were 0.17 dB and 0.22 dB, respectively.
  • Bitrate Efficiency: The method achieved comparable or slightly superior reconstruction quality to traditional codecs at similar bitrates. The weight-update itself was compressed to a very low bitrate (e.g., 0.01-0.03 bpp), fitting within the overall bitrate budget.
  • Qualitative Improvement: Figure 2 shows sample patches where the proposed finetuned method (NTCodec2019vJ2) yields visually superior results compared to the original image and the VTM-decoded image.

Limitations and Future Work

  • Fine-tuning Overhead: While the weight-update is small, the fine-tuning process itself occurs at the encoder, adding computational complexity.
  • Parameter Tuning: The compression process involves post-training pruning and quantization, requiring careful selection of threshold values and k-means cluster counts. The paper mentions finding optimal parameters through experimentation.
  • Future Work: The authors suggest that this approach of moving information from the encoded image bitstream to a neural network is a promising new direction for learned image compression. Further exploration of different network architectures, compression objectives, and integration strategies could yield even better results.

Summary updated

BibTeX
@inproceedings{lam2019compressing,
  author = {Yat Hong Lam and Alireza Zare and Çaglar Aytekin and Francesco Cricri and Jani Lainema and Emre Aksu and Miska M. Hannuksela},
  title = {{Compressing Weight-updates for Image Artifacts Removal Neural Networks}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
}
Figure