Multi-scale and Context-adaptive Entropy Model for Image Compression

Jing ZhouWorkshops @ CVPR 2019

TL;DR

This paper introduces a novel end-to-end image compression framework that significantly improves efficiency, especially at low bitrates. It achieves this through a multi-scale, context-adaptive entropy model that better estimates latent representation distributions, coupled with clever optimizations for speed and bitrate allocation.

What Problem Does This Paper Address?

Image compression aims to reduce file sizes while preserving visual quality. A key challenge is the trade-off between rate (file size) and distortion (quality loss). Traditional methods and even earlier deep learning approaches struggle to achieve optimal compression, particularly at very low bitrates where even small improvements in rate-distortion performance are highly valuable. This paper tackles the problem of creating a more accurate and efficient entropy model, which is crucial for minimizing the rate component of this trade-off, especially when aiming for perceptually pleasing compression metrics like MS-SSIM.

Methods

The proposed framework builds upon existing autoencoder-based compression but introduces several key innovations:

  • Multi-scale Masked Convolutional Network for Autoregressive Model: Instead of a single context model, they employ a multi-scale approach using three parallel masked convolutional layers. This allows the model to capture dependencies from different spatial ranges (rings) around a latent representation, amplifying the influence of closer context and improving probability estimation.
  • Complementary Autoregressive and Hierarchical Priors: The entropy model leverages both autoregressive (predicting based on previously encoded neighbors) and hierarchical (using a hyper-prior to model the distribution of the main latent variables) priors. This combination is key to accurately estimating the probability distribution of latent representations.
  • Speed Optimizations for Entropy Model:
    • Reduced Receptive Field: The context model's computation is limited to a 7x7 receptive field, significantly reducing redundant calculations without losing information.
  • Selective Arithmetic Coding: Recognizing that many latent feature maps can be entirely zero, the model uses a flag to indicate full-zero maps and skips encoding them, saving bits and computation.
  • Bit Allocation for MS-SSIM Maximization: To meet the specific low-rate constraints of benchmarks like CLIC-2019, they employ a knapsack solver to allocate bitrates across different images from a set of pre-trained models, optimizing for MS-SSIM.

Datasets

  • Training: The network is trained on over 6000 images, including datasets from CLIC, DIV2K, and Flicker2K. Patches of 256x256 are used for training.
  • Evaluation: The primary evaluation is performed on the CLIC 2019 test dataset. The Kodak dataset is also used to generate rate-distortion curves.

Key Insights and Ideas

  • Multi-scale context is crucial: Capturing dependencies at different spatial scales significantly improves the accuracy of the entropy model, leading to better compression.
  • Exploiting sparsity in latents: Identifying and skipping zero-valued feature maps in the latent space is an effective way to reduce bitrates without sacrificing performance.
  • Bit allocation is key for specific metrics: For challenging low-rate tasks with perceptual metrics like MS-SSIM, a strategy for allocating bitrates across multiple models is more effective than a single, fixed-rate model.
  • Trade-off between MS-SSIM and PSNR: The paper highlights that optimizing for MS-SSIM can sometimes lead to slightly lower PSNR values, underscoring the perceptual nature of the chosen metric.

Key Results

  • The proposed method achieves state-of-the-art performance on the CLIC 2019 validation dataset, outperforming other baseline methods in MS-SSIM.
  • The speed-up optimizations for the autoregressive model lead to significant reductions in decoding time (e.g., over 96% for large images) without performance degradation.
  • The bit allocation strategy allows for improved MS-SSIM scores by leveraging a range of bitrates. For example, using a rate range of [0.120, 0.267] yields the best results.
  • Table 1 shows that the default MS-SSIM weights lead to higher MS-SSIM scores compared to average weights, while average weights perform better on PSNR.

Limitations and Future Work

  • Speed: While significant speed-ups are achieved, the authors acknowledge that further improvements in speed are desirable, especially for real-time applications.
  • Future Work: The authors plan to focus on further speeding up their model while simultaneously improving its performance.

Summary updated

BibTeX
@inproceedings{zhou2019multi,
  author = {Jing Zhou},
  title = {{Multi-scale and Context-adaptive Entropy Model for Image Compression}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
}
Figure