Channel-Wise Autoregressive Entropy Models for Learned Image Compression

David Minnen, Saurabh Singh

TL;DR

This paper introduces a novel image compression architecture that significantly improves compression efficiency by processing latent representations channel-by-channel (channel-conditioning) and predicting quantization errors (latent residual prediction). This approach achieves better rate-distortion performance than existing methods while enabling more parallelizable decoding, making it faster on modern hardware.

What Problem Does This Paper Address?

Traditional image compression methods, while effective, often struggle to achieve the best possible compression rates without sacrificing image quality. Learned image compression, using deep neural networks, has shown promise but faces a critical bottleneck: the entropy model. The most effective entropy models rely on "backward adaptation," which uses information from already decoded pixels to predict the next. While this is powerful, it requires processing symbols sequentially, making it slow on parallel hardware like GPUs. This paper aims to bridge this gap by developing a learned compression model that offers state-of-the-art compression performance while being significantly more parallelizable for faster decoding.

Methods

The authors propose two key architectural enhancements:

  • Channel-Conditioning (CC): Instead of processing the entire latent representation sequentially, the latent tensor is split into multiple "slices" along the channel dimension. Each slice's entropy model is then conditioned on the previously decoded slices. This shifts the autoregressive dependency from spatial dimensions to channel dimensions, allowing for more parallel processing within each slice.
  • Latent Residual Prediction (LRP): During the encoding process, quantization introduces errors in the latent representation. LRP aims to reduce these errors by predicting the quantization residual based on the hyperprior and previously decoded slices. This predicted residual is then added to the quantized latents, leading to a more accurate reconstruction and potentially lower entropy.

Additionally, they found that a mixed training approach, using uniform noise for learning entropy models but rounding for passing quantized tensors to the synthesis transform, further boosted performance.

Datasets

The primary dataset used for evaluation is the Kodak image set [18], a standard benchmark for image compression research. They also evaluated on the Tecnick image set [19].

Key Insights and Ideas

  • Shifting Autoregression: Moving the autoregressive dependency from spatial to channel dimensions is a clever way to leverage parallel hardware. While still sequential across slices, the processing within a slice can be highly parallelized, unlike purely spatial autoregressive models.
  • LRP's Dependence on CC: Latent Residual Prediction (LRP) shows minimal benefit on its own. Its effectiveness is significantly amplified when combined with channel-conditioning, suggesting that the structured dependencies introduced by CC create a more fertile ground for LRP to exploit.
  • Low Bitrate Advantage: The proposed enhancements, particularly LRP, are most impactful at low bitrates. This is crucial for applications where file size is paramount, such as streaming or mobile data.
  • Rounding vs. Noise: The empirical finding that a mixed training strategy (noise for entropy, rounding for synthesis) outperforms purely noise-based training highlights the subtle but important interplay between different components of the learned compression pipeline.

Key Results

The proposed channel-wise autoregressive entropy models with latent residual prediction achieve significant rate savings:

  • Average Rate Savings: 6.7% on the Kodak dataset and 11.4% on the Tecnick dataset compared to a context-adaptive baseline.
  • Low Bitrate Performance: Up to 18% savings over the baseline at low bitrates.
  • Comparison to BPG: Outperforms hand-engineered codecs like BPG by up to 25% at low bitrates.
  • Parallelism: The architecture minimizes serial processing, leading to faster decoding times on GPUs/TPUs compared to purely backward-adaptive models.

Limitations and Future Work

  • Diminishing Returns with Slices: While increasing the number of channel-conditional slices improves performance, the gains diminish, and computational costs increase.
  • Model Capacity: The authors hypothesize that the relatively simple 4-layer convolutional networks might lack the capacity to fully exploit the learned representations, or that the Adam optimizer might struggle to find the optimal minimum.
  • Future Directions:
    • Investigating combining channel-conditioning with spatial context modeling to see if they are complementary.
  • Exploring more sophisticated image quality metrics beyond MSE, L1, and MS-SSIM, such as perceptual metrics and adversarial losses.
  • Further research into understanding the information stored in the hyperprior and its role in the compression process.

Summary updated

Abstract

In learning-based approaches to image compression, codecs are developed by optimizing a computational model to minimize a rate-distortion objective. Currently, the most effective learned image codecs take the form of an entropy-constrained autoencoder with an entropy model that uses both forward and backward adaptation. Forward adaptation makes use of side information and can be efficiently integrated into a deep neural network. In contrast, backward adaptation typically makes predictions based on the causal context of each symbol, which requires serial processing that prevents efficient GPU / TPU utilization. We introduce two enhancements, channel-conditioning and latent residual prediction, that lead to network architectures with better rate-distortion performance than existing context-adaptive models while minimizing serial processing. Empirically, we see an average rate savings of 6.7% on the Kodak image set and 11.4% on the Tecnick image set compared to a context-adaptive baseline model. At low bit rates, where the improvements are most effective, our model saves up to 18% over the baseline and outperforms hand-engineered codecs like BPG by up to 25%.
BibTeX
@misc{minnen2020channel,
  author = {David Minnen and Saurabh Singh},
  title = {{Channel-Wise Autoregressive Entropy Models for Learned Image Compression}},
  year = {2020},
  eprint = {2007.08739},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.08739},
}
Figure