Efficient and Effective Context-Based Convolutional Entropy Modeling for Image Compression

Mu Li, Kede Ma, Jane You, David Zhang, Wangmeng Zuo

TL;DR

This paper introduces Context-based Convolutional Networks (CCNs) to significantly improve image compression by moving away from the common (but inaccurate) assumption that compressed data points are independent. By using a novel 3D zigzag scanning order and a "code dividing" technique, the authors achieve state-of-the-art compression efficiency while enabling parallel decoding—overcoming the massive speed bottleneck that usually plagues context-aware models.

What Problem Does This Paper Address?

In the world of image compression, "entropy modeling" is the art of predicting the probability distribution of data. If your model can accurately predict the next piece of data, you can store it using fewer bits.

Most modern deep-learning-based compressors simplify this by assuming that the "latent codes" (the compressed representation of the image) are statistically independent. This is called a factorized model. While fast, it’s inefficient because it ignores the spatial relationships between pixels.

Existing "context-based" models (like PixelCNN) do look at neighboring pixels to improve prediction, but they suffer from a "sequential decoding" problem: to decode pixel B, you must first finish decoding pixel A. For a high-resolution image, this makes decoding painfully slow, often taking minutes or even hours for a single file. This paper aims to provide the accuracy of context-based models with the speed required for real-world applications.

Methods

The authors propose Context-based Convolutional Networks (CCNs). The core innovation lies in how the network "sees" its data:

  1. 3D Zigzag Scanning & Masking: To ensure the model only uses "already decoded" information to predict the next value, the authors apply translation-invariant binary masks to the convolution filters. They extend this to 3D to handle multi-channel latent representations.
  2. 3D Code Dividing: This is the "secret sauce" for speed. Instead of decoding one pixel at a time, they partition the data into groups (diagonal planes in 3D space). Within each group, the pixels are assumed to be conditionally independent given the previous groups. This allows an entire group of pixels to be decoded in parallel on a GPU.
  3. Lossless vs. Lossy Pipelines:
    • Lossless: The image is converted into bit-planes, and the CCN predicts a Bernoulli distribution for each bit.
  4. Lossy: The model uses a Discretized Mixture of Gaussian (MoG) distributions. Instead of assuming a simple bell curve, it models the data as a combination of multiple Gaussians, which better captures the complex structures of natural images.

Datasets

The models were trained on a dataset of 10,000 high-quality images from Flickr, processed into over a million patches. For evaluation, the authors used two gold-standard benchmarks in the compression community:

  • Kodak: A set of 24 images widely used for comparing compression algorithms.
  • Tecnick: A larger set of 100 high-resolution images.

Key Insights and Ideas

  • Parallelism via Geometry: The jump from sequential to parallel decoding isn't just a hardware trick; it’s a mathematical one. By choosing a zigzag order, the authors found a way to group pixels such that they don't "need" each other's information simultaneously, allowing the GPU to flex its muscles.
  • The Power of MoG: Most previous works used a single Gaussian distribution to model latent codes. This paper demonstrates that real-world image data is often "multimodal" (having multiple peaks). Using a Mixture of Gaussians (MoG) allows the model to be much more precise in its probability estimations.
  • Context Matters More at Low Bit Rates: The results show that context-based modeling provides the biggest gains when trying to squeeze images into very small file sizes, where every bit of redundant spatial information counts.

Key Results

  • Lossless Compression: The CCN model outperformed traditional standards like JPEG2000-LS and BPG, achieving between 5.9% and 6.2% bit savings on the Kodak and Tecnick datasets.
  • Lossy Compression: The model achieved comparable (and sometimes superior) rate-distortion performance to state-of-the-art DNN-based compressors.
  • Speedup: This is the headline result. Compared to previous context-based models like MCN, the proposed CCN is roughly 30 times faster at decoding. A light version of the model (CCNlight) achieved even higher speeds while maintaining competitive accuracy.

Limitations and Future Work

The authors note that while their model is excellent at low to medium bit rates, the performance starts to plateau at very high bit rates. They attribute this to "limited model capability," suggesting that as the amount of information increases, the current network architecture might not be complex enough to capture all the nuances.

Future directions suggested include:

  • Scaling the network depth and width to see if the high-bit-rate plateau can be overcome.
  • Applying the CCN framework to other tasks beyond compression, such as image restoration, quality assessment, or even generative modeling (image synthesis).

Summary updated

Abstract

Precise estimation of the probabilistic structure of natural images plays an essential role in image compression. Despite the recent remarkable success of end-to-end optimized image compression, the latent codes are usually assumed to be fully statistically factorized in order to simplify entropy modeling. However, this assumption generally does not hold true and may hinder compression performance. Here we present context-based convolutional networks (CCNs) for efficient and effective entropy modeling. In particular, a 3D zigzag scanning order and a 3D code dividing technique are introduced to define proper coding contexts for parallel entropy decoding, both of which boil down to place translation-invariant binary masks on convolution filters of CCNs. We demonstrate the promise of CCNs for entropy modeling in both lossless and lossy image compression. For the former, we directly apply a CCN to the binarized representation of an image to compute the Bernoulli distribution of each code for entropy estimation. For the latter, the categorical distribution of each code is represented by a discretized mixture of Gaussian distributions, whose parameters are estimated by three CCNs. We then jointly optimize the CCN-based entropy model along with analysis and synthesis transforms for rate-distortion performance. Experiments on the Kodak and Tecnick datasets show that our methods powered by the proposed CCNs generally achieve comparable compression performance to the state-of-the-art while being much faster.
BibTeX
@article{li2020efficient,
  author = {Mu Li and Kede Ma and Jane You and David Zhang and Wangmeng Zuo},
  title = {{Efficient and Effective Context-Based Convolutional Entropy Modeling for Image Compression}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=E6I6_CZv49},
}

Loading related content...

Figure