Image-Dependent Local Entropy Models for Learned Image Compression

David Minnen, George Toderici, Saurabh Singh, Sung Jin Hwang, Michele Covell

TL;DR

Researchers from Google have significantly boosted the efficiency of neural image compression by moving away from "one-size-fits-all" probability models. By introducing SLIMD (Spatially Local, Image-dependent Multinomial Dictionary), they allow the compressor to adapt to the specific textures and patterns within an image, resulting in an average bit-rate reduction of nearly 18% and up to 98% for simple images.

What Problem Does This Paper Address?

In learned image compression, an Artificial Neural Network (ANN) transforms an image into a set of "quantized codes" (discrete symbols). To actually save space, these symbols must be entropy coded—a process that uses a probability model to represent frequent symbols with fewer bits.

The standard approach has been to use a fixed, global entropy model trained on millions of images. The problem? This model assumes every part of every image follows the same statistical distribution. If you’re compressing a picture of a clear blue sky, a model trained on busy cityscapes will be highly inefficient because it expects "busy" data that isn't there. Existing neural codecs lacked a way to send "side information" that helps the decoder adapt to the specific structure of the image being processed.

Methods

The authors propose SLIMD, which introduces forward-adaptation to neural compression. Their approach relies on three main pillars:

  1. Local Modeling: Instead of one model for the whole image, they divide the quantized code tensor into a grid of spatial tiles (e.g., $16 \times 16$ pixels).
  2. Multinomial Dictionary: They pre-learn a "dictionary" of 256 different probability distributions (multinomials) using a modified K-means++ algorithm that minimizes KL-divergence.
  3. Adaptive Selection: For each tile, the encoder calculates which distribution in the dictionary results in the lowest cross-entropy (the fewest bits). It then sends the index of that distribution to the decoder as a small amount of side information.

Additionally, the system can detect if a tile is so unique that none of the dictionary models fit well. In these cases, the encoder can calculate and transmit a custom, image-specific distribution, provided the bit savings outweigh the cost of sending the new model.

Datasets

The models were evaluated on two industry-standard benchmarks:

  • Tecnick: 100 high-resolution images ($1200 \times 1200$).
  • Kodak PhotoCD: 24 images ($768 \times 512$). The baseline for comparison was a state-of-the-art ANN-based factorized entropy model.

Key Insights and Ideas

  • The Power of Side Info: Traditionally, neural compression avoided "side info" (extra data about the model) to keep the bitstream lean. This paper proves that spending a few bytes to tell the decoder how to read a specific tile pays for itself many times over in saved bits.
  • Local vs. Global: Local statistics vary wildly. While an image might look "Laplacian" (peaked at zero) on average, a specific tile might have a non-zero mean or multiple modes. SLIMD captures this "local personality."
  • Handling Low Complexity: Fixed models are notoriously bad at simple images (like a solid green square) because they "over-prepare" for complexity. By allowing the model to collapse onto a single symbol, SLIMD achieves near-perfect compression for simple regions.

Key Results

  • Rate Reduction: On the Tecnick dataset, SLIMD achieved an average 17.88% bit-rate reduction over the baseline when optimized for Mean Squared Error (MSE).
  • Perceptual Quality: When optimized for MS-SSIM (a metric closer to human vision), it saved 12.98% in bit-rate.
  • Extreme Cases: For low-complexity images, the gains were massive. A solid green image saw a 98.5% reduction in file size compared to the baseline.
  • Competitive Performance: The method makes neural compression competitive with BPG (Better Portable Graphics), one of the strongest traditional (non-neural) codecs available.

Limitations and Future Work

  • Computational Overhead: Calculating the best distribution for every tile adds complexity to the encoding process.
  • Dictionary Optimization: The dictionary is currently learned offline. The authors suggest that jointly learning the dictionary alongside the neural network transforms could lead to even better co-adaptation and higher performance.
  • Side Information Bottleneck: There is a constant trade-off between tile size and side-information cost. Smaller tiles provide better local fit but require more indices to be sent. Finding the "sweet spot" automatically for different bit-rates remains an open challenge.

Summary updated

Abstract

The leading approach for image compression with artificial neural networks (ANNs) is to learn a nonlinear transform and a fixed entropy model that are optimized for rate-distortion performance. We show that this approach can be significantly improved by incorporating spatially local, image-dependent entropy models. The key insight is that existing ANN-based methods learn an entropy model that is shared between the encoder and decoder, but they do not transmit any side information that would allow the model to adapt to the structure of a specific image. We present a method for augmenting ANN-based image coders with image-dependent side information that leads to a 17.8% rate reduction over a state-of-the-art ANN-based baseline model on a standard evaluation set, and 70-98% reductions on images with low visual complexity that are poorly captured by a fixed, global entropy model.
BibTeX
@inproceedings{minnen2018image,
  author = {David Minnen and George Toderici and Saurabh Singh and Sung Jin Hwang and Michele Covell},
  title = {{Image-Dependent Local Entropy Models for Learned Image Compression}},
  booktitle = {ICIP},
  year = {2018},
  url = {https://openreview.net/forum?id=nwmi6RSJTr},
}

Loading related content...

Figure