Spatially adaptive image compression using a tiled deep network

David Minnen, George Toderici, Michele Covell, Troy T. Chinen, Nick Johnston, Joel Shor, Sung Jin Hwang, Damien Vincent, Saurabh Singh

TL;DR

This paper introduces a tiled deep learning architecture for image compression that adapts its bit rate based on local image complexity. By combining spatial context prediction (using neighboring tiles to guess the current one) with a recurrent residual encoder, the model achieves higher reconstruction quality (PSNR) and better subjective visual results than both standard JPEG and previous fixed-rate neural network models.

What Problem Does This Paper Address?

Most early deep-learning-based image compression models suffer from a "one size fits all" problem: they apply a constant spatial bit rate across the entire image. In contrast, traditional codecs like JPEG 2000 or BPG are "spatially adaptive," meaning they spend more bits on complex, high-detail areas (like a textured forest) and fewer bits on simple regions (like a clear blue sky).

The challenge with state-of-the-art fully-convolutional neural networks is that their shared dependencies make it difficult to adjust the bit rate locally without affecting the rest of the image. The authors aim to bridge this gap by creating a neural codec that can intelligently allocate bits where they are needed most.

Methods

The authors propose a block-based (tiled) approach that processes images in 32x32 pixel tiles using two main components:

  1. Spatial Context Prediction: Before encoding a tile, the model looks at the already-decoded tiles to the top and left. A convolutional neural network (CNN) uses this "context" to predict what the current tile should look like. This exploits spatial redundancy—if a fence continues from one tile to the next, the model predicts it rather than re-encoding it from scratch.
  2. Recurrent Residual Encoding: The difference between the prediction and the actual pixels (the "residual") is passed to a second network. This network uses a recurrent auto-encoder (based on LSTMs) to progressively encode the residual.
  3. Progressive Refinement: Each iteration of the recurrent network produces more bits and a more accurate reconstruction.
  4. Adaptive Bit Allocation: Because the encoder is recurrent, it can simply stop once a specific quality target (like a PSNR threshold) is reached for that specific tile. This is how the "spatial adaptation" occurs.

To keep the model efficient, they use a "channel-wise, fully-connected" layer, which reduces the parameter count by 170x compared to a standard fully-connected layer.

Datasets

  • Training: The model was trained on a massive dataset of 6 million public web images (1280 × 720 resolution). From these, 64 × 64 patches were extracted, specifically targeting patches that were difficult to compress (as measured by PNG compression) to ensure the model learned to handle complex textures.
  • Evaluation: The authors used the Kodak image set, a standard benchmark in image processing consisting of 24 high-quality images.

Key Insights and Ideas

  • Tiling as a Feature, Not a Bug: While many researchers moved toward fully-convolutional designs to avoid "blockiness," this paper argues that a tiled architecture is actually superior for bit rate adaptation because it isolates local decisions.
  • Context Prevents Artifacts: To solve the "seams" or block artifacts common in tiled codecs, the spatial context predictor ensures that the edges of a new tile "mesh" naturally with its neighbors.
  • Recurrent Networks = Variable Rate: Using an LSTM-based encoder allows a single trained model to operate at multiple bit rates. You don't need a different model for "Low Quality" vs. "High Quality"; you just run more iterations of the same network.

Key Results

  • Quantitative: The adaptive model outperformed a non-adaptive baseline and previous neural models across the entire range of 0.25 to 1.5 bits per pixel (bpp). It also significantly outperformed JPEG at low bit rates.
  • Qualitative: In subjective tests with ten human raters, the adaptive algorithm was preferred over the constant bit rate version and the previous state-of-the-art neural baseline.
  • Visual Fidelity: The model successfully reconstructed fine details (like yellow ropes or life buoys) that were lost or blurred by other methods at the same bit rate.

Limitations and Future Work

  • Fixed Tile Size: The current model uses a rigid 32x32 grid. The authors suggest that "adaptive subdivision" (using larger tiles for flat areas and smaller tiles for detail) could further improve efficiency.
  • Computational Speed: Like many deep learning models of its era, the execution speed is significantly slower than traditional codecs. The authors note that modern GPU hardware and further optimization are required for practical deployment.
  • Quality Metrics: The bit allocation was based on PSNR (Peak Signal-to-Noise Ratio), which doesn't always align with human perception. Future work could involve using more sophisticated "perceptual" loss functions to decide where to spend bits.

Summary updated

Abstract

Deep neural networks represent a powerful class of function approximators that can learn to compress and reconstruct images. Existing image compression algorithms based on neural networks learn quantized representations with a constant spatial bit rate across each image. While entropy coding introduces some spatial variation, traditional codecs have benefited significantly by explicitly adapting the bit rate based on local image complexity and visual saliency. This paper introduces an algorithm that combines deep neural networks with quality-sensitive bit rate adaptation using a tiled network. We demonstrate the importance of spatial context prediction and show improved quantitative (PSNR) and qualitative (subjective rater assessment) results compared to a non-adaptive baseline and a recently published image compression model based on fully-convolutional neural networks.
BibTeX
@inproceedings{minnen2017spatially,
  author = {David Minnen and George Toderici and Michele Covell and Troy T. Chinen and Nick Johnston and Joel Shor and Sung Jin Hwang and Damien Vincent and Saurabh Singh},
  title = {{Spatially adaptive image compression using a tiled deep network}},
  booktitle = {ICIP},
  year = {2017},
  url = {https://openreview.net/forum?id=U-OSCR4lgex},
}

Loading related content...

Figure