Spatially adaptive image compression using a tiled deep network

David Minnen, George Toderici, Michele Covell, Troy T. Chinen, Nick Johnston, Joel Shor, Sung Jin Hwang, Damien Vincent, Saurabh Singh

Abstract

Deep neural networks represent a powerful class of function approximators that can learn to compress and reconstruct images. Existing image compression algorithms based on neural networks learn quantized representations with a constant spatial bit rate across each image. While entropy coding introduces some spatial variation, traditional codecs have benefited significantly by explicitly adapting the bit rate based on local image complexity and visual saliency. This paper introduces an algorithm that combines deep neural networks with quality-sensitive bit rate adaptation using a tiled network. We demonstrate the importance of spatial context prediction and show improved quantitative (PSNR) and qualitative (subjective rater assessment) results compared to a non-adaptive baseline and a recently published image compression model based on fully-convolutional neural networks.

Introduction

Most early deep-learning-based image compression models apply a constant spatial bit rate across the entire image, regardless of local content. In contrast, traditional codecs like JPEG 2000 or BPG are spatially adaptive, allocating more bits to complex, high-detail areas and fewer to simple regions like clear skies. The challenge with state-of-the-art fully-convolutional neural networks is that their shared dependencies make it difficult to adjust the bit rate locally without affecting the rest of the image. This paper introduces a tiled deep learning architecture that bridges this gap by intelligently allocating bits where they are needed most.

Methodology

The proposed system processes images in 32x32 pixel tiles using two primary components: spatial context prediction and recurrent residual encoding.

Spatial Context Prediction

Before encoding a tile, the model analyzes previously decoded tiles to the top and left. A convolutional neural network (CNN) uses this "context" to predict the current tile's appearance, exploiting spatial redundancy. For instance, if a linear texture like a fence continues across tiles, the model predicts it rather than re-encoding it from scratch.

Spatial context prediction framework using neighboring tiles to inform the current tile's reconstruction.

Recurrent Residual Encoding

The difference between the prediction and the actual pixels—the residual—is passed to a recurrent auto-encoder based on LSTMs. This component enables two critical features:

  1. Progressive Refinement: Each iteration of the recurrent network produces additional bits and a more accurate reconstruction.
  2. Adaptive Bit Allocation: Because the encoder is recurrent, it can terminate once a specific quality target (e.g., a PSNR threshold) is reached for that specific tile.
The recurrent residual encoder architecture progressively refines image quality through multiple iterations.

To maintain efficiency, the model employs a "channel-wise, fully-connected" layer, reducing parameter count by 170x compared to standard fully-connected layers.

Experimental Setup

The model was trained on a dataset of 6 million public web images (1280 × 720 resolution). Training patches (64 × 64) were specifically selected for their high compression difficulty to ensure the model could handle complex textures. Evaluation was conducted using the Kodak image set, a standard benchmark of 24 high-quality images.

Results and Discussion

The adaptive model outperformed non-adaptive baselines and previous neural models across a range of 0.25 to 1.5 bits per pixel (bpp). Notably, it significantly outperformed JPEG at low bit rates. In subjective tests with ten human raters, the adaptive algorithm was preferred over constant bit rate versions and previous state-of-the-art neural baselines.

Key Insights

  • Tiling for Adaptation: While many researchers avoid tiling to prevent "blockiness," this work demonstrates that a tiled architecture is superior for bit rate adaptation because it isolates local decisions.
  • Contextual Continuity: The spatial context predictor ensures that tile edges mesh naturally, preventing the seams common in traditional tiled codecs.
  • Variable Rate Modeling: Using an LSTM-based encoder allows a single trained model to operate at multiple bit rates by simply varying the number of iterations.

Conclusion and Future Work

This research demonstrates that spatial adaptation is a vital component of efficient neural image compression. While the current model uses a fixed 32x32 grid, future work could explore adaptive subdivision (quadtrees) to further improve efficiency. Additionally, moving beyond PSNR toward perceptual loss functions could better align bit allocation with human visual priorities.

Summary updated

Abstract

Deep neural networks represent a powerful class of function approximators that can learn to compress and reconstruct images. Existing image compression algorithms based on neural networks learn quantized representations with a constant spatial bit rate across each image. While entropy coding introduces some spatial variation, traditional codecs have benefited significantly by explicitly adapting the bit rate based on local image complexity and visual saliency. This paper introduces an algorithm that combines deep neural networks with quality-sensitive bit rate adaptation using a tiled network. We demonstrate the importance of spatial context prediction and show improved quantitative (PSNR) and qualitative (subjective rater assessment) results compared to a non-adaptive baseline and a recently published image compression model based on fully-convolutional neural networks.
BibTeX
@inproceedings{minnen2017spatially,
  author = {David Minnen and George Toderici and Michele Covell and Troy T. Chinen and Nick Johnston and Joel Shor and Sung Jin Hwang and Damien Vincent and Saurabh Singh},
  title = {{Spatially adaptive image compression using a tiled deep network}},
  booktitle = {ICIP},
  year = {2017},
  url = {https://openreview.net/forum?id=U-OSCR4lgex},
}
Figure