Computationally-Efficient Neural Image Compression with Shallow Decoders

Yibo Yang, Stephan MandtICCV 2023 · OpenReview

Abstract

Neural image compression methods have seen increasingly strong performance in recent years. However, they suffer orders of magnitude higher computational complexity compared to traditional codecs, which hinders their real-world deployment. This paper takes a step forward in closing this gap in decoding complexity by adopting shallow or even linear decoding transforms. To compensate for the resulting drop in compression performance, we exploit the often asymmetrical computation budget between encoding and decoding, by adopting more powerful encoder networks and iterative encoding. We theoretically formalize the intuition behind, and our experimental results establish a new frontier in the trade-off between rate-distortion and decoding complexity for neural image compression. Specifically, we achieve rate-distortion performance competitive with the established mean-scale hyperprior architecture of Minnen et al. (2018) at less than 50K decoding FLOPs/pixel, reducing the baseline’s overall decoding complexity by 80%, or over 90% for the synthesis transform alone.

Introduction

Neural Image Compression (NIC) has recently surpassed classical codecs like JPEG and BPG in terms of rate-distortion performance. However, NIC faces a significant "speed limit": it is often orders of magnitude more computationally expensive than classical methods. This complexity is a major barrier to real-world deployment, particularly on mobile devices where fast decoding is critical.

While a slow encoder might be acceptable for one-time server-side compression, end-users require lightning-fast decoders. Most NIC models use symmetric architectures with deep convolutional neural networks (CNNs) for both encoding and decoding. This research explores an asymmetric approach: can we offload the "intelligence" to a powerful encoder and iterative optimization process, allowing for an extremely lightweight, shallow decoder?

Methodology

The authors propose several strategies to bridge the performance gap caused by simplifying the decoder:

  1. Shallow Decoders: The deep CNN decoders are replaced with two lightweight alternatives: a JPEG-like synthesis (a single linear transposed convolution layer) and a two-layer nonlinear synthesis (two transposed convolution layers with a single hidden layer and GDN activation).
  2. Asymmetric Heavy Lifting: To compensate for the simpler decoder, a state-of-the-art, powerful encoder based on the ELIC architecture is employed.
  3. Iterative Encoding (SGA): Stochastic Gumbel Annealing (SGA) is used during encoding. This iterative optimization helps the encoder find the optimal latent representation for a specific image, effectively "squeezing" more performance out of the simple decoder.
  4. Theoretical Framework: The authors decompose compression loss into the irreducible cost, the modeling gap (decoder limitations), and the inference gap (encoder limitations). They demonstrate that minimizing the inference gap through superior encoding can offset the modeling gap introduced by a simpler decoder.

Key Insights: The "Flat" Manifold

A surprising finding of this work is that the learned image manifold in NIC is remarkably "flat." In contrast to generative models like GANs where latent space interpolations are highly non-linear, the authors found that linear interpolations in the latent space map almost perfectly to linear interpolations in the pixel space. This linearity justifies the effectiveness of shallow or even purely linear decoders. Furthermore, the authors found that increasing the kernel size of a linear "JPEG-like" decoder allows it to emulate a deeper linear CNN, effectively reducing the blocking artifacts common in traditional JPEG.

Experimental Results

The models were trained on the COCO 2017 dataset and evaluated on standard benchmarks including Kodak, Tecnick, and CLIC 2018.

Rate-distortion performance (PSNR vs. BPP) on the Kodak dataset, comparing proposed models with state-of-the-art neural and classical codecs.

The results demonstrate that the 2-layer decoder with SGA encoding achieves rate-distortion performance competitive with the established "Mean-scale Hyperprior" model (Minnen et al., 2018) while remaining under 50K MACs/pixel.

BD-rate savings over BPG vs. decoding complexity (MACs/pixel) on the Kodak dataset. The proposed shallow decoders establish a new efficiency frontier.

As shown in the figures, the proposed models reduce synthesis transform complexity by over 90% and overall decoding complexity by 80% compared to the baseline. This establishes a new frontier in the trade-off between decoding speed and compression quality, outperforming classical BPG in several configurations.

Discussion and Future Work

While the shallow decoders perform excellently on PSNR, they still lag slightly behind deep models on perceptual metrics like MS-SSIM, occasionally producing "ringing" or "blocking" artifacts at very low bitrates.

Additionally, by drastically reducing the synthesis transform time, the entropy decoder (which reads the bitstream) has become the new bottleneck, accounting for 50-80% of the remaining decoding time. Future research will focus on optimizing entropy models and potentially exploiting the sparsity of latent coefficients to achieve further system-wide gains.

Summary updated

Abstract

Neural image compression methods have seen increasingly strong performance in recent years. However, they suffer orders of magnitude higher computational complexity compared to traditional codecs, which hinders their real-world deployment. This paper takes a step forward in closing this gap in decoding complexity by adopting shallow or even linear decoding transforms. To compensate for the resulting drop in compression performance, we exploit the often asymmetrical computation budget between encoding and decoding, by adopting more powerful encoder networks and iterative encoding. We theoretically formalize the intuition behind, and our experimental results establish a new frontier in the trade-off between rate-distortion and decoding complexity for neural image compression. Specifically, we achieve rate-distortion performance competitive with the established mean-scale hyperprior architecture of Minnen et al. (2018) at less than 50K decoding FLOPs/pixel, reducing the baseline’s overall decoding complexity by 80%, or over 90% for the synthesis transform alone. Our code can be found at https://github.com/mandt-lab/shallow-ntc.
BibTeX
@inproceedings{yang2023computationally,
  author = {Yibo Yang and Stephan Mandt},
  title = {{Computationally-Efficient Neural Image Compression with Shallow Decoders}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=mvQqOPUISU},
}
Figure