Flexible Neural Image Compression via Code Editing

Chenjian Gao, Tongda Xu, Dailan He, Hongwei Qin, Yan Wang

Abstract

Neural image compression (NIC) has outperformed traditional image codecs in rate-distortion (R-D) performance. However, it usually requires a dedicated encoder-decoder pair for each point on the R-D curve, which greatly hinders its practical deployment. While some recent works have enabled bitrate control via conditional coding, they impose strong priors during training and provide limited flexibility. In this paper, we propose Code Editing, a highly flexible coding method for NIC based on semi-amortized inference and adaptive quantization. Our work represents a new paradigm for variable bitrate NIC. Furthermore, experimental results show that our method surpasses existing variable-rate methods and achieves ROI coding and multi-distortion trade-offs with a single decoder.

Introduction

In the landscape of image compression, the fundamental challenge is the trade-off between the "rate" (the number of bits used) and "distortion" (the loss of image quality), typically governed by a Lagrange multiplier $\lambda$. Traditional codecs like JPEG or BPG handle this trade-off with ease: a single decoder can process bitstreams generated at various quality settings by simply adjusting quantization parameters.

In contrast, Neural Image Compression (NIC) models have historically been "stiff." Standard architectures require training a specific encoder-decoder pair for every single point on the R-D curve. While "variable-rate" neural models have been proposed, they often rely on complex conditional networks that are difficult to train or provide limited flexibility in navigating the R-D space. This research introduces "Code Editing," a method that brings the "one decoder for all" flexibility of traditional standards to high-performance neural codecs.

The Code Editing Paradigm

The authors shift the complexity of bitrate control from the training phase to the inference (encoding) phase. This approach allows a single pre-trained decoder to remain fixed while the encoder adaptively optimizes the latent representation for specific goals.

1. Semi-Amortized Inference

Rather than relying solely on the output of a fixed encoder, Code Editing treats the initial encoder output as a starting point. At the time of compression, the system "edits" the latent code $y$ by iteratively optimizing it to minimize a target loss function. This allows the encoder to adapt to a new $\lambda$ or a specific Region of Interest (ROI) map without changing the decoder weights.

2. Adaptive Quantization ($\Delta$)

A significant challenge in editing latent codes is the potential mismatch between the edited distribution and the expectations of the pre-trained entropy model. If the code is edited too aggressively, the entropy model becomes inaccurate, leading to poor compression. The authors solve this by introducing an adaptive quantization step size ($\Delta$). By scaling the quantization, they can shift the bitrate over a wide range (approximately 0.1 to 1.1 bpp) while ensuring the underlying continuous density remains compatible with the fixed decoder.

3. Stochastic Gumbel Annealing (SGA)

Because the rounding and quantization processes are non-differentiable, they typically break standard gradient descent. To overcome this, the authors utilize Stochastic Gumbel Annealing (SGA) to provide a differentiable surrogate during the optimization process, allowing for effective gradient-based "editing" of the discrete latent codes.

Experimental Setup and Results

The researchers trained their models on a subset of 8,000 images from ImageNet and evaluated performance using the classic Kodak dataset and the CLIC2022 test set.

Performance Benchmarks

Code Editing was applied to several baseline architectures, including Ballé 2018, Minnen 2018, and Cheng 2020. The results demonstrated that "Code Editing Enhanced" consistently outperforms existing state-of-the-art variable-rate methods.

Beyond Bitrate: ROI and Multi-Distortion

One of the most compelling aspects of Code Editing is its versatility:

  • ROI Coding: The method allows for spatial bit allocation. By weighting the distortion loss with an ROI map, the encoder can prioritize high fidelity for specific objects (like faces or text) while allowing the background to remain at a lower bitrate. This is achieved without the model ever having seen these specific ROI shapes during training.
  • Multi-Distortion Trade-offs: A single decoder can navigate the trade-off between mathematical fidelity (PSNR) and perceptual quality (LPIPS). By simply adjusting the weights in the editing loss function, the encoder can produce bitstreams optimized for different visual priorities.

Discussion and Limitations

The primary insight of this work is that the decoder and entropy model act as an "anchor" or a prior. As long as the latent code is edited to satisfy that prior while meeting new R-D goals, the decoder does not need to be modified. To improve efficiency, the authors found that fine-tuning the encoder at a high-bitrate objective provides a superior "starting guess," allowing the system to achieve high-quality results with only 10% of the full optimization iterations.

However, the iterative nature of Code Editing means that encoding speed is significantly slower than standard feed-forward neural encoders. While this makes the method less ideal for real-time applications like video conferencing, it is highly suitable for "encode once, view many" scenarios, such as image hosting platforms and Content Delivery Networks (CDNs).

Conclusion

Code Editing represents a significant step toward making neural image compression practical for real-world deployment. By enabling continuous variable bitrate, ROI coding, and perceptual trade-offs within a single fixed decoder, it bridges the gap between the high performance of deep learning and the operational flexibility of traditional image codecs.

Summary updated

Abstract

Neural image compression (NIC) has outperformed traditional image codecs in rate-distortion (R-D) performance. However, it usually requires a dedicated encoder-decoder pair for each point on R-D curve, which greatly hinders its practical deployment. While some recent works have enabled bitrate control via conditional coding, they impose strong prior during training and provide limited flexibility. In this paper we propose Code Editing, a highly flexible coding method for NIC based on semi-amortized inference and adaptive quantization. Our work is a new paradigm for variable bitrate NIC. Furthermore, experimental results show that our method surpasses existing variable-rate methods, and achieves ROI coding and multi-distortion trade-off with a single decoder.
BibTeX
@inproceedings{gao2022flexible,
  author = {Chenjian Gao and Tongda Xu and Dailan He and Hongwei Qin and Yan Wang},
  title = {{Flexible Neural Image Compression via Code Editing}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=OEPWim1XCf},
}
Figure