Flexible Neural Image Compression via Code Editing

Chenjian Gao, Tongda Xu, Dailan He, Yan Wang, Hongwei Qin

Introduction

In the realm of image compression, the ultimate goal is balancing Rate (file size) and Distortion (visual quality)—known as the Rate-Distortion (R-D) trade-off. Traditional codecs like JPEG offer immense flexibility, allowing users to dial in any quality level using a single piece of software. Modern Neural Image Compression (NIC) models, however, use deep learning to outperform JPEG but are notoriously rigid. Typically, a single trained NIC model represents just one point on the R-D curve. Achieving ten different quality levels usually requires training and storing ten distinct models.

While some "conditional" models exist that accept a quality parameter, they often demand complex training procedures, offer limited flexibility, and struggle to provide smooth, continuous control over the bitrate. To solve this, researchers have developed Code Editing, a novel method that allows a single neural image compression model to achieve variable bitrates, region-of-interest (ROI) coding, and different distortion trade-offs at inference time. By iteratively optimizing the latent code and quantization step size for each specific image, Code Editing eliminates the need to train and deploy multiple specialized decoders.

The Code Editing Methodology

The authors introduce a "semi-amortized inference" approach. Instead of relying solely on a fixed encoder to produce the compressed representation (the "latent code"), they treat the encoder's output as a starting point and then "edit" it.

  • Code Editing Naive: Given a pre-trained model optimized for a specific quality ($\lambda_0$), the method takes an image and its initial latent code. At inference time, it uses gradient descent to iteratively update this code to minimize a new R-D cost ($\lambda_1$).
  • Code Editing Enhanced: The authors discovered that "naive" editing fails when the target bitrate is far from the original. To fix this, they introduce an adaptive quantization step size ($\Delta$). By optimizing $\Delta$ alongside the latent code, they can shift the bitrate across a much wider range without the model's entropy component becoming inaccurate.
  • Stochastic Gumbel Annealing (SGA): Because the rounding process in compression is non-differentiable, they use SGA to provide a smooth approximation during the optimization process.

Because optimization happens at inference time, the objective function can be easily swapped. This allows for ROI coding (weighting distortion loss higher for specific pixels) and Multi-distortion trade-offs (balancing Mean Squared Error against perceptual metrics like LPIPS) without retraining.

Key Insights and Performance

The most significant insight from this work is the "One Decoder" Paradigm. The decoder and entropy model inherently contain enough "knowledge" to handle various bitrates; the bottleneck was the encoder's inability to adapt. By optimizing the code per-image, a single decoder becomes a universal tool. Furthermore, adding the adaptive quantization step ($\Delta$) is the key to bridging the gap between high and low bitrates effectively.

Rate-Distortion performance comparison

As shown in the performance evaluations on the Kodak dataset, Code Editing Enhanced outperforms existing variable-rate NIC methods across the entire R-D curve. Unlike previous methods that could only hit a few discrete bitrate points, Code Editing provides a perfectly smooth, continuous R-D curve. It is also highly compatible, acting as a plug-and-play upgrade that can be applied to almost any existing pre-trained NIC model with a differentiable decoder.

Speed vs. Quality Trade-off

While iterative optimization is slower than a single forward pass, the authors demonstrate that by "fine-tuning" the encoder, they can reach high-quality results in a fraction of the usual iterations.

Effect of iteration count on PSNR

By leveraging an encoder fine-tuned for the target bitrate, the model can achieve competitive R-D performance in just 10% of the usual iterations (around 200 steps), significantly mitigating the computational cost of the iterative process.

Limitations and Future Work

The primary drawback of Code Editing is encoding time. Because it requires iterative optimization (gradient descent) for every image, it is significantly slower than standard "amortized" encoders. Consequently, it is currently better suited for "encode once, decode many" scenarios—like image hosting or streaming platforms—rather than real-time communication.

Additionally, the current implementation uses a single scalar $\Delta$ for the entire image. The authors suggest that using a vector of $\Delta$ values (spatially adaptive quantization) could further improve ROI performance and expand the bitrate range.

Summary updated

Abstract

Neural image compression (NIC) has outperformed traditional image codecs in rate-distortion (R-D) performance. However, it usually requires a dedicated encoder-decoder pair for each point on R-D curve, which greatly hinders its practical deployment. While some recent works have enabled bitrate control via conditional coding, they impose strong prior during training and provide limited flexibility. In this paper we propose Code Editing, a highly flexible coding method for NIC based on semi-amortized inference and adaptive quantization. Our work is a new paradigm for variable bitrate NIC, and experimental results show that our method surpasses existing variable-rate methods. Furthermore, our approach is so flexible that it can also achieves ROI coding and multi-distortion trade-off with a single decoder. Our approach is compatible to all NIC methods with differentiable decoder NIC, and it can be even directly adopted on existing pre-trained models.
BibTeX
@inproceedings{gao2022flexible,
  author = {Chenjian Gao and Tongda Xu and Dailan He and Yan Wang and Hongwei Qin},
  title = {{Flexible Neural Image Compression via Code Editing}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=OEPWim1XCf},
}
Figure