Controlling Rate, Distortion, and Realism: Towards a Single Comprehensive Neural Image Compression Model

Shoma Iwai, Tomo Miyazaki, Shinichiro Omachi

TL;DR

Researchers from Tohoku University have developed a single neural image compression (NIC) model that offers "ultra-controllability" over bit rate, distortion, and realism. By combining variable-rate architecture with a novel adversarial loss function (HRRGAN) and specialized discriminator designs, their model matches or outperforms state-of-the-art single-rate generative models while providing a continuous range of compression settings in a single, efficient package.

What Problem Does This Paper Address?

Traditional neural image compression (NIC) models face two major hurdles for real-world deployment. First, most models are "single-rate," meaning you need a separate set of weights for every desired bit rate—a storage nightmare for mobile apps or web services. Second, models optimized purely for low distortion (like Mean Squared Error) often produce blurry images at low bit rates. While generative models (like GANs) can add realistic textures, they are notoriously difficult to train in a variable-rate framework.

Existing variable-rate generative models often struggle to match the performance of their single-rate counterparts. This paper aims to bridge that gap, creating one model to rule them all: a single network that can smoothly transition from low-bitrate "realistic" reconstructions to high-bitrate "high-fidelity" ones.

Methods

The authors built their system on a state-of-the-art foundation (ELIC and Charm) and introduced three key innovations to enable multi-dimensional control:

  1. Variable-Rate & Realism Architecture: They integrated Interpolation Channel Attention (ICA) layers to handle variable bit rates ($q$) and used $\beta$-conditioning to allow users to dial the "realism" (GAN influence) up or down at inference time.
  2. Independent Discriminators: The team explored how to design a discriminator for a model whose output quality changes constantly during training. They found that using Independent Discriminators—separate discriminator heads for different quality levels—performed best, as it allowed the system to learn quality-specific features rather than getting confused by the varying levels of compression artifacts.
  3. Higher Rate Relativistic GAN (HRRGAN): Standard GAN losses often "over-penalize" low-bitrate reconstructions for not looking exactly like the original high-res image, leading to unstable training. HRRGAN solves this by having the model compare a reconstruction at quality level $q$ against a reconstruction at a slightly higher quality level $q+1$, rather than the original image. This provides a more achievable and stable training signal.
Figure 2. The overview of our NIC model. RBs and Attn in the encoder and generator stands for residual blocks and attention module used in ELIC [23]. ICA is an interpolation channel attention layer [45] (see right side for detail). AE and AD represent an arithmetic encoder, and arithmetic decoder, respectively. For the generator, we use beta-conditioning [2] to control the realism of the reconstruction.

Datasets

  • Training: The model was trained on a subset of the OpenImage dataset, utilizing approximately 1 million images.
  • Evaluation: Performance was benchmarked using the CLIC2020 test set (428 images) and the classic Kodak dataset (24 images).

Key Insights and Ideas

  • The Discriminator Quality Gap: A major "aha" moment was realizing that a shared discriminator is a bottleneck for variable-rate models. Because a low-rate reconstruction looks fundamentally different from a high-rate one, a single discriminator struggles to provide useful feedback for both. Independent discriminators (Figure 3 in the paper) allow for much finer perceptual tuning.
  • Relative Realism: The introduction of HRRGAN (Figure 4) shifts the goalpost. Instead of telling the network "make this 0.1 bpp image look like the 24-bit original," it says "make this 0.1 bpp image look at least as realistic as this 0.2 bpp version." This prevents the generator from being overwhelmed by impossible gradients.
  • Efficiency: By using ICA and $\beta$-conditioning, the authors added only about 2.7M parameters to the base model, a negligible increase compared to the cost of storing 5-10 separate single-rate models.
Figure 4. How to calculate the relative “reality score” on (a) Ra- GAN in unconditional GAN, (b) RGAN in variable-rate image compression, and (c) HRRGAN in variable-rate image compres- sion.

Key Results

The proposed model achieves impressive quantitative and qualitative results:

  • Performance Parity: It matches or exceeds the FID (realism) and PSNR (distortion) of SOTA single-rate models like HiFiC and Multi-Realism across a wide range of bit rates (0.08 to 1.3 bpp).
  • Superiority over Variable-Rate SOTA: It significantly outperforms previous variable-rate generative models like DIRAC in both PSNR and FID.
  • Visual Quality: Qualitative results show that the model preserves complex textures (like foliage or fabric) much better than standard codecs, even at very low bit rates.
Figure 1. Top left: rate-distortion (measured by PSNR) and top right: rate-realism (measured by FID) performance using a sin- gle model. “bpp” stands for bits-per-pixel. While state-of-the- art GAN-based NIC methods, Multi-Realism [2] (capable of ad- justing distortion-realism trade-off) and HiFiC [38] are optimized to a single bit rate, our method can control the balance between rate, distortion, and realism, covering the green area with just one model. This is achieved by adjusting two input parameters, q and β, which control the rate and the distortion-realism trade-off [6], respectively. Bottom: the original image and compression results of our method. It illustrates that our method can handle different compression settings like A⃝low-rate and low-distortion mode and

Limitations and Future Work

While the model offers excellent global control over rate and realism, it currently applies these settings uniformly across the entire image. The authors acknowledge that in real-world scenarios, users might want "pixel-level" control—for example, spending more bits to keep a face sharp while allowing a background to be more heavily compressed or "hallucinated" by the GAN. Integrating spatially-adaptive control is the primary direction for their future research.

Summary updated

Abstract

In recent years, neural network-driven image compression (NIC) has gained significant attention. Some works adopt deep generative models such as GANs and diffusion models to enhance perceptual quality (realism). A critical obstacle of these generative NIC methods is that each model is optimized for a single bit rate. Consequently, multiple models are required to compress images to different bit rates, which is impractical for real-world applications. To tackle this issue, we propose a variable-rate generative NIC model. Specifically, we explore several discriminator designs tailored for the variable-rate approach and introduce a novel adversarial loss. Moreover, by incorporating the newly proposed multi-realism technique, our method allows the users to adjust the bit rate, distortion, and realism with a single model, achieving ultra-controllability. Unlike existing variable-rate generative NIC models, our method matches or surpasses the performance of state-of-the-art single-rate generative NIC models while covering a wide range of bit rates using just one model. Code will be available at https://github.com/iwa-shi/CRDR
BibTeX
@inproceedings{iwai2024controlling,
  author = {Shoma Iwai and Tomo Miyazaki and Shinichiro Omachi},
  title = {{Controlling Rate, Distortion, and Realism: Towards a Single Comprehensive Neural Image Compression Model}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure