TL;DR
Researchers from Tohoku University have developed a single neural image compression (NIC) model that offers "ultra-controllability" over bit rate, distortion, and realism. By combining variable-rate architecture with a novel adversarial loss function (HRRGAN) and specialized discriminator designs, their model matches or outperforms state-of-the-art single-rate generative models while providing a continuous range of compression settings in a single, efficient package.
What Problem Does This Paper Address?
Traditional neural image compression (NIC) models face two major hurdles for real-world deployment. First, most models are "single-rate," meaning you need a separate set of weights for every desired bit rate—a storage nightmare for mobile apps or web services. Second, models optimized purely for low distortion (like Mean Squared Error) often produce blurry images at low bit rates. While generative models (like GANs) can add realistic textures, they are notoriously difficult to train in a variable-rate framework.
Existing variable-rate generative models often struggle to match the performance of their single-rate counterparts. This paper aims to bridge that gap, creating one model to rule them all: a single network that can smoothly transition from low-bitrate "realistic" reconstructions to high-bitrate "high-fidelity" ones.
Methods
The authors built their system on a state-of-the-art foundation (ELIC and Charm) and introduced three key innovations to enable multi-dimensional control:
- Variable-Rate & Realism Architecture: They integrated Interpolation Channel Attention (ICA) layers to handle variable bit rates ($q$) and used $\beta$-conditioning to allow users to dial the "realism" (GAN influence) up or down at inference time.
- Independent Discriminators: The team explored how to design a discriminator for a model whose output quality changes constantly during training. They found that using Independent Discriminators—separate discriminator heads for different quality levels—performed best, as it allowed the system to learn quality-specific features rather than getting confused by the varying levels of compression artifacts.
- Higher Rate Relativistic GAN (HRRGAN): Standard GAN losses often "over-penalize" low-bitrate reconstructions for not looking exactly like the original high-res image, leading to unstable training. HRRGAN solves this by having the model compare a reconstruction at quality level $q$ against a reconstruction at a slightly higher quality level $q+1$, rather than the original image. This provides a more achievable and stable training signal.
Datasets
- Training: The model was trained on a subset of the OpenImage dataset, utilizing approximately 1 million images.
- Evaluation: Performance was benchmarked using the CLIC2020 test set (428 images) and the classic Kodak dataset (24 images).
Key Insights and Ideas
- The Discriminator Quality Gap: A major "aha" moment was realizing that a shared discriminator is a bottleneck for variable-rate models. Because a low-rate reconstruction looks fundamentally different from a high-rate one, a single discriminator struggles to provide useful feedback for both. Independent discriminators (Figure 3 in the paper) allow for much finer perceptual tuning.
- Relative Realism: The introduction of HRRGAN (Figure 4) shifts the goalpost. Instead of telling the network "make this 0.1 bpp image look like the 24-bit original," it says "make this 0.1 bpp image look at least as realistic as this 0.2 bpp version." This prevents the generator from being overwhelmed by impossible gradients.
- Efficiency: By using ICA and $\beta$-conditioning, the authors added only about 2.7M parameters to the base model, a negligible increase compared to the cost of storing 5-10 separate single-rate models.
Key Results
The proposed model achieves impressive quantitative and qualitative results:
- Performance Parity: It matches or exceeds the FID (realism) and PSNR (distortion) of SOTA single-rate models like HiFiC and Multi-Realism across a wide range of bit rates (0.08 to 1.3 bpp).
- Superiority over Variable-Rate SOTA: It significantly outperforms previous variable-rate generative models like DIRAC in both PSNR and FID.
- Visual Quality: Qualitative results show that the model preserves complex textures (like foliage or fabric) much better than standard codecs, even at very low bit rates.
Limitations and Future Work
While the model offers excellent global control over rate and realism, it currently applies these settings uniformly across the entire image. The authors acknowledge that in real-world scenarios, users might want "pixel-level" control—for example, spending more bits to keep a face sharp while allowing a background to be more heavily compressed or "hallucinated" by the GAN. Integrating spatially-adaptive control is the primary direction for their future research.
Summary updated