Abstract
Neural network-based learned image compression allows for the direct use of differentiable image quality indices as loss functions, enabling end-to-end optimization of encoders and decoders. We hypothesize that images contain specific regions of high perceptual importance, such as faces and text. To address this, we propose a framework utilizing an additional selective decoder and a weighted loss function to achieve high perceptual quality at ultra-low bitrates. Our approach automatically identifies these critical regions for any input image. Experimental results demonstrate that the proposed method effectively recognizes and preserves important details more clearly than standard MS-SSIM training models.
Introduction
Traditional and early neural image compression methods often fail at "ultra-low" bitrates (below 0.15 bits per pixel), where standard metrics like Mean Squared Error (MSE) or MS-SSIM typically result in significant blurring. While Generative Adversarial Networks (GANs) can produce sharp textures by "hallucinating" details, they often suffer from "symbolic collapse," where critical information like text becomes unreadable or facial features are distorted. This research introduces a dual-decoder architecture designed to bridge this gap, maintaining GAN-driven efficiency for backgrounds while ensuring high-fidelity accuracy for essential details.
Proposed Architecture
The core of the framework is a specialized architecture that separates global structure from local detail preservation through two distinct decoding paths.
Dual-Decoder System
- Main Decoder ($G_m$): Optimized with a conditional GAN loss to generate sharp, plausible global structures and textures.
- Selective Detail Decoder ($G_s$): Specifically trained using a weighted MS-SSIM loss to maintain high-fidelity accuracy in regions identified as perceptually important.
Masking and Composition
A learned mask ($m$) is generated to determine the pixel-wise contribution of each decoder. The final reconstructed image $\hat{x}$ is a composite: $\hat{x} = \hat{x}_s \odot m + \hat{x}_m \odot (1 - m)$. This allows the network to seamlessly blend high-accuracy details into a GAN-generated base.
Advanced Entropy Estimation
To minimize bitrate, the model employs a causal attention entropy estimator inspired by PixelSNAIL. This module captures complex spatial dependencies in the latent features, allowing for more efficient compression. Additionally, the architecture utilizes super-resolution techniques, such as PixelShuffle layers and spatial attention, to better preserve spatial dimensions compared to standard strided convolutions.
Training and Evaluation
The model was trained on the Open Images Challenge 2018 dataset, using machine-generated semantic labels for faces and text to guide the selective training phase. Crucially, these labels are only required during training; the network learns to automatically identify important regions during inference.
Evaluation was conducted on the Kodak PhotoCD and CLIC2020 datasets. On CLIC2020, the model achieved a bitrate of 0.148 bpp with an MS-SSIM of 0.9648 and a PSNR of 28.028.
Key Results and Discussion
The primary advantage of this method is its visual fidelity. By prioritizing local accuracy in "important" zones at the expense of a slight decrease in global MS-SSIM, the overall subjective quality is significantly enhanced.
Visual comparisons show that the proposed method keeps faces and text sharp and readable at bitrates where standard models produce blur and pure GAN models distort symbolic information. Furthermore, the boundaries between the selective and main regions remain visually imperceptible due to the multi-scale filtering effect of the weighted loss strategy.
Conclusion and Future Work
This dual-decoder approach successfully mitigates the trade-off between sharpness and accuracy in ultra-low bitrate compression. While the selective decoder protects labeled categories, future work will focus on reducing "symbolic collapse" in non-labeled regions and decreasing the model's dependency on the quality of initial training labels. Leveraging further super-resolution advancements could also allow for even higher definition outputs from the same compressed latent representations.
Summary updated