Ultra Low Bitrate Learned Image Compression by Selective Detail Decoding

Hiroaki Akutsu, Akifumi Suzuki, Zhisheng Zhong, Kiyoharu AizawaWorkshops @ CVPR 2020 · OpenReview

Abstract

Neural network-based learned image compression allows for the direct use of differentiable image quality indices as loss functions, enabling end-to-end optimization of encoders and decoders. We hypothesize that images contain specific regions of high perceptual importance, such as faces and text. To address this, we propose a framework utilizing an additional selective decoder and a weighted loss function to achieve high perceptual quality at ultra-low bitrates. Our approach automatically identifies these critical regions for any input image. Experimental results demonstrate that the proposed method effectively recognizes and preserves important details more clearly than standard MS-SSIM training models.

Introduction

Traditional and early neural image compression methods often fail at "ultra-low" bitrates (below 0.15 bits per pixel), where standard metrics like Mean Squared Error (MSE) or MS-SSIM typically result in significant blurring. While Generative Adversarial Networks (GANs) can produce sharp textures by "hallucinating" details, they often suffer from "symbolic collapse," where critical information like text becomes unreadable or facial features are distorted. This research introduces a dual-decoder architecture designed to bridge this gap, maintaining GAN-driven efficiency for backgrounds while ensuring high-fidelity accuracy for essential details.

Proposed Architecture

The core of the framework is a specialized architecture that separates global structure from local detail preservation through two distinct decoding paths.

The proposed architecture featuring dual decoders ($G_m, G_s$), an entropy estimator ($H$), and a masking mechanism for composite image reconstruction.

Dual-Decoder System

  1. Main Decoder ($G_m$): Optimized with a conditional GAN loss to generate sharp, plausible global structures and textures.
  2. Selective Detail Decoder ($G_s$): Specifically trained using a weighted MS-SSIM loss to maintain high-fidelity accuracy in regions identified as perceptually important.

Masking and Composition

A learned mask ($m$) is generated to determine the pixel-wise contribution of each decoder. The final reconstructed image $\hat{x}$ is a composite: $\hat{x} = \hat{x}_s \odot m + \hat{x}_m \odot (1 - m)$. This allows the network to seamlessly blend high-accuracy details into a GAN-generated base.

Advanced Entropy Estimation

To minimize bitrate, the model employs a causal attention entropy estimator inspired by PixelSNAIL. This module captures complex spatial dependencies in the latent features, allowing for more efficient compression. Additionally, the architecture utilizes super-resolution techniques, such as PixelShuffle layers and spatial attention, to better preserve spatial dimensions compared to standard strided convolutions.

Training and Evaluation

The model was trained on the Open Images Challenge 2018 dataset, using machine-generated semantic labels for faces and text to guide the selective training phase. Crucially, these labels are only required during training; the network learns to automatically identify important regions during inference.

Evaluation was conducted on the Kodak PhotoCD and CLIC2020 datasets. On CLIC2020, the model achieved a bitrate of 0.148 bpp with an MS-SSIM of 0.9648 and a PSNR of 28.028.

Key Results and Discussion

The primary advantage of this method is its visual fidelity. By prioritizing local accuracy in "important" zones at the expense of a slight decrease in global MS-SSIM, the overall subjective quality is significantly enhanced.

A visual example demonstrating the method's ability to preserve sharp, readable text (e.g., "SIX-SHOOTER") even at ultra-low bitrates.

Visual comparisons show that the proposed method keeps faces and text sharp and readable at bitrates where standard models produce blur and pure GAN models distort symbolic information. Furthermore, the boundaries between the selective and main regions remain visually imperceptible due to the multi-scale filtering effect of the weighted loss strategy.

Conclusion and Future Work

This dual-decoder approach successfully mitigates the trade-off between sharpness and accuracy in ultra-low bitrate compression. While the selective decoder protects labeled categories, future work will focus on reducing "symbolic collapse" in non-labeled regions and decreasing the model's dependency on the quality of initial training labels. Leveraging further super-resolution advancements could also allow for even higher definition outputs from the same compressed latent representations.

Summary updated

Abstract

Neural network-based learned image compression has a special feature in that a differentiable image quality index can be used as a loss function directly, and a decoder and an encoder can be optimized by the quality index through end-to-end learning. From a perceptual view, we hypothesized that there were detailed important parts in pictures. For those parts, we applied an additional decoder and weighted loss function to achieve both low bitrate image compression and perceptual quality. Furthermore, our approach can automatically determine which region an additional decoder will take for an input image. Experiments visually showed that the proposed method can recognize important parts, such as text and faces, and we show that our method can decode images more clearly than the simple MS-SSIM training model.
BibTeX
@inproceedings{akutsu2020ultra,
  author = {Hiroaki Akutsu and Akifumi Suzuki and Zhisheng Zhong and Kiyoharu Aizawa},
  title = {{Ultra Low Bitrate Learned Image Compression by Selective Detail Decoding}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  url = {https://openreview.net/forum?id=Zy54hDw9a_Bc},
}
Figure