Extreme Learned Image Compression with GANs

Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, Luc Van GoolWorkshops @ CVPR 2018 · OpenReview

TL;DR

This paper introduces a novel Generative Adversarial Network (GAN)-based framework for extreme image compression, achieving visually pleasing results at significantly lower bitrates than existing methods. By leveraging GANs and a multi-scale discriminator, their approach can intelligently synthesize less important image regions, leading to substantial storage savings and outperforming state-of-the-art codecs in user studies.

What Problem Does This Paper Address?

Traditional image compression algorithms, while effective, often struggle to maintain visual quality at extremely low bitrates (below 0.1 bits per pixel). At these low rates, common metrics like PSNR and MS-SSIM become less meaningful, as they penalize minor structural deviations rather than overall perceptual realism. This paper tackles the challenge of achieving high visual quality at these extreme compression levels, where preserving fine details becomes less critical than generating a perceptually convincing image. They aim to push the boundaries of learned image compression by exploring how GANs can be used to generate realistic image content, especially in regions where exact pixel preservation is too costly.

Methods

The core of this work is a GAN-based framework for learned image compression. The authors propose two modes of operation:

  • Global Generative Compression (GC): This mode focuses on preserving the overall image content while using the GAN to generate finer structures. The encoder compresses the image into a latent representation, which is then fed to a generator (decoder) that reconstructs the image. A multi-scale discriminator is used to ensure the generated images are realistic. The objective function balances a distortion term (measuring the difference between the original and reconstructed image) with a GAN loss and an entropy term (to control the bitrate).
  • Selective Generative Compression (SC): This more advanced mode allows for generating entire image regions from a semantic label map. This means only the important regions (e.g., people in a video call) need to be preserved with high fidelity, while background elements (streets, trees) can be synthesized from semantic information. This significantly reduces the amount of data that needs to be stored.

A key innovation is the generator/decoder operating on the full-resolution image and being trained with a multi-scale discriminator, which helps in capturing both global and local image characteristics.

Datasets

The authors trained and evaluated their models on two publicly available datasets with hand-annotated semantic label maps:

  • Cityscapes: A dataset of urban street scenes.
  • ADE20k: A dataset containing a wide variety of scenes and objects.

They also evaluated their model's generalization capabilities on the Kodak image dataset, a standard benchmark for image compression.

Key Insights and Ideas

  • GANs for Perceptual Quality at Low Bitrates: The paper demonstrates that GANs are a powerful tool for achieving perceptually pleasing images at extreme compression rates, where traditional metrics fail.
  • Synthesizing Unimportant Regions: The ability to synthesize less critical image regions (like backgrounds) from semantic information is a game-changer for achieving significant storage savings.
  • Multi-Scale Discriminator for Realism: Employing a multi-scale discriminator is crucial for ensuring that the generated images are realistic across different levels of detail.
  • User Studies as the Ultimate Metric: The authors highlight the importance of user studies for evaluating perceptual quality at low bitrates, as objective metrics can be misleading.

Key Results

  • Significant Bitrate Reduction: The proposed GC method significantly outperforms state-of-the-art methods like BPG, achieving up to a 67% bitrate saving in user studies.
  • Superior Visual Quality: User studies consistently showed that images produced by their GAN-based approach were preferred over those from BPG and AEDC, even when BPG used more than double the bits.
  • Generalization: The model trained on ADE20k demonstrated good generalization to the Kodak dataset, producing preferred results over BPG.
  • Qualitative Improvements: The GC method produces images with finer structure and fewer artifacts (like blocking) compared to BPG.

Limitations and Future Work

  • Color Shift in Generalization: When generalizing to the Kodak dataset, a slight color shift was observed, which the authors suggest could be addressed by reducing domain mismatch or increasing the weight of perceptual loss.
  • Focus on GC: The paper primarily focuses on the Global Generative Compression (GC) mode, with Selective Generative Compression (SC) described but not fully evaluated within the paper itself. The full version [2] is referenced for a more comprehensive evaluation of SC.
  • Semantic Label Map Requirement for SC: The SC mode requires a semantic/instance label map, which adds a small overhead for storage and requires a separate segmentation step. Future work could explore more efficient ways to obtain or integrate this information.

Summary updated

Abstract

We propose a framework for extreme learned image compression based on Generative Adversarial Networks (GANs), obtaining visually pleasing images at significantly lower bitrates than previous methods. This is made possible through our GAN formulation of learned compression combined with a generator/decoder which operates on the full-resolution image and is trained in combination with a multi-scale discriminator. Additionally, our method can fully synthesize unimportant regions in the decoded image such as streets and trees from a semantic label map extracted from the original image, therefore only requiring the storage of the preserved region and the semantic label map. A user study confirms that for low bitrates, our approach significantly outperforms state-of-the-art methods, saving up to 67% compared to the next-best method BPG.
BibTeX
@inproceedings{agustsson2018extreme,
  author = {Eirikur Agustsson and Michael Tschannen and Fabian Mentzer and Radu Timofte and Luc Van Gool},
  title = {{Extreme Learned Image Compression with GANs}},
  booktitle = {Workshops @ CVPR},
  year = {2018},
  url = {https://openreview.net/forum?id=BkWy-yf_WS},
}
Figure