Coarse-to-Fine Hyper-Prior Modeling for Learned Image Compression

Yueyu Hu, Wenhan Yang, Jiaying Liu

Introduction

Image compression is a fundamental challenge in digital media, requiring a delicate balance between file size (rate) and image quality (distortion). While traditional codecs like JPEG and BPG (based on HEVC) remain industry standards, learned image compression using deep neural networks has recently begun to surpass them in performance. However, existing learned models often encounter two significant hurdles: a serial bottleneck caused by context models that predict pixels sequentially, and information loss due to aggressive downsampling and non-linear activations like ReLU.

This paper introduces a hierarchical, coarse-to-fine hyper-prior framework designed to address these limitations. By replacing slow, serial context models with a parallelizable multi-layer architecture and a novel "Signal Preserving Hyper Transform," the authors achieve state-of-the-art rate-distortion performance while significantly increasing execution speed on modern hardware.

Proposed Architecture

The core of the proposed approach is a multi-layer hierarchy that captures spatial redundancies at various scales. Unlike standard Variational Auto-Encoder (VAE) models that use a single side-information layer, this framework stacks multiple hyper-prior layers to effectively replace serial dependencies with parallelizable computations.

Overview of the proposed coarse-to-fine hyper-prior architecture.

Key Innovations

  1. Multi-Layer Coarse-to-Fine Hyper-Priors: By using multiple layers of side information, the model captures long-range spatial dependencies across different resolutions. This allows the entropy coder to estimate probabilities more accurately without the need for a slow, pixel-by-pixel context model.
  2. Signal Preserving Hyper Transform: Traditional transforms often discard fine-grained details through large kernels and ReLU activations. The authors instead utilize $1 \times 1$ convolutions and "space-to-depth" operations. This approach reduces spatial resolution while preserving the original signal information within the channel dimension.
  3. Information Aggregation Reconstruction: In a departure from standard practices where hyper-priors only assist the entropy coder, this model feeds hyper-prior data back into the final reconstruction sub-network. This provides the decoder with global structural context, leading to higher-quality image synthesis.

Experimental Results

The model was trained on the DIV2K dataset and evaluated using the Kodak and Tecnick benchmarks. The results demonstrate that the coarse-to-fine approach consistently outperforms traditional codecs and remains competitive with the most advanced serial models.

Rate-distortion performance comparison (PSNR vs Bit-rate) against traditional and learned codecs.

Performance and Efficiency

  • Superior Rate-Distortion: The proposed model consistently outperforms BPG across both PSNR and MS-SSIM metrics.
  • GPU Acceleration: Because the architecture is fully convolutional and avoids serial bottlenecks, it is significantly faster on GPUs. The authors report a decoding speedup of up to 284% compared to CPU-based context-model methods.
  • Visual Fidelity: Qualitatively, the model minimizes the "blocking" artifacts seen in JPEG and the "smearing" often found in BPG, particularly on complex textures and multi-directional edges.

Conclusion

By shifting from serial context modeling to a hierarchical, parallelizable architecture, this research provides a path toward high-efficiency, high-quality neural image compression. While the training process is multi-staged and complex, the resulting gains in speed and visual quality are substantial. Future work may extend this coarse-to-fine methodology to video compression, where hierarchical modeling can further exploit temporal redundancies.

Summary updated

Abstract

Approaches to image compression with machine learning now achieve superior performance on the compression rate compared to existing hybrid codecs. The conventional learning-based methods for image compression exploits hyper-prior and spatial context model to facilitate probability estimations. Such models have limitations in modeling long-term dependency and do not fully squeeze out the spatial redundancy in images. In this paper, we propose a coarse-to-fine framework with hierarchical layers of hyper-priors to conduct comprehensive analysis of the image and more effectively reduce spatial redundancy, which improves the rate-distortion performance of image compression significantly. Signal Preserving Hyper Transforms are designed to achieve an in-depth analysis of the latent representation and the Information Aggregation Reconstruction sub-network is proposed to maximally utilize side-information for reconstruction. Experimental results show the effectiveness of the proposed network to efficiently reduce the redundancies in images and improve the rate-distortion performance, especially for high-resolution images. Our project is publicly available at https://huzi96.github.io/coarse-to-fine-compression.html.
BibTeX
@inproceedings{hu2020coarse,
  author = {Yueyu Hu and Wenhan Yang and Jiaying Liu},
  title = {{Coarse-to-Fine Hyper-Prior Modeling for Learned Image Compression}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6736},
}
Figure