Hierarchical Semantic Compression for Consistent Image Semantic Restoration

Shengxi Li, Zifu Zhang, Mai Xu, Lai Jiang, Yufan Liu, Ce Zhu

Introduction

Traditional image compression standards like JPEG or VVC prioritize pixel-wise accuracy, often measured by Mean Squared Error (MSE). While effective at moderate bitrates, these methods suffer from severe artifacts—blurring and blockiness—when pushed to extreme compression levels. This paper introduces Hierarchical Semantic Compression (HSC), a framework designed to maintain visual and semantic consistency at ultra-low bitrates by operating within the intrinsic semantic spaces of generative models like StyleGAN.

The HSC Framework

The HSC framework moves away from pixel-matching toward preserving the "meaning" or "concept" of an image. It leverages the pre-learned, disentangled latent space of StyleGAN through three primary components:

  1. General Inversion Encoder (GIE): This module inverts the input image into the StyleGAN space, extracting high-level core semantics ($s$) and middle-level features ($f$).
  2. Semantic Compression Network (SCN): This network compresses the core latent codes. Because these codes are highly disentangled (e.g., separating pose from hair color), they represent the image's essence with minimal data.
  3. Feature Compression Network (FCN): To capture details beyond the core semantics, the FCN compresses mid-level features using a Semantic Context Model. This model uses the high-level "concept" as a prior for channel-wise autoregressive entropy modeling, ensuring structural details are compressed efficiently.

Key Results and Perceptual Quality

HSC significantly outperforms traditional codecs and existing neural compressors in perceptual metrics at ultra-low bitrates. On the CelebA-HQ dataset, HSC achieved a Fréchet Inception Distance (FID) of 12.93, surpassing state-of-the-art methods like HiFiC.

Reconstruction quality at 0.022 bpp, showing preserved fine details in eyes and lips.

A unique strength of HSC is its "semantic floor." Even at bitrates as low as 0.003 bpp, where traditional methods fail completely, HSC maintains a recognizable and semantically consistent reconstruction by relying on the GAN's learned priors.

Extreme compression at 0.003 bpp; while pixel-level detail is lost, the semantic identity remains consistent.

Beyond human perception, HSC also excels in machine vision tasks. In face segmentation tests, it maintained high accuracy (FWIoU > 0.8) at bitrates where standard codecs' performance dropped below 0.4.

Editable Bitstreams and Future Work

Because HSC compresses images directly into a generative latent space, the resulting bitstream is "live." Users can perform semantic editing—such as adding a smile or changing hair color—by modifying the latent codes within the compressed data without full decompression.

Currently, the framework is domain-specific, requiring a pre-trained StyleGAN for each category (e.g., faces, cars). Future research aims to scale this approach to more general-purpose generative models, such as Diffusion Models, to handle complex, multi-object scenes.

Summary updated

BibTeX
@article{li2025hierarchical,
  author = {Shengxi Li and Zifu Zhang and Mai Xu and Lai Jiang and Yufan Liu and Ce Zhu},
  title = {{Hierarchical Semantic Compression for Consistent Image Semantic Restoration}},
  journal = {IEEE Trans. Image Process.},
  year = {2025},
  doi = {10.1109/tip.2025.3618379},
}
Figure