Generative Visual Compression: A Review

Bolin Chen, Shanzhi Yin, Peilin Chen, Shiqi Wang, Yan Ye

Abstract

Artificial Intelligence Generated Content (AIGC) is leading a new technical revolution for the acquisition of digital content and impelling the progress of visual compression towards competitive performance gains and diverse functionalities over traditional codecs. This paper provides a thorough review on the recent advances of generative visual compression, illustrating great potentials and promising applications in ultra-low bitrate communication, user-specified reconstruction/filtering, and intelligent machine analysis. In particular, we review the visual data compression methodologies with deep generative models, and summarize how compact representation and high-fidelity reconstruction could be actualized via generative techniques. In addition, we generalize related generative compression technologies for machine vision and intelligent analytics. Finally, we discuss the fundamental challenges on generative visual compression techniques and envision their future research directions.

Introduction

The digital landscape is undergoing a paradigm shift driven by Artificial Intelligence Generated Content (AIGC). Traditional video compression standards, such as HEVC (H.265) and VVC (H.266), have long relied on minimizing mathematical error—specifically Mean Squared Error (MSE)—to reconstruct images. However, as we push toward ultra-low bitrate environments, these "pixel-perfect" approaches often result in severe blocking and blurring artifacts.

Generative Visual Compression (GVC) offers a transformative alternative. By leveraging deep generative models like Generative Adversarial Networks (GANs), Variational Auto-Encoders (VAEs), and Diffusion Models, GVC shifts the objective from exact pixel reconstruction to "visually-pleasing" synthesis. Instead of saving every pixel, these models learn the underlying distribution of visual data, allowing decoders to synthesize realistic textures and details that maintain perceptual integrity even at extreme compression ratios.

Methodologies in Generative Compression

The research categorizes GVC methodologies into two primary tracks based on the end-user: humans or machines.

1. Compression for Human Vision

To satisfy human observers, GVC focuses on perceptual fidelity through several innovative frameworks:

  • End-to-End Latent Representation: Utilizing VAEs, images are mapped into a compact latent space. This space is entropy-coded to ensure the most efficient representation of the data's essential features.
  • Conceptual and Cross-Modal Coding: These methods disentangle images into "structure" (geometry and edges) and "texture" (style). In some advanced cases, visual data is converted into high-level semantic maps or even text transcripts. The decoder then uses these descriptions to "re-generate" the scene, significantly reducing the bits required for transmission.
  • Temporal Evolution in Video: For video, GVC moves beyond traditional motion vectors. In applications like video conferencing, encoders may only transmit facial landmarks or keypoints. The decoder then uses a generative model to animate a high-quality reference frame of the user, achieving fluid motion at a fraction of the traditional bandwidth.

2. Compression for Machine Vision

As autonomous systems become primary consumers of visual data, "Video Coding for Machines" (VCM) has emerged as a critical field. GVC supports this through:

  • Feature Domain Analysis: Rather than reconstructing a full image for a machine to analyze, these models perform tasks like object detection or segmentation directly on the compressed bitstream or latent features. This skips the computationally expensive reconstruction step.
  • Scalable Architectures: These frameworks utilize a layered approach where a "base layer" provides enough data for machine tasks, while an optional "enhancement layer" provides the additional information necessary for human viewing.

Performance and Benchmarking

While GVC is a burgeoning field, current research highlights significant quantitative and qualitative trends. Generative models consistently outperform traditional codecs on perceptual metrics such as Learned Perceptual Image Patch Similarity (LPIPS) and Deep Image Structure and Texture Similarity (DISTS), even when they yield lower Peak Signal-to-Noise Ratio (PSNR) values.

In specialized domains like facial video coding, GVC can achieve recognizable quality at bitrates 10x to 100x lower than H.264 or H.265. Furthermore, feature-domain analysis for machine vision has demonstrated the ability to maintain over 90% task accuracy while utilizing only a small fraction of the bandwidth required for full-frame transmission.

Challenges and Future Directions

Despite its potential, GVC faces several hurdles before widespread adoption:

  • The Hallucination Problem: Because generative models synthesize details, they can occasionally introduce artifacts or alter critical information, such as the identity of a face or the text on a sign. This presents significant risks in legal, medical, or security contexts.
  • Evaluation Metrics: Traditional metrics like PSNR and SSIM are insufficient for GVC, as they penalize the very synthesis that makes GVC effective. The field requires new, standardized perceptual metrics that correlate more closely with human judgment.
  • Computational Complexity: The deployment of GANs and Diffusion Models at the decoder side requires substantial computational power, posing a challenge for mobile and edge devices.
  • Generalization: Most current GVC models are highly specialized (e.g., optimized specifically for faces or street scenes). Developing a "universal" generative codec capable of handling diverse, unconstrained environments remains a primary goal for future research.

Conclusion

Generative Visual Compression represents a fundamental shift in how we approach digital content. By treating compression as a generative task, we can achieve unprecedented efficiency and functionality. As standardization bodies like the Joint Video Experts Team (JVET) begin to investigate these technologies, GVC is poised to become a cornerstone of future communication and intelligent analysis systems.

Summary updated

Abstract

Artificial Intelligence Generated Content (AIGC) is leading a new technical revolution for the acquisition of digital content and impelling the progress of visual compression towards competitive performance gains and diverse functionalities over traditional codecs. This paper provides a thorough review on the recent advances of generative visual compression, illustrating great potentials and promising applications in ultra-low bitrate communication, user-specified reconstruction/filtering, and intelligent machine analysis. In particular, we review the visual data compression methodologies with deep generative models, and summarize how compact representation and high-fidelity reconstruction could be actualized via generative techniques. In addition, we generalize related generative compression technologies for machine vision and intelligent analytics. Finally, we discuss the fundamental challenges on generative visual compression techniques and envision their future research directions.
BibTeX
@inproceedings{chen2024generative,
  author = {Bolin Chen and Shanzhi Yin and Peilin Chen and Shiqi Wang and Yan Ye},
  title = {{Generative Visual Compression: A Review}},
  booktitle = {ICIP},
  year = {2024},
  doi = {10.1109/icip51287.2024.10647820},
}
Figure