Perceptual Fidelity in Extreme Image Compression

38 concepts39 papers1 proposal

The Breakdown of Signal Reconstruction

Traditional image compression standards—such as JPEG, BPG, and Versatile Video Coding (VVC)—rely on mathematical optimization metrics like Mean Squared Error (MSE) or Peak Signal-to-Noise Ratio (PSNR). These metrics treat every pixel's numerical variance with equal weight. When operating at comfortable bandwidths, this approach works well. However, when pushed to extreme compression levels—typically below 0.15 bits per pixel (bpp), and down to less than 0.01 bpp—the data budget becomes too small to represent every pixel accurately.

As noted by Wei-Cheng Lee et al. in A Hybrid Image Codec with Learned Residual Coding and Weimin Tan et al. in Multi-Modality Deep Network for Extreme Learned Image Compression, minimizing pixel-wise error under severe bandwidth constraints forces models to smooth out high-frequency details. This mathematical compromise introduces severe, structured visual degradation: blocky edges, ringing around sharp lines, and blurry textures.

In a standard block-based codec or a basic Variational Autoencoder (VAE), discarding high-frequency coefficients minimizes MSE globally but introduces grid-like discontinuities at block boundaries that human vision is highly sensitive to. To overcome this, modern research shifts the paradigm from exact signal reconstruction to perceptual synthesis and structural preservation. Instead of trying to save every pixel, the goal becomes saving the meaning and structure of the image, allowing the decoder to realistically fill in the blanks.

Decomposing Semantics and Scale

To maximize the utility of a tiny bit budget, a core strategy is to separate the image into structural or semantic components and fine-grained details, compressing each adaptively.

The authors of Extreme Image Compression with Dual-generative Latent Fusion introduce a dual-branch framework that explicitly decomposes latent representations into semantic and detail components. By handling these streams separately, the network avoids wasting bits on stochastic textures (like grass or brick patterns) that can be synthesized later, focusing its budget instead on maintaining structural boundaries. Similarly, Mohammad Akbari in Learned Bi-Resolution Image Coding using Generalized Octave Convolutions leverages multi-scale units to factorize latents into high- and low-resolution components, ensuring structural integrity across varying bitrates.

Another facet of this idea integrates multi-scale resolution learning directly into the codec loop. Wei Gao et al. in Low-rate Image Compression with Super-resolution Learning pair a VAE with "Compact-Resolution" and "Super-Resolution" networks. Instead of compressing a full-resolution image into a blurry mess, the encoder downsamples the image to a compact representation, compresses it cleanly, and relies on a learned upsampling network to restore fine details at the decoder side.

[Input Image] ---> [Downsample / Decompose] ---> [Compact Latent (Low Bits)]
                                                           |
                                                           v
[Restored Image] <--- [Generative Upsample / Fusion] <--- [Decoder]

Generative Hallucination and Semantic Priors

When the bitrate drops to ultra-low levels (e.g., below 0.05 bpp), there simply is not enough data to transmit structural details explicitly. The solution is generative hallucination: using powerful generative models to synthesize realistic textures based on highly compact anchors or external priors.

Naifu Xue et al. in Unifying Generation and Compression: Ultra-low bitrate Image Coding Via Multi-stage Transformer combine vector-quantized tokenization with a multi-stage transformer. Instead of reconstructing pixels from a continuous latent space, the decoder acts as a generative model that "hallucinates" missing details based on learned spatial priors. Taking this further, Jinchang Xu et al. in Decouple Distortion from Perception: Region Adaptive Diffusion for Extreme-low Bitrate Perception Image Compression propose a diffusion-based codec that adaptively masks latent representations based on Regions of Interest (RoI). By using a bidirectional transformer to predict missing details, they successfully decouple local distortion from global perceptual quality, achieving realistic reconstructions at bitrates as low as 0.01 bpp.

Priors can also come from other modalities entirely. Weimin Tan et al. in Multi-Modality Deep Network for Extreme Learned Image Compression introduce text-guided image compression, using brief text descriptions as high-level semantic priors to guide the reconstruction process, maintaining semantic consistency with a fraction of the data budget.

To ensure these hallucinations look authentic rather than artificial, researchers incorporate adversarial losses and discrete quantization. Jiayu Yang et al. in Learned Low Bit-rate Image Compression with Adversarial Mechanism employ Generative Adversarial Networks (GANs) paired with attention mechanisms to synthesize visually convincing textures, while Alaaeldin El-Nouby et al. in Image Compression with Product Quantized Masked Image Modeling combine Product Quantization with Masked Image Modeling to squeeze recognizable images into a mere 200 bytes.

Decoder-Side Artifact Mitigation

An alternative or complementary approach focuses entirely on the decoder side, cleaning up artifacts introduced by standard lossy codecs without altering the underlying bitstream.

Why can a neural network fix these artifacts without extra data? Because compressed streams still contain implicit structural correlations that standard decoders ignore. Compression artifacts are highly structured (e.g., 8x8 grids in JPEG, predictable ringing patterns in VVC). Neural networks can learn the inverse mapping of these artifacts by exploiting remaining redundancies like cross-channel correlation or joint spatial-frequency distributions.

For instance, Kai Cui in Decoder Side Image Quality Enhancement exploiting Inter-channel Correlation in a 3-stage CNN: Submission to CLIC 2018 exploits inter-channel correlation, using the fact that the Green channel typically retains the most detail during compression to structurally guide the restoration of the degraded Red and Blue channels.

Other methods look across different mathematical domains. Honggang Chen et al. in DPW-SDNet: Dual Pixel-Wavelet Domain Deep CNNs for Soft Decoding of JPEG-Compressed Images process compressed images simultaneously in the spatial pixel domain and the frequency-based wavelet domain, stripping away grid-like block boundaries while restoring high-frequency details. To make these restoration networks more efficient and interpretable, Feng Wu 吴枫 et al. in JPEG Artifacts Reduction via Deep Convolutional Sparse Coding unroll a classical sparse coding optimization algorithm into a deep neural network with dilated convolutions, achieving state-of-the-art JPEG artifact reduction with a highly compact parameter footprint.

Key Foundational Work

Incorporating Primal Sketch Based Learning Into Low Bit-Rate Image Compression (2007) — contributed 1 concept elements to this direction.
Sparse representation based down-sampling image compression (2011) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_low_bitrate_compression_artifacts_4cf49cae,
  title = {low-bitrate compression artifacts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/low-bitrate-compression-artifacts/rdr_4cf49caebc7445389cae57645e161060},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_low_bitrate_compression_artifacts_4cf49cae_v1,
  title = {low-bitrate compression artifacts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/low-bitrate-compression-artifacts/rdr_4cf49caebc7445389cae57645e161060/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake