Lossless Latent Diffusion Models

Tongda Xu, Yu Zhang, Jun Hu, Huijuan HuangarXiv 2026 · arXiv

TL;DR

Lossless Latent Diffusion Models (LLDM) bridge the gap between high-quality "lossy" latent diffusion and "lossless" pixel-space models. By using a simple residual decomposition and a clever "uneven scaling" trick, LLDM achieves pixel-exact reconstruction (infinite PSNR) while matching or exceeding the generation quality of state-of-the-art lossy models—all without modifying the underlying diffusion architecture.

What Problem Does This Paper Address?

In the world of image generation, there has long been a "reconstruction-generation trade-off."

Standard Latent Diffusion Models (LDMs), like Stable Diffusion, are popular because they operate in a compressed "latent space" created by a Variational Autoencoder (VAE). This makes training efficient and generation high-quality, but it is inherently lossy: the VAE discards fine-grained details that can never be recovered, making these models unsuitable for tasks requiring bit-perfect accuracy (like medical imaging or professional photo editing).

Conversely, Pixel-space diffusion models preserve every pixel, but they are notoriously difficult to train. Pixels are high-dimensional and contain a lot of "noise" that isn't semantically meaningful, often leading to lower-quality images or requiring complex, specialized architectures to work.

The authors of this paper ask: Can we have the best of both worlds? Can we create a representation that is mathematically lossless but as easy for a diffusion model to learn as a lossy latent space?

Methods

The core of LLDM is a surprisingly simple, two-part lossless transformation:

  1. Residual Decomposition: The authors take a pre-trained lossy VAE. For any image $x$, they extract the standard latent $z$ and the reconstruction residual $r$ (the difference between the original image and the VAE's best attempt at reconstructing it). By concatenating these ($[z, r]$), they create a representation that contains 100% of the original image information.
  2. Uneven Latent Scaling: This is the "secret sauce." If you train a diffusion model on the raw $[z, r]$ stack, it performs poorly because it gets distracted by the high-frequency details in the residual. The authors introduce a scaling factor $\gamma$ (typically very small, like 0.01) to the residual. This "uneven scaling" forces the diffusion model to prioritize the semantically rich $z$ components during the early stages of denoising, while still allowing it to learn the fine-grained $r$ details needed for perfect reconstruction.

To explain why this works, they introduce a new metric called Semantic Rank Priority (SRP). SRP measures how well semantic information (like object category) is concentrated in the top principal components of the latent space.

Figure 1. Lossless LDM achieves pixel-exact reconstruction (PSNR= ∞) while outperforming many lossy LDMs in generation quality. Left: ImageNet class-to-image generation. Right: Diffusion-Bench text-to-image generation.

Datasets

The authors evaluated LLDM on two primary benchmarks:

  • ImageNet (256×256): Used for class-to-image generation tasks.
  • Diffusion-Bench: A comprehensive text-to-image benchmark that includes sub-metrics like GenEval, DPG-Bench, and GenAI-Bench to evaluate prompt following and image quality.

Key Insights and Ideas

  • Information loss isn't necessary for quality: A common belief was that VAEs must discard information to "clean up" the data for diffusion. LLDM proves that as long as the information is organized correctly (prioritizing semantics over residuals), you can keep everything.
  • The Power of Scaling: Simply down-weighting the "hard" part of the data (the residual) is more effective than complex architectural changes. It acts as a form of curriculum learning, guiding the model to learn the "what" before the "fine details."
  • SRP as a Predictor: The authors found that their SRP metric correlates much more strongly with actual generation quality (gFID) than standard linear classification accuracy. This suggests that "diffusibility" is specifically about how semantic information is ranked within the representation.
Figure 3. Correlation of SRP with generation quality (gFID). SRP exhibits stronger correlations than standard accuracy.

Key Results

  • Pixel-Exact Reconstruction: By definition, LLDM achieves infinite PSNR, whereas even high-quality lossy VAEs like FLUX2-VAE top out around 34 dB.
  • Superior Generation: On ImageNet, LLDM achieved a gFID of 11.64, vastly outperforming vanilla pixel diffusion (gFID 104.29) and even beating several widely used lossy VAEs like SD-VAE and SD3-VAE.
  • Visual Fidelity: As shown in Figure 2, LLDM eliminates the blurring and artifacting common in lossy reconstructions while maintaining the high-level structure of the source image.
Figure 2. Reconstruction comparison. LLDM achieves pixel-exact reconstruction (infinite PSNR) unlike lossy VAEs.

Limitations and Future Work

  • Base VAE Dependency: LLDM currently relies on a pre-trained lossy VAE to define the semantic part of its latent space. The quality of the "lossless" model is still somewhat tethered to the quality of the "lossy" base.
  • Hyperparameter Tuning: The optimal scaling factor $\gamma$ varies depending on the base VAE's reconstruction quality. A VAE with poor reconstruction needs a larger $\gamma$, while a high-fidelity VAE needs a smaller one.
  • Learned Transformations: While the authors experimented with Normalizing Flows for the transformation, they didn't see improvements over their manual scaling method. Future work could involve end-to-end learning of these invertible, diffusion-friendly mappings.

Summary updated

Abstract

We study the problem of training image diffusion models while preserving pixel-exact reconstruction. Prior works typically address this problem by directly modeling images in pixel space, requiring modifications to the diffusion training objective or model architecture. In this work, we introduce a different paradigm: rather than modifying diffusion models, we construct a lossless transformation that maps pixels into a representation more amenable to diffusion modeling. We term our approach the Lossless Latent Diffusion Model (LLDM). We construct LLDM using residual latent transformation and uneven latent scaling, and explain its effectiveness through a proposed metric, semantic rank priority (SRP). Empirically, LLDM substantially outperforms prior pixel-space diffusion models in generation quality and even surpasses several widely used lossy VAEs. Furthermore, we show that SRP is strongly correlated with downstream generative performance.
BibTeX
@misc{xu2026lossless,
  author = {Tongda Xu and Yu Zhang and Jun Hu and Huijuan Huang},
  title = {{Lossless Latent Diffusion Models}},
  year = {2026},
  eprint = {2611.25454},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25454},
}
Figure