Abstract We present Unified Latents (UL), a framework for learning latent representations that are jointly regularized by a diffusion prior and decoded by a diffusion model. By linking the encoder's output noise to the prior's minimum noise level, we obtain a simple training objective that provides a tight upper bound on the latent bitrate. On ImageNet-512, our approach achieves competitive FID of 1.4, with high reconstruction quality (PSNR) while requiring fewer training FLOPs than models trained on Stable Diffusion latents. On Kinetics-600, we set a new state-of-the-art FVD of 1.3.
Introduction
Latent Diffusion Models (LDMs), such as Stable Diffusion, achieve high efficiency by operating in a compressed latent space rather than raw pixel space. However, the optimal method for compressing images into these latents remains an open challenge. Standard approaches typically employ a Variational Autoencoder (VAE) with a manually tuned KL-divergence penalty to regularize the latent distribution toward a Gaussian. This manual tuning makes it difficult to quantify the actual information content, or bitrate, of the latents.
This leads to a fundamental trade-off:
- High-information latents provide excellent reconstructions (high PSNR) but are complex to model, hindering the downstream diffusion model's ability to generate high-quality novel samples.
- Low-information latents are easier for the diffusion model to learn (yielding better generation FID), but they often lose high-frequency details, resulting in blurry or artifact-prone images.
The Unified Latents (UL) framework addresses this by asking: How should latents be regularized if they are intended to be modeled by a diffusion process?
The Unified Latents Framework
UL replaces the traditional VAE setup with a fully diffusion-based stack consisting of three primary components:
- Deterministic Encoder with Fixed Noise: Unlike traditional VAE encoders that predict both a mean and a learned variance—often leading to training instability—the UL encoder is deterministic. It predicts a single latent vector to which a fixed amount of Gaussian noise is added.
- Diffusion Prior: A diffusion model is trained to model the trajectory from pure noise to the slightly noisy latent produced by the encoder. By explicitly linking the encoder's noise level to the prior's minimum noise level, the training objective simplifies into a tight upper bound on the latent bitrate.
- Diffusion Decoder: Instead of a GAN-based decoder, UL utilizes a diffusion decoder in pixel space, conditioned on the noisy latents. A re-weighted loss function allows the decoder to focus on high-frequency details, while the latent space captures low-frequency semantics.
Training proceeds in two stages: first, the encoder, prior, and decoder are trained jointly. Subsequently, the encoder and decoder are frozen, and a larger "base" diffusion model is trained on the learned latents to generate final samples.
Key Insights and the Information Bottleneck
A central finding of this work is that fixing the noise variance in the encoder, rather than learning it, prevents posterior collapse and stabilizes training. Furthermore, the framework makes the trade-off between reconstruction and generation quality explicit.
By adjusting a loss factor in the decoder, researchers can precisely control the latent bitrate. Smaller base models tend to perform better with lower-bitrate latents, sacrificing some reconstruction fidelity for significantly improved generation quality. Conversely, larger models can effectively leverage higher bitrates. Interestingly, the authors found that the exact number of latent channels (beyond a minimum of 8) has a negligible impact on performance, as the diffusion prior effectively manages the information bottleneck regardless of spatial dimensions.
Experimental Results
The UL framework demonstrates superior performance across multiple benchmarks:
- ImageNet-512: Achieved a generation FID (gFID) of 1.4, outperforming models trained on standard Stable Diffusion latents with fewer training FLOPs.
- Kinetics-600: Set a new state-of-the-art Fréchet Video Distance (FVD) of 1.3, showing the framework's scalability to video data.
- Text-to-Image: In large-scale tests, UL reached a gFID of 4.1, significantly better than pixel-space diffusion (5.0) and standard latent-based models (6.8), while maintaining competitive CLIP scores for text alignment.
Limitations and Future Outlook
While powerful, the use of a diffusion decoder introduces a trade-off in sampling speed, as it is significantly slower than the GAN-based decoders used in current LDMs. Practical real-time applications would likely require distillation of the decoder. Additionally, comparing latent spaces remains challenging due to differences in training data scales between UL and established models like Stable Diffusion. Future work includes establishing scaling laws to predict optimal bitrates for given compute budgets and extending the framework to discrete data types like text.
Summary updated