Unified Latents (UL): How to train your latents

Jonathan Heek, Emiel Hoogeboom, Thomas Mensink, Tim Salimans

TL;DR

Unified Latents (UL) is a novel framework that co-trains a diffusion prior and a diffusion decoder to learn continuous latent representations for generative modeling. By explicitly linking the encoder's output noise to the prior's minimum noise level, UL provides a tight, interpretable bound on the latent bitrate and eliminates the need for manual KL-weight tuning. The approach achieves state-of-the-art generation efficiency and quality on both image and video benchmarks.

What Problem Does This Paper Address?

Latent Diffusion Models (LDMs) have revolutionized generative AI by performing diffusion in a compressed latent space rather than pixel space. However, designing and regularizing this latent space remains a major challenge.

Standard approaches, like the Variational Autoencoder (VAE) used in Stable Diffusion, apply a KL-divergence penalty against a standard Gaussian prior. Because the decoder lacks a likelihood-based loss, the weight of this penalty must be tuned manually, making it difficult to optimize the information content of the latents. Conversely, recent methods using heavily regularized or discrete semantic latents are easier for downstream models to learn, but they often discard high-frequency details, leading to poor reconstruction quality (low PSNR).

The authors address a fundamental question: How should latents be regularized when they will subsequently be modeled by a diffusion model? They aim to systematically navigate the trade-off between a latent's information density (how easy it is to model) and its reconstruction fidelity.

Methods

The core idea of Unified Latents (UL) is to jointly train the encoder, a diffusion prior, and a diffusion decoder.

Schematic overview of the Unified Latents framework, showing the joint training of the Encoder, Diffusion Prior, and Diffusion Decoder.

The framework relies on three key algorithmic innovations:

  1. Deterministic Encoding with Fixed Noise: Instead of learning a complex, unstable encoder distribution, a deterministic encoder predicts a single clean latent ($z_{\text{clean}}$). This latent is then forward-noised to time $t=0$ using a fixed diagonal Gaussian noise ($\sigma_0 \approx 0.08$, corresponding to a log-SNR of 5).
  2. Aligned Prior Precision: The prior diffusion model is explicitly aligned with this minimum noise level. This mathematical alignment simplifies the VAE's complex KL-divergence term into a straightforward, weighted mean squared error (MSE) loss over noise levels.
  3. Sigmoid-Weighted Decoder Loss: The diffusion decoder operates in image space and uses a reweighted Evidence Lower Bound (ELBO) with a sigmoid weighting function.

Training is executed in two stages. In Stage 1, the encoder, prior, and decoder are trained jointly using an unweighted ELBO for the prior to maintain a strict information bound. In Stage 2, the encoder and decoder are frozen, and the prior is retrained as a "base model" using a quality-friendly sigmoid weighting to maximize generation performance.

Datasets

The authors evaluated their framework on several standard and large-scale generative benchmarks:

  • ImageNet-512: Used for class-conditional image generation at $512 \times 512$ resolution.
  • Kinetics-600: Used to evaluate video generation performance ($16$ frames at $128 \times 128$ resolution).
  • Internal Text-to-Image/Video Datasets: Large-scale datasets used to evaluate the framework's scalability and text-alignment capabilities.

Key Insights and Ideas

  • Explicit Control via Loss Factoring: By introducing a decoder loss factor ($c_{\text{lf}}$) and adjusting the sigmoid bias ($b$), the framework provides explicit, interpretable control over the latent bitrate (bits per dimension). This allows practitioners to easily choose whether the latent or the decoder should handle high-frequency details.
  • Insensitivity to Channel Count: Traditional LDMs are highly sensitive to the number of latent channels, which act as a strict information bottleneck. In contrast, UL is remarkably robust; varying the latent channels from 16 to 64 yields nearly identical reconstruction and generation metrics because the diffusion prior dynamically manages the information flow.
  • The Power of Diffusion Decoders: Moving from a traditional GAN-based decoder to a diffusion decoder allows the system to model a full distribution over images rather than a single deterministic output. This prevents mode collapse and handles highly compressed, low-bitrate latents much more gracefully.
The decoder loss weight curve illustrates how adjusting the loss factor determines whether information is preferred in the latent space or handled by the decoder.

Key Results

  • ImageNet-512 Efficiency: UL achieved a highly competitive generation Fréchet Inception Distance (FID) of 1.4. Crucially, it demonstrated superior pre-training efficiency, requiring significantly fewer training FLOPs to reach low FID scores compared to baselines trained on Stable Diffusion latents.
FID vs. training cost on ImageNet-512, demonstrating that Unified Latents outperform existing baselines in compute efficiency.
  • Kinetics-600 State-of-the-Art: On video generation, the medium UL model established a new state-of-the-art Fréchet Video Distance (FVD) of 1.3.
  • Text-to-Image Quality: In large-scale text-to-image evaluations, UL (with a loss factor of 1.5) outperformed both pixel diffusion and Stable Diffusion latents in both generation quality (gFID of 4.1 vs 5.0 and 6.8) and text alignment (CLIP score of 27.1).

Limitations and Future Work

The primary limitation of Unified Latents is sampling computational cost. Because the decoder is itself a diffusion model, generating an image requires running a multi-step diffusion process through both the base model (prior) and the decoder. This makes sampling an order of magnitude more expensive than using standard GAN-based decoders. The authors note that downstream distillation steps will be necessary to make this practical for real-time deployment.

For future work, the authors propose establishing formal scaling laws for Unified Latents to automatically predict the optimal latent bitrate given a specific training compute budget.

Summary updated

Abstract

We present Unified Latents (UL), a framework for learning latent representations that are jointly regularized by a diffusion prior and decoded by a diffusion model. By linking the encoder's output noise to the prior's minimum noise level, we obtain a simple training objective that provides a tight upper bound on the latent bitrate. On ImageNet-512, our approach achieves competitive FID of 1.4, with high reconstruction quality (PSNR) while requiring fewer training FLOPs than models trained on Stable Diffusion latents. On Kinetics-600, we set a new state-of-the-art FVD of 1.3.
BibTeX
@misc{heek2026unified,
  author = {Jonathan Heek and Emiel Hoogeboom and Thomas Mensink and Tim Salimans},
  title = {{Unified Latents (UL): How to train your latents}},
  year = {2026},
  eprint = {2602.17270},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.17270},
}
Figure