Bit Prioritization in Variational Autoencoders via Progressive Coding

Rui Shu, Stefano ErmonICML 2022

Introduction

Why do diffusion models produce stunning images while standard Hierarchical Variational Autoencoders (HVAEs) often turn out blurry messes? In the world of generative modeling, we face a persistent trade-off: HVAEs are fantastic at representation learning and understanding the underlying structure of data, but they historically struggle with image synthesis, often producing fuzzy or low-fidelity samples. On the other hand, Diffusion Models (like DDPM) have recently dominated the field with incredible sample quality.

This paper argues that the secret to bridging this gap lies in Progressive Coding—the ability to prioritize the most important "bits" of information first. By forcing HVAEs to become better progressive compressors, the authors significantly boost their image synthesis quality, bringing them closer to state-of-the-art diffusion models.

The Progressive Coding Hypothesis

The authors investigate the idea that a generative model’s synthesis quality is intrinsically tied to how well it performs "progressive lossy compression." In simple terms, if you only have a small "budget" of bits to describe an image, a highly effective model should use those bits to describe the most important structural features first.

Standard HVAEs, it turns out, are "lazy" coders. They do not naturally prioritize information in this hierarchical manner, which leads to poor sample quality when reconstructing from limited latent information. Conversely, the authors formally demonstrate that the objective function used by diffusion models is actually a special case of a progressive coding objective. This provides a strong theoretical link explaining why diffusion models excel at synthesis.

A Two-Stage Training Solution

To make a standard HVAE (specifically the VDVAE architecture) a better progressive coder, the researchers propose a novel two-stage training recipe:

  1. Stage 1 (Bit Prioritization): They isolate the first $K$ layers of the latent hierarchy. Instead of standard training, they use constrained optimization (via Lagrangian multipliers) to force these layers to minimize reconstruction error (distortion) for a fixed bit-rate. This ensures the "top" of the hierarchy carries the most critical structural information.
  2. Stage 2 (Detail Filling): They freeze the first stage and train the remaining latent layers to "fill in the details" using a conditional VAE approach.

Key Insights: Compression and Perception

Rate-distortion curves showing Mean-Squared Error versus Rate across SVHN, CelebA, and LSUN datasets, alongside progressive image reconstructions.
  • HVAEs are Suboptimal Compressors: The authors empirically prove that in conventionally trained HVAEs, the first few layers of latent variables are not "Pareto-optimal." Models trained specifically to prioritize these layers achieve much better reconstructions with the exact same number of bits.
  • The MSE-Perception Link: One of the most surprising findings is the relationship between Mean Squared Error (MSE) and perceptual quality. While MSE is often criticized as a poor metric for images (e.g., a slightly shifted image has high MSE but looks identical to the human eye), the authors find that for rate-optimal models, MSE actually correlates very strongly with perceptual metrics like LPIPS.
  • Bit Ordering Matters: The success of diffusion models isn't just about iterative denoising; it's about the fact that their training objective implicitly forces the model to learn a "bit ordering" that is highly efficient for progressive reconstruction.

Key Results

The researchers evaluated their method on three standard image synthesis benchmarks: SVHN (32x32 house numbers), CelebA (64x64 celebrity faces), and LSUN-Bedrooms (64x64 interiors). The "Progressive Coding" training significantly outperformed conventional HVAE training across the board.

Qualitative comparison of image synthesis on SVHN, CelebA, and LSUN-Bedrooms. The proposed progressively-coded HVAE yields sharper global structures compared to baselines.
  • Quantitative Improvements: On SVHN, the FID (Fréchet Inception Distance) score improved dramatically from 18.92 to 10.07. On CelebA, it dropped from 13.33 to 8.54. Lower FID indicates samples that are much closer to the real data distribution.
  • Qualitative Enhancements: Visual inspections of the samples show that the progressively-coded HVAE produces images with much sharper global structure and fewer "mushy" artifacts compared to the baseline.
  • Efficiency: The first $K$ latent variables in the proposed model achieved roughly half the MSE of the baseline model for the same bit-rate, proving the efficacy of the bit prioritization.

Limitations and Future Work

While promising, the approach has a few limitations:

  • Two-Stage Complexity: The current method requires a two-stage training process and manual selection of the "split point" ($K$) in the hierarchy. Future work could explore making this process end-to-end or applying it across more granular groups of latents.
  • Hyperparameter Sensitivity: The choice of the distortion target ($\lambda$) introduces a new hyperparameter that requires careful tuning, akin to designing noise schedules in diffusion models.
  • Theoretical Gap: Although a strong correlation between MSE and LPIPS for rate-optimal models was observed empirically, a comprehensive theoretical explanation of why this happens remains an open question for future research.

Summary updated

BibTeX
@inproceedings{shu2022bit,
  author = {Rui Shu and Stefano Ermon},
  title = {{Bit Prioritization in Variational Autoencoders via Progressive Coding}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
}
Figure