Abstract
Energy-based models (EBMs) have recently been successful in representing complex distributions of small images. However, sampling from them requires expensive Markov chain Monte Carlo (MCMC) iterations that mix slowly in high dimensional pixel space. Unlike EBMs, variational autoencoders (VAEs) generate samples quickly and are equipped with a latent space that enables fast traversal of the data manifold. However, VAEs tend to assign high probability density to regions in data space outside the actual data distribution and often fail at generating sharp images. In this paper, we propose VAEBM, a symbiotic composition of a VAE and an EBM that offers the best of both worlds. VAEBM captures the overall mode structure of the data distribution using a state-of-the-art VAE and it relies on its EBM component to explicitly exclude non-data-like regions from the model and refine the image samples. Moreover, the VAE component in VAEBM allows us to speed up MCMC updates by reparameterizing them in the VAE's latent space. Our experimental results show that VAEBM outperforms state-of-the-art VAEs and EBMs in generative quality on several benchmark image datasets by a large margin. It can generate high-quality images as large as 256x256 pixels with short MCMC chains. We also demonstrate that VAEBM provides complete mode coverage and performs well in out-of-distribution detection.
Introduction
Deep generative modeling typically forces a choice between two paradigms: Variational Autoencoders (VAEs) and Energy-Based Models (EBMs). VAEs are efficient to sample from and provide a structured latent space, but they are notorious for producing blurry images and "over-generalizing" by assigning high probability to regions where no data exists. EBMs, while flexible and capable of producing sharp details, rely on computationally expensive MCMC sampling in high-dimensional pixel space, which often fails to mix effectively.
VAEBM addresses these limitations by creating a "symbiosis" that uses a VAE to capture the broad structure of the data manifold while using an EBM to prune away the VAE's mistakes and sharpen the final output.
Methodology
The VAEBM framework defines the generative distribution as the product of a VAE generator and an EBM component. To avoid complex joint optimization, the model is trained in two stages. First, a state-of-the-art VAE (specifically NVAE) is trained. Second, the VAE is frozen, and a ResNet-based energy function is trained to refine the VAE's output.
A key innovation is Latent Space MCMC. Instead of running MCMC in the massive pixel space, VAEBM reparameterizes the updates into the joint space of the VAE’s latent variables ($z$) and reconstruction noise ($\epsilon_x$). Because the VAE has already learned a smooth manifold, MCMC chains can traverse the distribution much faster, requiring only 6–16 steps to produce high-quality samples.
Key Results
The model was evaluated on several standard benchmarks, including CIFAR-10, CelebA-HQ ($256 \times 256$), and LSUN Church.
Image Quality and Refinement
On CIFAR-10, VAEBM achieved an FID (Fréchet Inception Distance) of 12.19, significantly outperforming both standalone VAEs (NVAE: 51.67) and previous EBMs (IGEBM: 40.58). The EBM component acts as a corrector, explicitly excluding "non-data-like" regions that the VAE might have erroneously included.
Efficiency and Robustness
VAEBM is roughly 12x faster at sampling than popular score-based models like NCSN. It also demonstrates superior mode coverage; on the StackedMNIST benchmark, VAEBM captured all 1,000 modes, whereas many GAN-based models suffer from mode collapse. Furthermore, the EBM component significantly improves Out-of-Distribution (OOD) detection, showing a massive jump in AUROC compared to standalone VAEs (e.g., 0.83 vs 0.42 on SVHN).
Conclusion
VAEBM successfully scales energy-based refinement to high-resolution images, marking a significant step forward for EBMs. By moving MCMC into the latent space, the model effectively "walks" along the learned data manifold, making every step more meaningful. While the authors note that long chains can still suffer from local mode trapping, the hybrid approach provides a powerful framework for high-quality, efficient generative modeling.
Summary updated