Discrete Predictor-Corrector Diffusion Models for Image Synthesis

Jose Lezama, Tim Salimans, Lu Jiang, Huiwen Chang, Jonathan Ho, Irfan EssaICLR 2023 · OpenReview

Abstract / TL;DR Researchers have introduced Discrete Predictor-Corrector (DPC) diffusion models, bringing the powerful "refinement" capabilities of continuous diffusion to the world of discrete image tokens. By using a learned Markov Chain Monte Carlo (MCMC) corrector to fix errors during parallel sampling, DPC achieves state-of-the-art image synthesis quality on ImageNet while remaining significantly faster than traditional continuous diffusion models.

The Challenge of Discrete Generative Models

In the current generative AI landscape, two heavyweights dominate: continuous Diffusion Models (like Stable Diffusion) and Generative Transformers (like MaskGIT). Continuous diffusion models produce incredible quality but are notoriously slow, often requiring hundreds of steps to "denoise" an image. Discrete models, which work with quantized image "tokens" (similar to words in a sentence), can be much faster because they predict many tokens in parallel.

However, this speed comes at a cost: compounding decoding error. When a model predicts all missing tokens at once, it ignores the complex dependencies between them. Small mistakes in early steps accumulate, leading to a "train-test mismatch" where the intermediate images seen during sampling look nothing like the clean data the model was trained on. While continuous models use a technique called "Langevin correction" to nudge samples back toward the right distribution, discrete spaces lacked a direct equivalent.

Introducing the Discrete Predictor-Corrector (DPC)

To bridge this gap, the authors introduce the Discrete Predictor-Corrector (DPC) framework, which splits the sampling process into two distinct phases:

  1. The Predictor: A standard reverse diffusion step that makes a "best guess" at the missing tokens.
  2. The Corrector: A novel, learned discrete MCMC transition.

Because discrete space lacks the gradients found in continuous space, standard Langevin dynamics cannot be applied. Instead, the authors trained a corrector kernel to satisfy "detailed balance." Essentially, the corrector acts as a neural network trained to recognize when a set of predicted tokens is unlikely, resampling the mask to correct errors.

To make this process efficient, the model utilizes Plackett-Luce Modeling to rank and select which tokens to fix using the Gumbel-top-k trick, and Shortcut Time Transitions, a mathematical optimization that combines the predictor and corrector steps into a single operation to reduce Neural Function Evaluations (NFE).

Key Results and Image Synthesis Quality

DPC was rigorously tested on ImageNet (256×256 and 512×512) for class-conditional generation and Places2 (512×512) for unconditional generation. The framework sets a new bar for discrete generative models:

  • Superior Quality: On ImageNet 512×512, DPC-light(5) achieved a Fréchet Inception Distance (FID) of 6.09 and an Inception Score (IS) of 228.1. It outperformed the continuous ADM model (FID 23.24) and even the powerful StyleGAN-XL (FID 8.43) without external guidance.
  • Efficiency: DPC is roughly 4x to 10x faster than continuous diffusion models. While ADM requires ~250 steps, DPC-light achieves better results in just 66 steps.
  • Human Preference: In user studies, human evaluators preferred DPC-generated images over those from StyleGAN-XL and ADM for both realism and diversity more than 67% of the time.
Koala samples generated by DPC
Elephant samples generated by DPC

These figures demonstrate the model's ability to generate highly realistic, diverse, and coherent biological subjects, showcasing the practical impact of the corrector step in fixing visual inconsistencies.

Limitations and Future Directions

While DPC represents a significant leap forward, it is not without limitations:

  • Inference Overhead: Although much faster than continuous diffusion, DPC still requires more steps (66 NFE) than GANs (1 NFE) or basic MaskGIT (18 NFE). There remains a "speed tax" for the higher quality.
  • Hyperparameter Sensitivity: The model is sensitive to the sampling temperature ($\beta$). Finding the optimal balance for different datasets requires careful tuning.

Future work may explore scaling DPC to even larger transformer architectures or applying the predictor-corrector framework to other discrete domains, such as text generation or protein folding sequences.

Summary updated

Abstract

We introduce Discrete Predictor-Corrector diffusion models (DPC), extending predictor-corrector samplers in Gaussian diffusion models to the discrete case. Predictor-corrector samplers are a class of samplers for diffusion models, which improve on ancestral samplers by correcting the sampling distribution of intermediate diffusion states using MCMC methods. In DPC, the Langevin corrector, which does not have a direct counterpart in discrete space, is replaced with a discrete MCMC transition defined by a learned corrector kernel. The corrector kernel is trained to make the correction steps achieve asymptotic convergence, in distribution, to the correct marginal of the intermediate diffusion states. Equipped with DPC, we revisit recent transformer-based non-autoregressive generative models through the lens of discrete diffusion, and find that DPC can alleviate the compounding decoding error due to the parallel sampling of visual tokens. Our experiments show that DPC improves upon existing discrete latent space models for class-conditional image generation on ImageNet, and outperforms continuous diffusion models and GANs, according to standard metrics and user preference studies.
BibTeX
@inproceedings{lezama2023discrete,
  author = {Jose Lezama and Tim Salimans and Lu Jiang and Huiwen Chang and Jonathan Ho and Irfan Essa},
  title = {{Discrete Predictor-Corrector Diffusion Models for Image Synthesis}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=VM8batVBWvg},
}
Figure