ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis

Patrick Esser, Robin Rombach, Andreas Blattmann, Björn Ommer

TL;DR

ImageBART addresses the "tunnel vision" of standard autoregressive (AR) image models by combining them with a multinomial diffusion process. By generating images through a coarse-to-fine hierarchy, the model gains bidirectional context and global structural awareness, enabling high-fidelity synthesis and flexible local editing (like inpainting) without needing specialized training for those tasks.

The Problem: "Tunnel Vision" in Autoregressive Models

Autoregressive models, which predict the next "token" in a sequence based on previous ones, have seen massive success in language (e.g., GPT) and images (e.g., PixelCNN, Taming Transformers). However, applying them to images introduces a few unnatural biases:

  1. Unidirectional Context: Standard AR models process images like a book—top-left to bottom-right. This means when the model is drawing the top of a face, it has no idea what the chin looks like. This missing bidirectional context makes global consistency difficult.
  2. Scale Insensitivity: They often operate on a single scale, ignoring the "gist" or global structure of a scene until the very end of the sequence.
  3. Exposure Bias: Small errors during the sequential sampling process tend to accumulate, leading to artifacts or "drifting" in long sequences.

The authors aim to give AR models "global eyes" by introducing a hierarchical approach that allows the model to look at the whole image (at a coarse level) while generating the fine details.

Methods: Bridging AR and Diffusion

ImageBART uses a three-step recipe to bridge the gap between autoregressive modeling and diffusion:

ImageBART Architecture
  1. Latent Compression: Instead of working with raw pixels, the model uses a discrete autoencoder (specifically a VQGAN) to compress images into a grid of codebook indices. This reduces the sequence length, making it computationally feasible for Transformers.
  2. Multinomial Diffusion: The authors define a "forward" process that gradually destroys information in the latent grid by randomly replacing tokens with other random tokens from the codebook. This creates a hierarchy of increasingly "noisy" or coarse representations.
  3. The ImageBART Architecture: To invert this process (the "reverse" chain), the authors use an encoder-decoder Transformer at each stage of the hierarchy.
  4. The Encoder: A bidirectional Transformer that looks at the entire image from the previous (coarser) stage.
  5. The Decoder: An autoregressive Transformer that generates the current (finer) stage, token-by-token, while cross-attending to the global context provided by the encoder.

By repeating this across several scales, the model progressively "refines" a random noise grid into a high-fidelity image.

Key Insights

  • Bidirectional Context via Hierarchy: Even though the decoder is still technically autoregressive (one token at a time), the fact that it cross-attends to a global "coarse" version of the image effectively gives it bidirectional context. It "knows" what's to the right and below because it saw the blurry version in the previous stage.
  • Efficiency in Latent Space: Unlike standard diffusion models (like DDPM) that operate in pixel space and require thousands of steps, ImageBART operates in a compressed latent space and requires very few steps (typically $T=2$ to $T=7$) to achieve high quality.
  • Zero-Shot Local Editing: Because the model is trained to "refill" or refine tokens based on surrounding context, it can perform local image editing out-of-the-box. A user can provide a mask, and the model will treat the unmasked area as "coarse context" to generate a consistent completion in the masked area.

Key Results

The model was evaluated on several standard benchmarks for image synthesis, including FFHQ, LSUN (Churches, Bedrooms, Cats), ImageNet, and Conceptual Captions (CC).

Quantitative and Qualitative Results
  • State-of-the-Art Quality: ImageBART outperforms standard AR models (Taming Transformers) and pixel-based diffusion models (DDPM) on complex datasets. For example, on LSUN-Churches, it achieved an FID (Fréchet Inception Distance) of 7.32, beating both TT (7.81) and DDPM (7.89).
  • Speed: It is significantly faster than pixel-based diffusion models because it works on a compressed representation. While slightly slower than a single-pass AR model, the trade-off in quality and control is substantial.
  • Text-to-Image Control: The model shows a strong ability to handle "image-cloze" tasks—altering specific semantic parts of an image (e.g., changing the weather or the texture of an object) based on text prompts while keeping the rest of the scene intact.

Limitations and Future Work

  • Sampling Speed: Although faster than pixel-diffusion, the autoregressive nature of the decoder at each stage still means sampling is slower than purely parallel models like GANs.
  • Architectural Complexity: The hierarchical nature requires training and managing multiple Transformer models, which increases the complexity of the training pipeline.
  • Codebook Bottleneck: The final image quality is ultimately limited by the reconstruction ability of the initial VQGAN. If the autoencoder can't represent a certain detail, the ImageBART stages cannot recover it.

The authors suggest that future work could involve optimizing the number of layers in the encoder vs. decoder to further improve inference speed without sacrificing the global context that makes the model so effective.

Summary updated

Abstract

Autoregressive models and their sequential factorization of the data likelihood have recently demonstrated great potential for image representation and synthesis. Nevertheless, they incorporate image context in a linear 1D order by attending only to previously synthesized image patches above or to the left. Not only is this unidirectional, sequential bias of attention unnatural for images as it disregards large parts of a scene until synthesis is almost complete. It also processes the entire image on a single scale, thus ignoring more global contextual information up to the gist of the entire scene. As a remedy we incorporate a coarse-to-fine hierarchy of context by combining the autoregressive formulation with a multinomial diffusion process: Whereas a multistage diffusion process successively compresses and removes information to coarsen an image, we train a Markov chain to invert this process. In each stage, the resulting autoregressive ImageBART model progressively incorporates context from previous stages in a coarse-to-fine manner. Experiments demonstrate the gain over current autoregressive models, continuous diffusion probabilistic models, and latent variable models. Moreover, the approach enables to control the synthesis process and to trade compression rate against reconstruction accuracy, while still guaranteeing visually plausible results.
BibTeX
@inproceedings{esser2021imagebart,
  author = {Patrick Esser and Robin Rombach and Andreas Blattmann and Björn Ommer},
  title = {{ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=rqg91dhdUm},
}
Figure