ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis

Patrick Esser, Robin Rombach, Andreas Blattmann, Björn Ommer

TL;DR

ImageBART tackles the limitations of traditional autoregressive image generation models by introducing a novel coarse-to-fine hierarchical approach that leverages multinomial diffusion. This allows for bidirectional context, enabling more natural image synthesis and powerful local image editing capabilities, such as free-form inpainting and text-guided local modifications, without requiring specialized training for these tasks.

What Problem Does This Paper Address?

Autoregressive (AR) models have shown great promise in image synthesis by factorizing the data likelihood sequentially. However, they suffer from a significant limitation: they process images in a strict top-left to bottom-right order, attending only to previously generated patches. This unidirectional bias is unnatural for images, as it ignores large parts of the scene until late in the generation process. Furthermore, this sequential processing occurs at a single scale, missing out on crucial global contextual information that could inform the synthesis. This limitation also hinders practical image editing tasks like inpainting, where a model needs to understand context from all directions. Additionally, AR models are prone to "exposure bias" during training, where they rely on ground truth (teacher-forcing) but face previously generated tokens during inference, leading to error accumulation.

Methods

ImageBART proposes a coarse-to-fine hierarchical approach to overcome these limitations. The core idea is to combine autoregressive modeling with a multinomial diffusion process.

  1. Discrete Representation Learning: First, images are compressed into a compact, discrete representation using a VQGAN-like approach. This stage focuses on learning a high-quality, low-dimensional representation, trading off perfect reconstruction for compression.
  2. Multinomial Diffusion Process: A fixed multinomial diffusion process is used to progressively coarsen this representation, effectively creating a hierarchy of increasingly abstract image representations.
  3. Reverse Markov Chain: The model then learns to invert this diffusion process by training a short Markov chain. Each step in this reverse chain is modeled autoregressively using an encoder-decoder transformer. Crucially, each autoregressive step attends to the previous, coarser stage in the hierarchy, providing global context. This is akin to a denoising cloze task where missing information is filled in.

This hierarchical structure allows each AR model to leverage global context from its preceding, coarser stage, addressing the unidirectional bias and single-scale processing issues. The model is trained to invert the diffusion process, which helps mitigate exposure bias.

Datasets

The paper uses several standard datasets for training and evaluation:

  • FFHQ (Flickr-Faces-HQ): Used for high-fidelity unconditional image synthesis.
  • LSUN (Cats, Churches, Bedrooms): Used for unconditional image synthesis.
  • ImageNet (cIN): Used for class-conditional image synthesis.
  • Conceptual Captions (CC): Used for text-conditional image synthesis.

Key Insights and Ideas

  • Bidirectional Context is Key: The paper strongly emphasizes that for image generation and manipulation, attending to context from all directions (not just top-left) is crucial for naturalness and capability.
  • Hierarchy for Global Context: Introducing a coarse-to-fine hierarchy is an effective way to provide global context to local autoregressive steps, leading to more coherent and globally consistent image generation.
  • Diffusion as a Compression Tool: Using a multinomial diffusion process to create a hierarchy of compressed representations is an elegant way to manage complexity and enable efficient training of powerful AR models.
  • Local Editing without Special Training: The ability to perform free-form inpainting and local image editing is a direct consequence of the bidirectional context and hierarchical structure, achieved without needing to train specific models for these tasks.

Key Results

  • High-Fidelity Synthesis: ImageBART achieves competitive FID scores on various datasets (LSUN, FFHQ, ImageNet) compared to state-of-the-art diffusion and AR models. For complex datasets like LSUN-Cats and cIN, it outperforms pixel-based diffusion approaches.
  • Improved Image Modification: The model demonstrates significantly improved capabilities in image modification tasks, such as upper-half completion, outperforming previous AR models like TT [18].
  • Local Image Editing: ImageBART successfully performs free-form inpainting and local, text-guided image editing, even with arbitrary masks, without requiring mask-specific training.
  • Efficient Training: Training is efficient due to the compressed latent space and the ability to train hierarchical levels in parallel.
  • Conditional Generation: The model excels at class-conditional and text-conditional image synthesis, producing realistic and contextually relevant outputs. For text-conditional generation on CC, it improves upon previous methods in FID, IS, and CLIP scores.
  • Scalability: The approach can generate high-resolution images (megapixel regime) by applying models in a patch-wise manner.

Limitations and Future Work

  • Compression Artifacts: The strong compression required for efficiency can lead to artifacts in reconstructed images, making it unsuitable for applications demanding extremely high fidelity.
  • Data Requirements: Transformer architectures, while less biased than CNNs, require substantial data to avoid overfitting, making the approach less suitable for very small datasets.
  • Computational Resources: Training on the largest datasets still requires significant computational resources, and performance continues to improve with more training.
  • Future Work: The authors suggest exploring different architectures, regularizations, and augmentations for smaller datasets. They also highlight the potential for further improvements with increased computational resources. The paper also notes that while the model can be made faster by reducing decoder layers, this might come at the cost of controllability.

Summary updated

Abstract

Autoregressive models and their sequential factorization of the data likelihood have recently demonstrated great potential for image representation and synthesis. Nevertheless, they incorporate image context in a linear 1D order by attending only to previously synthesized image patches above or to the left. Not only is this unidirectional, sequential bias of attention unnatural for images as it disregards large parts of a scene until synthesis is almost complete. It also processes the entire image on a single scale, thus ignoring more global contextual information up to the gist of the entire scene. As a remedy we incorporate a coarse-to-fine hierarchy of context by combining the autoregressive formulation with a multinomial diffusion process: Whereas a multistage diffusion process successively removes information to coarsen an image, we train a (short) Markov chain to invert this process. In each stage, the resulting autoregressive ImageBART model progressively incorporates context from previous stages in a coarse-to-fine manner. Experiments show greatly improved image modification capabilities over autoregressive models while also providing high-fidelity image generation, both of which are enabled through efficient training in a compressed latent space. Specifically, our approach can take unrestricted, user-provided masks into account to perform local image editing. Thus, in contrast to pure autoregressive models, it can solve free-form image inpainting and, in the case of conditional models, local, text-guided image modification without requiring mask-specific training.
BibTeX
@inproceedings{esser2021imagebart,
  author = {Patrick Esser and Robin Rombach and Andreas Blattmann and Björn Ommer},
  title = {{ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=rqg91dhdUm},
}
Figure