TL;DR
ImageBART addresses the "tunnel vision" of standard autoregressive (AR) image models by combining them with a multinomial diffusion process. By generating images through a coarse-to-fine hierarchy, the model gains bidirectional context and global structural awareness, enabling high-fidelity synthesis and flexible local editing (like inpainting) without needing specialized training for those tasks.
The Problem: "Tunnel Vision" in Autoregressive Models
Autoregressive models, which predict the next "token" in a sequence based on previous ones, have seen massive success in language (e.g., GPT) and images (e.g., PixelCNN, Taming Transformers). However, applying them to images introduces a few unnatural biases:
- Unidirectional Context: Standard AR models process images like a book—top-left to bottom-right. This means when the model is drawing the top of a face, it has no idea what the chin looks like. This missing bidirectional context makes global consistency difficult.
- Scale Insensitivity: They often operate on a single scale, ignoring the "gist" or global structure of a scene until the very end of the sequence.
- Exposure Bias: Small errors during the sequential sampling process tend to accumulate, leading to artifacts or "drifting" in long sequences.
The authors aim to give AR models "global eyes" by introducing a hierarchical approach that allows the model to look at the whole image (at a coarse level) while generating the fine details.
Methods: Bridging AR and Diffusion
ImageBART uses a three-step recipe to bridge the gap between autoregressive modeling and diffusion:
- Latent Compression: Instead of working with raw pixels, the model uses a discrete autoencoder (specifically a VQGAN) to compress images into a grid of codebook indices. This reduces the sequence length, making it computationally feasible for Transformers.
- Multinomial Diffusion: The authors define a "forward" process that gradually destroys information in the latent grid by randomly replacing tokens with other random tokens from the codebook. This creates a hierarchy of increasingly "noisy" or coarse representations.
- The ImageBART Architecture: To invert this process (the "reverse" chain), the authors use an encoder-decoder Transformer at each stage of the hierarchy.
- The Encoder: A bidirectional Transformer that looks at the entire image from the previous (coarser) stage.
- The Decoder: An autoregressive Transformer that generates the current (finer) stage, token-by-token, while cross-attending to the global context provided by the encoder.
By repeating this across several scales, the model progressively "refines" a random noise grid into a high-fidelity image.
Key Insights
- Bidirectional Context via Hierarchy: Even though the decoder is still technically autoregressive (one token at a time), the fact that it cross-attends to a global "coarse" version of the image effectively gives it bidirectional context. It "knows" what's to the right and below because it saw the blurry version in the previous stage.
- Efficiency in Latent Space: Unlike standard diffusion models (like DDPM) that operate in pixel space and require thousands of steps, ImageBART operates in a compressed latent space and requires very few steps (typically $T=2$ to $T=7$) to achieve high quality.
- Zero-Shot Local Editing: Because the model is trained to "refill" or refine tokens based on surrounding context, it can perform local image editing out-of-the-box. A user can provide a mask, and the model will treat the unmasked area as "coarse context" to generate a consistent completion in the masked area.
Key Results
The model was evaluated on several standard benchmarks for image synthesis, including FFHQ, LSUN (Churches, Bedrooms, Cats), ImageNet, and Conceptual Captions (CC).
- State-of-the-Art Quality: ImageBART outperforms standard AR models (Taming Transformers) and pixel-based diffusion models (DDPM) on complex datasets. For example, on LSUN-Churches, it achieved an FID (Fréchet Inception Distance) of 7.32, beating both TT (7.81) and DDPM (7.89).
- Speed: It is significantly faster than pixel-based diffusion models because it works on a compressed representation. While slightly slower than a single-pass AR model, the trade-off in quality and control is substantial.
- Text-to-Image Control: The model shows a strong ability to handle "image-cloze" tasks—altering specific semantic parts of an image (e.g., changing the weather or the texture of an object) based on text prompts while keeping the rest of the scene intact.
Limitations and Future Work
- Sampling Speed: Although faster than pixel-diffusion, the autoregressive nature of the decoder at each stage still means sampling is slower than purely parallel models like GANs.
- Architectural Complexity: The hierarchical nature requires training and managing multiple Transformer models, which increases the complexity of the training pipeline.
- Codebook Bottleneck: The final image quality is ultimately limited by the reconstruction ability of the initial VQGAN. If the autoencoder can't represent a certain detail, the ImageBART stages cannot recover it.
The authors suggest that future work could involve optimizing the number of layers in the encoder vs. decoder to further improve inference speed without sacrificing the global context that makes the model so effective.
Summary updated