Locally Hierarchical Auto-Regressive Modeling for Image Generation

Tackgeun You, Saehoon Kim, Chiheon Kim, Doyup Lee, Bohyung HanNeurIPS 2022 · OpenReview

TL;DR

This paper introduces a locally hierarchical approach to auto-regressive (AR) image generation that significantly boosts sampling speed without sacrificing image quality. By disentangling coarse structural information from fine local details and exploiting a conditional independence assumption, the authors enable parallel decoding of detail-oriented "bottom codes." This innovation achieves up to a 650x throughput increase over some diffusion models while maintaining competitive fidelity.

The Bottleneck in Auto-Regressive Image Generation

Auto-regressive (AR) models, which predict the next element in a sequence based on previous ones, have proven highly effective for generating diverse, high-quality images (e.g., VQ-VAE, VQ-GAN). However, they suffer from a major bottleneck: computational cost.

Because images are high-dimensional, representing them as a single sequence of discrete codes results in very long sequences. Since standard AR models generate these codes one-by-one, the process is agonizingly slow. While hierarchical models (like VQ-VAE-2) attempt to solve this by using multiple resolutions of codes, they often struggle to properly "disentangle" information between levels or still rely on slow, sequential decoding for every single code.

A Two-Stage Locally Hierarchical Framework

The authors propose a two-stage framework consisting of HQ-VAE and HQ-Transformer.

Architecture of HQ-VAE and HQ-Transformer

1. HQ-VAE (Hierarchically Quantized VAE)

This stage represents an image using a pyramid of two-level discrete codes: Top codes (coarse, global structure) and Bottom codes (fine, local details).

  • The "Aha" Resampling: Instead of using standard average pooling to downsample images for the top codes, they use a pixel-unshuffle operation. This reshapes the tensor to preserve more information in the channel dimension, forcing the top codes to capture the primary "essence" of the image.
  • Residual Learning: The bottom codes are trained to represent the residual—the difference between the original image and the reconstruction from the top codes.

2. HQ-Transformer

This is the generative engine, consisting of three sub-modules:

  • Input Embedding Transformer (IET): Refines local embeddings of a top code and its associated bottom codes.
  • Main Transformer (MT): Predicts the sequence of top codes one-by-one (standard AR).
  • Prediction Head Transformer (PHT): Based on the "global landscape" provided by the top code, the model assumes that the local details (bottom codes) are conditionally independent. This allows the model to sample all bottom codes associated with a single top code in parallel, drastically reducing the number of sequential steps required.

Key Insights and Ideas

  • Disentanglement is Key: By using pixel-unshuffle/shuffle instead of naive pooling, the authors successfully separated "structure" from "texture."
Visual disentanglement of top and bottom codes alongside the PixelUnShuffle mechanism

As visually demonstrated above, top codes alone generate a blurry but recognizable "ghost" of the image, while bottom codes alone look like high-frequency edge maps. When combined, they perfectly reconstruct the original image.

  • Parallelism via Independence: The most significant insight is that once you know the high-level structure (the top code), the exact arrangement of fine details in that local patch doesn't strictly depend on the details of the neighboring patch. This "local hierarchy" allows for a hybrid generation strategy: sequential for the "big picture," parallel for the "fine brushstrokes."
  • Efficiency without Quality Loss: Unlike many methods that trade off quality for speed, HQ-TVAE actually achieves better reconstruction (rFID) than previous single-scale or hierarchical baselines because it utilizes its codebook more effectively.

Key Results

  • Speed: The smallest model (HQ-TVAE-S) achieves a throughput of ~79 images/sec, which is roughly 650x faster than the ADM diffusion model and significantly faster than VQ-GAN (~7 images/sec).
  • Fidelity: On ImageNet class-conditional generation (1.2M images), their largest model (HQ-TVAE-L) achieved an FID of 7.15, outperforming VQ-GAN (15.78) and RQ-Transformer (7.55) while remaining much faster.
  • Reconstruction: The HQ-VAE (32x32) achieved an rFID of 0.57, indicating nearly perfect reconstruction of the original validation images, far surpassing standard VQ-GAN (4.90).

Limitations and Future Work

  • Diversity Constraints: The authors note that when the top code is fixed, the diversity of the generated image is restricted to very fine details. The "big decisions" are all made by the sequential top-level AR model.
  • Sequential Top-Level: While the bottom codes are parallel, the top codes are still generated one-by-one. For extremely high-resolution images, the top-level sequence might still become a bottleneck.
  • Future Directions: The authors suggest exploring more than two levels of hierarchy and applying this "locally hierarchical" logic to other modalities like video or audio, where temporal/spatial redundancy is high.

Summary updated

Abstract

We propose a locally hierarchical auto-regressive model with multiple resolutions of discrete codes. In the first stage of our algorithm, we represent an image with a pyramid of codes using Hierarchically Quantized Variational AutoEncoder (HQ-VAE), which disentangles the information contained in the multi-level codes. For an example of two-level codes, we create two separate pathways to carry high-level coarse structures of input images using top codes while compensating for missing fine details by constructing a residual connection for bottom codes. An appropriate selection of resizing operations for code embedding maps enables top codes to capture maximal information within images and the first stage algorithm achieves better performance on both vector quantization and image generation. The second stage adopts Hierarchically Quantized Transformer (HQ-Transformer) to process a sequence of local pyramids, which consist of a single top code and its corresponding bottom codes. Contrary to other hierarchical models, we sample bottom codes in parallel by exploiting the conditional independence assumption on the bottom codes. This assumption is naturally harvested from our first-stage model, HQ-VAE, where the bottom code learns to describe local details. On class-conditional and text-conditional generation benchmarks, our model shows competitive performance to previous AR models in terms of fidelity of generated images while enjoying lighter computational budgets.
BibTeX
@inproceedings{you2022locally,
  author = {Tackgeun You and Saehoon Kim and Chiheon Kim and Doyup Lee and Bohyung Han},
  title = {{Locally Hierarchical Auto-Regressive Modeling for Image Generation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=B5PVakGLZFV},
}
Figure