TL;DR
This paper introduces a locally hierarchical approach to auto-regressive (AR) image generation that significantly boosts sampling speed without sacrificing image quality. By disentangling coarse structural information from fine local details and exploiting a conditional independence assumption, the authors enable parallel decoding of detail-oriented "bottom codes." This innovation achieves up to a 650x throughput increase over some diffusion models while maintaining competitive fidelity.
The Bottleneck in Auto-Regressive Image Generation
Auto-regressive (AR) models, which predict the next element in a sequence based on previous ones, have proven highly effective for generating diverse, high-quality images (e.g., VQ-VAE, VQ-GAN). However, they suffer from a major bottleneck: computational cost.
Because images are high-dimensional, representing them as a single sequence of discrete codes results in very long sequences. Since standard AR models generate these codes one-by-one, the process is agonizingly slow. While hierarchical models (like VQ-VAE-2) attempt to solve this by using multiple resolutions of codes, they often struggle to properly "disentangle" information between levels or still rely on slow, sequential decoding for every single code.
A Two-Stage Locally Hierarchical Framework
The authors propose a two-stage framework consisting of HQ-VAE and HQ-Transformer.
1. HQ-VAE (Hierarchically Quantized VAE)
This stage represents an image using a pyramid of two-level discrete codes: Top codes (coarse, global structure) and Bottom codes (fine, local details).
- The "Aha" Resampling: Instead of using standard average pooling to downsample images for the top codes, they use a pixel-unshuffle operation. This reshapes the tensor to preserve more information in the channel dimension, forcing the top codes to capture the primary "essence" of the image.
- Residual Learning: The bottom codes are trained to represent the residual—the difference between the original image and the reconstruction from the top codes.
2. HQ-Transformer
This is the generative engine, consisting of three sub-modules:
- Input Embedding Transformer (IET): Refines local embeddings of a top code and its associated bottom codes.
- Main Transformer (MT): Predicts the sequence of top codes one-by-one (standard AR).
- Prediction Head Transformer (PHT): Based on the "global landscape" provided by the top code, the model assumes that the local details (bottom codes) are conditionally independent. This allows the model to sample all bottom codes associated with a single top code in parallel, drastically reducing the number of sequential steps required.
Key Insights and Ideas
- Disentanglement is Key: By using pixel-unshuffle/shuffle instead of naive pooling, the authors successfully separated "structure" from "texture."
As visually demonstrated above, top codes alone generate a blurry but recognizable "ghost" of the image, while bottom codes alone look like high-frequency edge maps. When combined, they perfectly reconstruct the original image.
- Parallelism via Independence: The most significant insight is that once you know the high-level structure (the top code), the exact arrangement of fine details in that local patch doesn't strictly depend on the details of the neighboring patch. This "local hierarchy" allows for a hybrid generation strategy: sequential for the "big picture," parallel for the "fine brushstrokes."
- Efficiency without Quality Loss: Unlike many methods that trade off quality for speed, HQ-TVAE actually achieves better reconstruction (rFID) than previous single-scale or hierarchical baselines because it utilizes its codebook more effectively.
Key Results
- Speed: The smallest model (HQ-TVAE-S) achieves a throughput of ~79 images/sec, which is roughly 650x faster than the ADM diffusion model and significantly faster than VQ-GAN (~7 images/sec).
- Fidelity: On ImageNet class-conditional generation (1.2M images), their largest model (HQ-TVAE-L) achieved an FID of 7.15, outperforming VQ-GAN (15.78) and RQ-Transformer (7.55) while remaining much faster.
- Reconstruction: The HQ-VAE (32x32) achieved an rFID of 0.57, indicating nearly perfect reconstruction of the original validation images, far surpassing standard VQ-GAN (4.90).
Limitations and Future Work
- Diversity Constraints: The authors note that when the top code is fixed, the diversity of the generated image is restricted to very fine details. The "big decisions" are all made by the sequential top-level AR model.
- Sequential Top-Level: While the bottom codes are parallel, the top codes are still generated one-by-one. For extremely high-resolution images, the top-level sequence might still become a bottleneck.
- Future Directions: The authors suggest exploring more than two levels of hierarchy and applying this "locally hierarchical" logic to other modalities like video or audio, where temporal/spatial redundancy is high.
Summary updated