Overview
The Image Local Autoregressive Transformer (iLAT) is a novel framework designed for locally guided image synthesis, such as face editing or pose-guided person generation. By combining continuous global features with discrete local representations and a specialized "two-stream" convolution, iLAT achieves superior semantic consistency and significantly faster inference speeds than traditional autoregressive models.
What Problem Does This Paper Address?
Autoregressive (AR) models, which predict pixels or tokens sequentially, have recently challenged Generative Adversarial Networks (GANs) in image generation quality. However, they struggle with locally guided synthesis—tasks where the goal is to change only a specific part of an image (e.g., altering a person's pose or editing a face based on a sketch) while leaving the rest intact.
The authors identify three critical failures in existing AR models for these tasks:
- Missing Global Information: Standard AR models generate from top-left to bottom-right. This means the initial pixels have no "awareness" of the global context or the guidance provided for the bottom of the image.
- Slow Inference: Generating an entire high-resolution image token-by-token is computationally expensive and slow.
- Information Leakage: During training, standard convolutions can "leak" information from the target masked area into the context. This allows the model to "cheat" during training, leading to poor generalization during actual testing.
Methods
The iLAT framework introduces several architectural innovations to bridge the gap between Autoencoders (AE) and Autoregressive (AR) models:
- TS-VQGAN (Two-Stream VQGAN): To prevent information leakage, the authors developed a two-stream convolution mechanism. It explicitly separates features from masked and unmasked regions, ensuring the encoder doesn't "see" the pixels it is supposed to be predicting.
- Local Discrete Representation: Unlike standard VQGANs that tokenize the whole image, iLAT uses continuous features for the unmasked background (preserving detail) and discrete codebook tokens only for the regions to be generated.
- Local Autoregressive (LA) Transformer: The heart of the model is a transformer with a custom attention mask.
This mask allows "global" tokens (unmasked areas) to be seen by everything, while "causal" tokens (the area being edited) follow a sequential dependency. This ensures the generated part is perfectly anchored in the existing global context.
Datasets and Tasks
The model was evaluated on several challenging benchmarks for local synthesis:
- Penn Action (PA): 2,326 video sequences of humans performing actions, used for pose-guided generation in "non-iconic" (cluttered) scenes.
- Synthetic DeepFashion (SDF): A mix of DeepFashion foregrounds and Places2 backgrounds to test complex composition.
- FFHQ & CelebA-HQ: High-resolution face datasets used for sketch-to-face editing tasks.
Key Insights and Results
- Hybrid Representations are Better: The core insight is that you don't need to treat the whole image as a sequence of discrete tokens. By keeping the background as continuous features and only tokenizing the "edit zone," the model maintains higher fidelity and generates much faster.
- Efficiency through Sparsity: Because the model only autoregressively generates the masked tokens, the computational cost scales with the size of the edit, not the size of the whole image.
- Dramatic Speedup: In face editing tasks (FFHQ), iLAT reduced inference time from 8.18 seconds (vanilla AR) to just 1.18 seconds per image—a nearly 7x speedup.
- Quantitative Superiority: On the Penn Action dataset, iLAT achieved an FID (Fréchet Inception Distance) of 27.36, significantly outperforming the "Taming Transformer" baseline (33.53) and GAN-based methods like PATN (82.79).
- Qualitative Consistency: Visual results show that iLAT preserves the identity and background details of the original image much better than previous methods, which often suffered from "identity drift" or blurry artifacts in edited regions.
Limitations and Future Work
The authors note a few areas for improvement:
- Sensitivity to Guidance: In face editing, the model's performance is somewhat dependent on the quality of the input sketch; unprofessional or messy sketches can lead to unnatural results.
- Task-Specific Tuning: The GAN learning weight required manual fixing for face editing to maintain stability, suggesting the adversarial training component could be made more robust.
- Video Potential: While the paper shows a brief example of sequential video generation, fully extending iLAT to temporally consistent video editing remains a promising future direction.
Summary updated