The Image Local Autoregressive Transformer

Chenjie Cao, Yuxin Hong, Xiang Li, Chengrong Wang, Chengming Xu, Yanwei Fu, Xiangyang Xue

Overview

The Image Local Autoregressive Transformer (iLAT) is a novel framework designed for locally guided image synthesis, such as face editing or pose-guided person generation. By combining continuous global features with discrete local representations and a specialized "two-stream" convolution, iLAT achieves superior semantic consistency and significantly faster inference speeds than traditional autoregressive models.

What Problem Does This Paper Address?

Autoregressive (AR) models, which predict pixels or tokens sequentially, have recently challenged Generative Adversarial Networks (GANs) in image generation quality. However, they struggle with locally guided synthesis—tasks where the goal is to change only a specific part of an image (e.g., altering a person's pose or editing a face based on a sketch) while leaving the rest intact.

The authors identify three critical failures in existing AR models for these tasks:

  1. Missing Global Information: Standard AR models generate from top-left to bottom-right. This means the initial pixels have no "awareness" of the global context or the guidance provided for the bottom of the image.
  2. Slow Inference: Generating an entire high-resolution image token-by-token is computationally expensive and slow.
  3. Information Leakage: During training, standard convolutions can "leak" information from the target masked area into the context. This allows the model to "cheat" during training, leading to poor generalization during actual testing.

Methods

The iLAT framework introduces several architectural innovations to bridge the gap between Autoencoders (AE) and Autoregressive (AR) models:

  • TS-VQGAN (Two-Stream VQGAN): To prevent information leakage, the authors developed a two-stream convolution mechanism. It explicitly separates features from masked and unmasked regions, ensuring the encoder doesn't "see" the pixels it is supposed to be predicting.
  • Local Discrete Representation: Unlike standard VQGANs that tokenize the whole image, iLAT uses continuous features for the unmasked background (preserving detail) and discrete codebook tokens only for the regions to be generated.
  • Local Autoregressive (LA) Transformer: The heart of the model is a transformer with a custom attention mask.
Local Autoregressive Mask

This mask allows "global" tokens (unmasked areas) to be seen by everything, while "causal" tokens (the area being edited) follow a sequential dependency. This ensures the generated part is perfectly anchored in the existing global context.

Datasets and Tasks

The model was evaluated on several challenging benchmarks for local synthesis:

  • Penn Action (PA): 2,326 video sequences of humans performing actions, used for pose-guided generation in "non-iconic" (cluttered) scenes.
Pose-guided generation on Penn Action
  • Synthetic DeepFashion (SDF): A mix of DeepFashion foregrounds and Places2 backgrounds to test complex composition.
  • FFHQ & CelebA-HQ: High-resolution face datasets used for sketch-to-face editing tasks.

Key Insights and Results

  • Hybrid Representations are Better: The core insight is that you don't need to treat the whole image as a sequence of discrete tokens. By keeping the background as continuous features and only tokenizing the "edit zone," the model maintains higher fidelity and generates much faster.
  • Efficiency through Sparsity: Because the model only autoregressively generates the masked tokens, the computational cost scales with the size of the edit, not the size of the whole image.
  • Dramatic Speedup: In face editing tasks (FFHQ), iLAT reduced inference time from 8.18 seconds (vanilla AR) to just 1.18 seconds per image—a nearly 7x speedup.
  • Quantitative Superiority: On the Penn Action dataset, iLAT achieved an FID (Fréchet Inception Distance) of 27.36, significantly outperforming the "Taming Transformer" baseline (33.53) and GAN-based methods like PATN (82.79).
  • Qualitative Consistency: Visual results show that iLAT preserves the identity and background details of the original image much better than previous methods, which often suffered from "identity drift" or blurry artifacts in edited regions.

Limitations and Future Work

The authors note a few areas for improvement:

  • Sensitivity to Guidance: In face editing, the model's performance is somewhat dependent on the quality of the input sketch; unprofessional or messy sketches can lead to unnatural results.
  • Task-Specific Tuning: The GAN learning weight required manual fixing for face editing to maintain stability, suggesting the adversarial training component could be made more robust.
  • Video Potential: While the paper shows a brief example of sequential video generation, fully extending iLAT to temporally consistent video editing remains a promising future direction.

Summary updated

Abstract

Recently, AutoRegressive (AR) models for the whole image generation empowered by transformers have achieved comparable or even better performance compared to Generative Adversarial Networks (GANs). Unfortunately, directly applying such AR models to edit/change local image regions, may suffer from the problems of missing global information, slow inference speed, and information leakage of local guidance. To address these limitations, we propose a novel model -- image Local Autoregressive Transformer (iLAT), to better facilitate the locally guided image synthesis. Our iLAT learns the novel local discrete representations, by the newly proposed local autoregressive (LA) transformer of the attention mask and convolution mechanism. Thus iLAT can efficiently synthesize the local image regions by key guidance information. Our iLAT is evaluated on various locally guided image syntheses, such as pose-guided person image synthesis and face editing. Both quantitative and qualitative results show the efficacy of our model.
BibTeX
@inproceedings{cao2021image,
  author = {Chenjie Cao and Yuxin Hong and Xiang Li and Chengrong Wang and Chengming Xu and Yanwei Fu and Xiangyang Xue},
  title = {{The Image Local Autoregressive Transformer}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=u6j2Wd_wA8},
}
Figure