Iterative Parallel Decoding with Masked Transformers

This direction is merged into another direction: sequential generation bottleneck (autoregressive inference latency). Its last version stays readable and citable here.

Version 1 of 19 papers15 conceptsfrozen 2026-10-04

Generative models for images and video have traditionally relied on autoregressive "raster-scan" generation—predicting one token at a time from top-left to bottom-right. While effective, this sequential approach is computationally expensive and fails to capture the global structure of a scene until the very end. A new paradigm, exemplified by MaskGIT and MaskViT, replaces this linear sequence with a bidirectional approach that treats generation as an iterative refinement of a masked canvas.

The Intuition of Global-to-Local Refinement

The core idea is to start with a completely masked grid of tokens and fill them in over a fixed number of steps. In each iteration, the model predicts all missing tokens simultaneously, but only "commits" to the most confident ones. These committed tokens then serve as anchors for the next round of predictions.

This mimics the way a human artist sketches: first blocking out large shapes (global structure), then refining those shapes, and finally adding fine-grained textures. By using bidirectional attention, researchers like Huiwen Chang and Lu Jiang allow the model to use context from the "future" (the right and bottom of the image) to inform the "past," leading to better structural consistency.

The Role of Concave Mask Schedules

The speed and quality of this process are governed by a mask schedule, which determines how many tokens are revealed at each step. A critical finding across this research is that linear schedules—revealing the same number of tokens every step—are often suboptimal.

Instead, researchers favor concave schedules, such as the cosine function. As noted in MaskGIT and BIGFix, these schedules reveal very few tokens in the early stages. This "slow start" forces the model to be highly selective, ensuring that the initial structural anchors are correct. Once the global layout is stable, the schedule accelerates, filling in the remaining details rapidly. Jiaxin Shi and the team at Google DeepMind further demonstrated that non-linear sampling grids reduce "token conflicts," where a model might otherwise try to predict too many interdependent details at once without enough context.

Self-Correction and Robustness

Because parallel decoding predicts many tokens at once, it is prone to "hallucinating" inconsistent features—such as an object with two heads. To combat this, BIGFix by Victor Besnier et al. introduces a "token fixing" mechanism. By intentionally injecting errors during training, the model learns to not only fill in masks but also to identify and overwrite incorrect tokens during the generation process. This self-correction allows the model to maintain high fidelity even at generation speeds up to 512x faster than autoregressive baselines, as seen in the video prediction work of Agrim Gupta and Stephen Tian.

By moving from sequential prediction to iterative refinement, these models transform generative transformers from slow, one-way processes into dynamic, bidirectional systems capable of real-time synthesis and editing.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_iterative_parallel_decoding_with_masked__a3af4d33,
  title = {Iterative Parallel Decoding with Masked Transformers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/iterative-parallel-decoding-with-masked-transformers/rdr_a3af4d339c5249e4a0a003f44a2ac940},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_iterative_parallel_decoding_with_masked__a3af4d33_v1,
  title = {Iterative Parallel Decoding with Masked Transformers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/iterative-parallel-decoding-with-masked-transformers/rdr_a3af4d339c5249e4a0a003f44a2ac940/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}