StageVAR: Stage-Aware Acceleration for Visual Autoregressive Models

Senmao Li, Kai Wang, Salman Khan, Fahad Shahbaz Khan, jian Yang, Yaxing WangSubmission to ICLR 2026 · Rejected

TL;DR

Visual Autoregressive (VAR) models achieve high-quality image generation via coarse-to-fine "next-scale" prediction, but fine-scale steps suffer from severe computational bottlenecks. StageVAR analyzes how image content emerges during inference, dividing generation into semantic establishment, structure establishment, and fidelity refinement stages. By dropping classifier-free guidance (CFG) and applying fast random-projection low-rank approximation strictly in the final fidelity refinement stage, StageVAR achieves up to a 3.4× speedup without any retraining and with virtually no drop in visual quality.

What Problem Does This Paper Address?

Traditional autoregressive (AR) image generation predicts images token-by-token (raster-scan order), requiring thousands of sequential decoding steps. Recent Visual Autoregressive (VAR) architectures instead predict whole token maps across progressively higher resolution scales (e.g., from $16 \times 16$ up to $1024 \times 1024$).

While VAR drastically cuts the total number of autoregressive steps, the sequence length in the final, highest-resolution scale steps explodes. Prior acceleration methods like FastVAR or SkipVAR rely on hand-tuned token pruning or step skipping at arbitrarily chosen large scales. These heuristics overlook how visual information is actually formed across scales, leading to suboptimal trade-offs between speed and perceptual quality.

Methods

StageVAR is a training-free, plug-and-play acceleration pipeline based on two core technical interventions applied exclusively during the final stage of generation:

[Scales 1..32] (Semantic & Structure Stages) ──> Full VAR Attention (Unmodified)
[Scales 40..64] (Fidelity Refinement Stage)   ──> Set CFG = 0 (Bypass Text Prompt)
                                             ──> Random Projection (M tokens -> r tokens)
                                             ──> Run VAR Transformer Block on r tokens
                                             ──> Representative Token Restoration (RTR) via Cache
  1. Semantic Irrelevance (Bypassing Text Conditioning):
    Because prompt alignment is already finalized in earlier scales, StageVAR sets the Classifier-Free Guidance (CFG) scale to 0 in the fidelity refinement stage. This eliminates the text-conditioned forward pass, running only the unconditional pass and delivering an instant ~1.5× speedup.

  2. Random Projection (RP) for Low-Rank Feature Compression:
    Intermediate features at high scales are inherently low-rank. Rather than executing expensive Singular Value Decomposition (SVD) on-the-fly, StageVAR predetermines an optimal rank $r$ offline and projects the $M$-token feature space down to $r$ tokens ($r \ll M$) using a standard Gaussian random projection matrix $Q \in \mathbb{R}^{M \times r}$.

  3. Representative Token Restoration (RTR):
    After running the lightweight transformer forward pass on $r$ tokens, the model must restore the output back to $M$ dimensions. Instead of solving a slow least-squares reconstruction problem, StageVAR samples the $r$ most important token rows based on feature norms, maps the transformer outputs to those indices, and fills the remaining $M - r$ positions using upsampled cached representations from the previous scale step.

Datasets

The authors evaluate StageVAR using several standard image generation benchmarks:

  • GenEval: Evaluates compositional text-to-image alignment across attributes, positions, and object counts.
  • DPG (Dense Prompt Graph): Benchmarks generation quality and alignment under complex, highly descriptive prompts.
  • MS COCO 2014 (30K) & COCO 2017 (5K): Used to assess distribution fidelity via Fréchet Inception Distance (FID), Kernel Inception Distance (KID), and Inception Score (IS).

Models evaluated include Infinity-2B, Infinity-8B, and STAR-1.7B.

Key Insights and Ideas

  • The Three Stages of VAR: Systematic analysis using CLIP, DINO, LPIPS, and DISTS reveals distinct evolutionary phases:
  • Semantic establishment converges first (by scale $16 \times 16 \to 256 \times 256$).
  • Structure establishment stabilizes next (by scale $32 \times 32 \to 512 \times 512$).
  • Fidelity refinement occupies the remaining large scales ($40 \times 40$ through $64 \times 64$), altering only fine local textures.
  • CFG Is Unnecessary for Fine Details: Disabling CFG at late stages causes no drop in semantic fidelity or CLIP score, proving that high-frequency detail generation is essentially prompt-agnostic.
  • Non-Monotonic Low-Rank Regularization: Compressing tokens via random projection exhibits a "peak" phenomenon: keeping ~17.6% rank ($\alpha = 0.96$) not only speeds up inference by 1.8× but slightly improves GenEval and detail clarity over full rank before sharply decaying at lower ranks—a behavior consistent with regularized random projections.

Key Results

  • Infinity-2B: Achieves a 3.4× speedup (latency drops from 2.2s to 0.64s on an RTX 3090) with essentially no performance penalty: GenEval drops by just 0.01 (0.731 $\to$ 0.720) and DPG by 0.26 (83.12 $\to$ 82.86). It outperforms both FastVAR (2.75×, 0.80s) and SkipVAR (2.62×).
  • Infinity-8B: Delivers a 2.7× speedup (4.80s $\to$ 1.77s on an NVIDIA A100) with competitive DPG (86.05 vs. 86.60).
  • STAR-1.7B: Obtains a 1.74× speedup (2.0s $\to$ 1.15s) with identical GenEval scores (0.51).
  • Perceptual Quality: On COCO2014, FID shifts marginally from 26.64 to 26.91. A 69-user study showed that human evaluators considered StageVAR outputs virtually indistinguishable from the baseline (50.5% vs. 49.5% preference for text alignment).

Limitations and Future Work

  • Offline Rank Calibration: The target rank $r$ is calibrated via offline statistics across sample prompt distributions. While experiments show this rank generalizes well across datasets (COCO vs. GenEval), dynamic, per-image rank estimation without SVD overhead remains an open challenge.
  • Untouched Early Stages: Because early and middle scales are kept fully intact to preserve semantic integrity, the speedup ceiling is bounded by the computational cost of the first 10 scales.
  • Extension to Other Modalities: The current study focuses exclusively on text-to-image VAR; extending stage-aware acceleration to unified multimodal understanding-generation architectures (e.g., Chameleon, Show-o) is a natural next step.

Summary updated

Abstract

Visual Autoregressive (VAR) modeling departs from the next-token prediction paradigm of traditional Autoregressive (AR) models through next-scale prediction, enabling high-quality image generation. However, the VAR paradigm suffers from sharply increased computational complexity and running time at large-scale steps. Although existing acceleration methods reduce runtime for large-scale steps, but rely on manual step selection and overlook the varying importance of different stages in the generation process. To address this challenge, we present StageVAR, a systematic study and stage-aware acceleration framework for VAR models. Our analysis shows that early steps are critical for preserving semantic and structural consistency and should remain intact, while later steps mainly refine details and can be pruned or approximated for acceleration. Building on these insights, StageVAR introduces a plug-and-play acceleration strategy that exploits semantic irrelevance and low-rank properties in late-stage computations, without requiring additional training. Our proposed StageVAR achieves up to 3.4× speedup with only a 0.01 drop on GenEval and a 0.26 decrease on DPG, consistently outperforming existing acceleration baselines. These results highlight stage-aware design as a powerful principle for efficient visual autoregressive image generation. Our codes will be open-sourced.
BibTeX
@inproceedings{li2026stagevar,
  author = {Senmao Li and Kai Wang and Salman Khan and Fahad Shahbaz Khan and jian Yang and Yaxing Wang},
  title = {{StageVAR: Stage-Aware Acceleration for Visual Autoregressive Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=C6kxImRizt},
}
Figure