TL;DR
Visual Autoregressive (VAR) modeling recently proved that coarse-to-fine next-scale prediction can rival diffusion models, but its full-attention mechanism scales quadratically across scales. M-VAR decouples this process: it keeps bidirectional self-attention for short, local intra-scale interactions, but replaces cross-scale attention with Mamba (a state-space model with linear complexity) for inter-scale progression. The result is a faster, leaner autoregressive generator that achieves state-of-the-art results, scoring 1.78 FID on ImageNet $256 \times 256$.
What Problem Does This Paper Address?
Standard autoregressive visual models flatten 2D images into 1D sequences and predict tokens sequentially (raster scan). VAR challenged this by predicting entire feature maps across scales (e.g., $1\times1 \to 2\times2 \to \dots \to 16\times16$). However, VAR models all tokens across all scales using standard attention mechanisms.
The authors discover an architectural inefficiency:
- Intra-scale tokens (within the same resolution level) consume ~80% of the attention scores but only ~25% of the computational cost.
- Inter-scale tokens (across resolution levels) dominate sequence lengths and eat up ~75% of the compute, despite receiving only ~20% of the attention budget.
Standard transformers waste immense compute attending between scales using quadratic attention when that capacity is better targeted elsewhere.
Methods
M-VAR decouples next-scale prediction into a two-stage hybrid architecture:
Input Scales: [C] s_1 s_2 ...
↓ ↓ ↓
[ Attention ] [ Attention ] [ Attention ] (Intra-Scale: Shared parameters,
↓ ↓ ↓ captures local spatial detail)
s'_1 s'_2 s'_3
└───────────────┬────────────────┘
↓
[ Concat & Mamba ] (Inter-Scale: Linear complexity SSM,
↓ models coarse-to-fine progression)
Output
- Intra-Scale Modeling (Bidirectional Attention): Each scale is processed independently and bidirectionally using standard self-attention (accelerated via FlashAttention). Because the sequence length within each separate scale is relatively small, quadratic scaling is inexpensive, fully capturing 2D spatial context and high-frequency details. Weights are shared across scales.
- Inter-Scale Modeling (Mamba SSM): The intra-scale representations are concatenated sequentially from coarse to fine and fed into Mamba layers. Mamba provides linear $O(N)$ computational complexity with respect to sequence length, easily handling sequences up to 2,240 tokens without the prohibitive quadratic attention bottleneck.
Datasets
- ImageNet-1K ($256 \times 256$ and $512 \times 512$): Standard benchmark for class-conditional image generation. No new benchmarks or datasets were introduced.
Key Insights and Ideas
- Attention distribution $\neq$ computational expense: Empirical profiling of VAR revealed that 79.6% of the attention weight focuses on same-scale relationships, yet 76.1% of the compute is burned modeling cross-scale connections.
- Decoupling representation roles: High-frequency visual synthesis needs full bidirectional awareness locally (which attention excels at), but hierarchical scale-stepping is fundamentally an ordered, coarse-to-fine sequence that Mamba's causal state-space formulation models naturally and efficiently.
- Pure intra-scale is insufficient: Ablations show that completely dropping inter-scale modeling collapses generation quality (FID drops from 3.55 to 7.17), confirming that cross-scale conditioning is mandatory—it just doesn't require quadratic attention.
Key Results
- ImageNet $256 \times 256$:
- M-VAR-d32 (3.0B) achieves 1.78 FID (and 1.67 FID with rejection sampling), outperforming prior autoregressive baselines (VAR-d30: 1.97, LlamaGen-3B: 2.18) and diffusion baselines (DiT-XL/2: 2.27, L-DiT-3B: 2.10).
- M-VAR-d24 (1.5B) reaches 1.93 FID, outperforming the baseline VAR-d30 (2.0B) while utilizing 25% fewer parameters and running 14% faster.
- ImageNet $512 \times 512$:
- M-VAR-d24 scores 2.65 FID, matching the baseline VAR-d36 (2.63 FID) while cutting inference time in half ($1\times$ vs. $2\times$ relative cost).
Limitations and Future Work
- Strictly Class-Conditional: The experiments focus entirely on class-conditional ImageNet generation. Adapting M-VAR to text-to-image synthesis or multimodal settings remains unverified.
- Resolution Ceilings: Experiments top out at $512 \times 512$; testing whether Mamba's linear scaling advantages widen at megapixel resolutions ($1024\times1024$ and beyond) is a logical next step.
- Heavier Scan Exploration: The inter-scale module uses basic unidirectional Mamba scans; more advanced multidirectional SSM routing within the hierarchy was not explored.
Summary updated