TL;DR
This paper challenges the common assumption that target representations (such as raw pixels, VAE latents, or self-supervised features) are interchangeable plug-and-play interfaces for image generators. By evaluating multiple spaces within a unified masked autoregressive framework, the authors demonstrate that different target representations fundamentally redistribute generative difficulty across contextual modeling, per-token denoising, and inference-time control. Crucially, standard properties like reconstruction fidelity or token dimensionality fail to predict how easily a representation space can be learned.
What Problem Does This Paper Address?
When designing modern image generators—like diffusion models or masked autoregressive transformers—a foundational choice is the target representation space in which the model predicts its outputs. Traditionally, researchers treat these spaces as interchangeable: as long as an encoder and decoder exist, the generative architecture is expected to transfer seamlessly.
However, in continuous masked generative models, the problem is dual-natured. The model must perform contextual inference (aggregating visible tokens to understand global structure) and conditional token modeling (predicting the continuous distribution of a missing local token). This paper investigates how different target spaces—raw pixels, Stable Diffusion VAE latents, DINOv2 features, and MAE features—alter the balance between these two tasks, asking: Where does generative difficulty actually move when we change the target representation?
Methods
To isolate the impact of the target space, the authors introduce a unified framework: a masked autoregressive rectified-flow model (FlowMAR).
The architecture splits the generative burden into two distinct components:
- Context Transformer: A bidirectional, MAE-style encoder-decoder that reasons across visible tokens to produce a contextual vector ($z_i$) for each masked location.
- Local Denoiser: A per-token residual MLP (without self-attention) that takes a noisy token, its timestep, and the context vector to predict a rectified-flow velocity target.
The authors fix the spatial layout to a $16 \times 16$ grid (256 tokens) across all experiments and systematically vary the target spaces:
- Raw Pixels: Flattened $16 \times 16 \times 3$ patches (768-dim).
- SD-VAE: Latents from the Stable Diffusion VAE (16-dim).
- DINOv2-B & MAE-B: Features extracted from pretrained vision foundation models (768-dim), mapped back to pixels via frozen pretrained decoders.
They evaluate how each space responds to architectural tweaks (such as denoiser width and direct context fusion) and inference-time sampling configurations.
Datasets
All models are trained and evaluated on the ImageNet-1k dataset for class-conditional generation at $256 \times 256$ resolution. To ensure a fair comparison, all models are restricted to a shared training budget of 200 epochs (~503k optimization steps).
Key Insights and Ideas
- Difficulty is Redistributed, Not Uniformly Removed: A representation that simplifies global context reasoning might complicate local token denoising. For example, DINOv2 features retain clear global semantic layouts even under extreme (95%) masking, allowing for a shallower context transformer. However, its high-dimensional tokens require a much wider local denoiser and direct context fusion to generate successfully.
- Reconstruction Fidelity $\neq$ Generative Learnability: Intuitively, one might think an autoencoder that reconstructs images perfectly would be the easiest to generate within. The authors disprove this by comparing MAE and DINOv2. MAE features achieve superior reconstruction metrics (lower LPIPS and reconstruction FID), yet yield substantially worse generation quality than DINOv2 under the exact same generative architecture.
- Dimensionality is Not Destiny: Both raw pixels and DINOv2 tokens are 768-dimensional. Yet, DINOv2 exhibits the fastest unguided optimization, while pixels exhibit the slowest, proving that token geometry and semantic structure matter far more than raw dimensionality.
- Guidance is Representation-Dependent: Classifier-free guidance (CFG) affects spaces differently. SD-VAE benefits immensely from progressive guidance (increasing scale as more tokens become visible). Pixels and MAE require linear guidance from the very first step. Surprisingly, DINOv2 performs best without any CFG at all.
Key Results
- Optimization Speed: DINOv2 converged significantly faster than all other spaces in both training iterations and accumulated computation (GFLOPs).
- Quantitative Quality: Without classifier-free guidance, DINOv2 achieved an impressive unguided Fréchet Inception Distance (FID) of 6.43, compared to 15.98 for SD-VAE and 43.60 for raw pixels.
- Single-Step Generation: DINOv2 supports highly effective single-step sampling. Using just a single Euler integration step, DINOv2 retained an FID@10k of 20.82 with sharp, recognizable objects. Under the same single-step constraint, SD-VAE degraded to an FID of 139.80, producing only coarse outlines.
- Precision-Recall Trade-off: DINOv2 achieves the highest precision but suffers from low recall (narrower sample diversity). Conversely, SD-VAE provides broader distribution coverage, ultimately achieving the best overall guided FID of 3.19 when paired with progressive CFG.
Limitations and Future Work
- Frozen Decoder Ceiling: Generative quality in the latent and feature spaces remains fundamentally bounded by the reconstruction artifacts of their respective frozen decoders.
- DINOv2's Low Recall: The underlying cause behind DINOv2's limited distribution coverage (low recall) remains unresolved and warrants alternative guidance or representation alignment strategies.
- Scale and Conditioning Constraints: The study was bounded to models between 165M and 222M parameters and class-conditional generation. The authors note that the allocation of generative difficulty across components might shift at larger scales or under text-to-image conditioning regimes.
Summary updated