Where Does Generative Difficulty Reside? An Empirical Study of Target Representations

Marcel Plocher, Bernhard Schölkopf, Andreas Geiger, Gege GaoarXiv 2026 · arXiv

TL;DR

This paper challenges the common assumption that target representations (such as raw pixels, VAE latents, or self-supervised features) are interchangeable plug-and-play interfaces for image generators. By evaluating multiple spaces within a unified masked autoregressive framework, the authors demonstrate that different target representations fundamentally redistribute generative difficulty across contextual modeling, per-token denoising, and inference-time control. Crucially, standard properties like reconstruction fidelity or token dimensionality fail to predict how easily a representation space can be learned.

What Problem Does This Paper Address?

When designing modern image generators—like diffusion models or masked autoregressive transformers—a foundational choice is the target representation space in which the model predicts its outputs. Traditionally, researchers treat these spaces as interchangeable: as long as an encoder and decoder exist, the generative architecture is expected to transfer seamlessly.

However, in continuous masked generative models, the problem is dual-natured. The model must perform contextual inference (aggregating visible tokens to understand global structure) and conditional token modeling (predicting the continuous distribution of a missing local token). This paper investigates how different target spaces—raw pixels, Stable Diffusion VAE latents, DINOv2 features, and MAE features—alter the balance between these two tasks, asking: Where does generative difficulty actually move when we change the target representation?

Methods

To isolate the impact of the target space, the authors introduce a unified framework: a masked autoregressive rectified-flow model (FlowMAR).

Fig. 1: Overview of our masked autoregressive rectified-flow model. (a) Dur- ing training, an image is encoded into continuous tokens (or directly patchified in pixel space), randomly masked, and processed by a bidirectional transformer to obtain con- textual features zi. A local denoiser predicts the rectified-flow target for each masked token. (b) During sampling, the model starts from a fully masked grid and alternates between context prediction and conditional token generation until all positions are filled. Blue and orange denote frozen and trainable modules, respectively.

The architecture splits the generative burden into two distinct components:

  1. Context Transformer: A bidirectional, MAE-style encoder-decoder that reasons across visible tokens to produce a contextual vector ($z_i$) for each masked location.
  2. Local Denoiser: A per-token residual MLP (without self-attention) that takes a noisy token, its timestep, and the context vector to predict a rectified-flow velocity target.

The authors fix the spatial layout to a $16 \times 16$ grid (256 tokens) across all experiments and systematically vary the target spaces:

  • Raw Pixels: Flattened $16 \times 16 \times 3$ patches (768-dim).
  • SD-VAE: Latents from the Stable Diffusion VAE (16-dim).
  • DINOv2-B & MAE-B: Features extracted from pretrained vision foundation models (768-dim), mapped back to pixels via frozen pretrained decoders.

They evaluate how each space responds to architectural tweaks (such as denoiser width and direct context fusion) and inference-time sampling configurations.

Datasets

All models are trained and evaluated on the ImageNet-1k dataset for class-conditional generation at $256 \times 256$ resolution. To ensure a fair comparison, all models are restricted to a shared training budget of 200 epochs (~503k optimization steps).

Key Insights and Ideas

  • Difficulty is Redistributed, Not Uniformly Removed: A representation that simplifies global context reasoning might complicate local token denoising. For example, DINOv2 features retain clear global semantic layouts even under extreme (95%) masking, allowing for a shallower context transformer. However, its high-dimensional tokens require a much wider local denoiser and direct context fusion to generate successfully.
  • Reconstruction Fidelity $\neq$ Generative Learnability: Intuitively, one might think an autoencoder that reconstructs images perfectly would be the easiest to generate within. The authors disprove this by comparing MAE and DINOv2. MAE features achieve superior reconstruction metrics (lower LPIPS and reconstruction FID), yet yield substantially worse generation quality than DINOv2 under the exact same generative architecture.
  • Dimensionality is Not Destiny: Both raw pixels and DINOv2 tokens are 768-dimensional. Yet, DINOv2 exhibits the fastest unguided optimization, while pixels exhibit the slowest, proving that token geometry and semantic structure matter far more than raw dimensionality.
  • Guidance is Representation-Dependent: Classifier-free guidance (CFG) affects spaces differently. SD-VAE benefits immensely from progressive guidance (increasing scale as more tokens become visible). Pixels and MAE require linear guidance from the very first step. Surprisingly, DINOv2 performs best without any CFG at all.

Key Results

  • Optimization Speed: DINOv2 converged significantly faster than all other spaces in both training iterations and accumulated computation (GFLOPs).
  • Quantitative Quality: Without classifier-free guidance, DINOv2 achieved an impressive unguided Fréchet Inception Distance (FID) of 6.43, compared to 15.98 for SD-VAE and 43.60 for raw pixels.
Fig. 2: Unguided optimization across target representations.FID (a) and In- ception Score (b) over 503k training steps, and FID versus accumulated training com- putation (c). DINOv2-B converges fastest under both iteration- and compute-based comparisons; SD-VAE remains intermediate, while the pixel model continues to im- prove at the end of the shared budget.
  • Single-Step Generation: DINOv2 supports highly effective single-step sampling. Using just a single Euler integration step, DINOv2 retained an FID@10k of 20.82 with sharp, recognizable objects. Under the same single-step constraint, SD-VAE degraded to an FID of 139.80, producing only coarse outlines.
  • Precision-Recall Trade-off: DINOv2 achieves the highest precision but suffers from low recall (narrower sample diversity). Conversely, SD-VAE provides broader distribution coverage, ultimately achieving the best overall guided FID of 3.19 when paired with progressive CFG.

Limitations and Future Work

  • Frozen Decoder Ceiling: Generative quality in the latent and feature spaces remains fundamentally bounded by the reconstruction artifacts of their respective frozen decoders.
  • DINOv2's Low Recall: The underlying cause behind DINOv2's limited distribution coverage (low recall) remains unresolved and warrants alternative guidance or representation alignment strategies.
  • Scale and Conditioning Constraints: The study was bounded to models between 165M and 222M parameters and class-conditional generation. The authors note that the allocation of generative difficulty across components might shift at larger scales or under text-to-image conditioning regimes.

Summary updated

Abstract

The target representation defines the distribution an image generator must learn, yet it is often treated as an interchangeable interface. This assumption is particularly questionable for continuous masked generators, which combine contextual inference from visible tokens with conditional modeling of each missing token. We study raw pixels, SD-VAE latents and DINOv2 as well as MAE representation-autoencoder features within a unified masked autoregressive rectified-flow model. Under a shared ImageNet training budget, these spaces exhibit distinct optimization and inference regimes. DINOv2 converges fastest in both iterations and computation but benefits strongly from a wider local denoiser and direct context fusion. Pixels optimize substantially more slowly and require a different prediction, masking, and guidance configuration. MAE reconstructs images more faithfully and exhibits clear semantic clustering, yet produces generations substantially worse than DINOv2. The representations also respond differently to classifier-free guidance and occupy distinct precision-recall trade-offs. Together, our results show that compression, reconstruction fidelity, token dimensionality, and visible semantic clustering do not individually predict generative behavior. Instead, target representations redistribute difficulty across contextual modeling, per-token denoising, and inference-time distributional control.
BibTeX
@misc{plocher2026where,
  author = {Marcel Plocher and Bernhard Schölkopf and Andreas Geiger and Gege Gao},
  title = {{Where Does Generative Difficulty Reside? An Empirical Study of Target Representations}},
  year = {2026},
  eprint = {2608.00626},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.00626},
}
Figure