TL;DR
Training diffusion transformers is notoriously slow and computationally expensive due to a mismatch between generative and discriminative representations. This paper introduces Visual Representation Policy Optimization (VRPO), a reinforcement learning framework that replaces rigid, static alignment losses with a dynamic, reward-guided objective. By optimizing for fidelity, semantics, and stability, VRPO achieves up to a +1.8 FID improvement and converges 2.3× faster than previous state-of-the-art alignment methods.
What Problem Does This Paper Address?
While Diffusion Transformers (like DiT and SiT) generate high-quality images, training them requires massive compute budgets. Recent methods like REPA try to accelerate training by forcing the diffusion model's internal features to align with a pretrained, frozen vision encoder (like DINOv2).
However, these existing approaches rely on static similarity losses (like fixed cosine distance). The authors point out a major flaw here: static losses treat feature distributions as temporally invariant. They ignore the fact that image quality and semantic content evolve drastically across different noise levels and training steps. This rigidity leads to "over-alignment" (where the model overfits to early-phase representations at the expense of fine details) or "under-alignment" (failing to capture semantic structure), ultimately slowing down convergence.
Methods
Instead of treating representation alignment as a rigid constraint, VRPO reformulates the denoising process as a stochastic policy ($\pi_\theta$) optimized via reinforcement learning.
The core of VRPO is a Composite Reward Function ($R$) that balances three distinct objectives:
- Semantic Reward ($R_{\text{semantic}}$): Measures the cosine similarity between the diffusion model's internal features and the frozen DINOv2 embeddings to maintain high-level structural meaning.
- Fidelity Reward ($R_{\text{fidelity}}$): Uses batch-level proxy statistics for FID and LPIPS to ensure the generated images remain perceptually realistic and sharp.
- Stability Reward ($R_{\text{stability}}$): Enforces smooth feature transitions between consecutive denoising timesteps, preventing representation drift.
To stabilize training, the total reward is dynamically scaled using an inverse-variance weighting rule, meaning more reliable (lower variance) reward signals automatically receive higher weights. Furthermore, VRPO uses Layer-wise Reinforcement: reward gradients are selectively injected only into early-to-mid transformer layers (which handle global layout and semantics), leaving deeper layers untouched to focus purely on high-frequency texture reconstruction.
Datasets
The authors evaluated their approach using the following benchmarks:
- ImageNet-256×256: The primary dataset used for class-conditional image generation training and evaluation.
- CIFAR-10 & CelebA-HQ: Used strictly for zero-shot cross-domain transfer evaluation to test the robustness of the learned representations without any fine-tuning.
Key Insights and Ideas
- Dynamic Self-Calibration: Unlike static losses that apply uniform supervision throughout training, VRPO's reward-driven framework naturally shifts its focus. Early in training, when outputs are noisy, it prioritizes semantic structure. As semantics stabilize, it automatically shifts optimization pressure toward fine-grained visual fidelity.
- Inference-Time Persistence: In prior methods, the alignment mechanism is an auxiliary loss that vanishes during inference. Because VRPO bakes the alignment directly into the policy gradient optimization, the semantic consistency naturally persists at test time without needing extra modules.
- Structural Disentanglement: Feature activation maps reveal that VRPO forces the model to sharply isolate foreground objects from backgrounds at the latent level before synthesizing textures, leading to significantly better prompt adherence.
Key Results
- Superior Image Quality: On ImageNet-256×256, VRPO achieved a Fréchet Inception Distance (FID) of 4.12, outperforming REPA (5.93) and the baseline DiT (7.95).
- Massive Speedup: VRPO reached convergence in just 170K iterations, making it 2.3× faster than REPA (which required 400K iterations) under identical compute budgets.
- Minimal Overhead: Computing the scalar rewards adds only about 3% computational overhead per iteration.
- Excellent Generalization: When transferred zero-shot to CIFAR-10 and CelebA-HQ, VRPO retained over 92% of its representation alignment performance, beating REPA by 7.6%.
Limitations and Future Work
Despite its efficiency, calculating the fidelity reward proxy during the forward pass requires tracking exponential moving average (EMA) statistics across batches, which increases VRAM consumption slightly. Additionally, because VRPO relies on a frozen vision encoder, its performance is fundamentally bounded by the biases and domain distribution of that encoder (e.g., DINOv2). If applied to highly specialized domains like medical imaging or satellite data, the semantic rewards might become suboptimal.
For future work, the authors suggest expanding the reinforcement learning framework to include negative rewards, which could explicitly penalize the generation of unsafe, biased, or low-quality content.
Summary updated