TL;DR
L2P (Latent-to-Pixel) is an efficient transfer learning framework that migrates the rich semantic priors of pre-trained Latent Diffusion Models (LDMs) directly into pure pixel-space models using just 8 GPUs and zero real-world data. By discarding the VAE bottleneck, L2P achieves high-quality text-to-image synthesis and seamlessly unlocks native 4K ultra-high-resolution generation.
What Problem Does This Paper Address?
Most state-of-the-art text-to-image models are Latent Diffusion Models (LDMs). They use a Variational Autoencoder (VAE) to compress images into a lower-dimensional latent space, reducing computational overhead. However, this compression comes at a cost: the VAE inevitably discards high-frequency details, leading to sub-optimal image reconstruction. Furthermore, the VAE's decoding process introduces a severe memory bottleneck, making native 4K generation practically intractable.
While pure pixel-space diffusion models avoid these VAE artifacts, training them from scratch is astronomically expensive, requiring hundreds of high-end GPUs and billions of real-world image-text pairs. L2P addresses this "cold-start" dilemma by directly transferring the world knowledge embedded in pre-trained LDMs into a pixel-space model without the prohibitive training costs.
Methods
The L2P framework adapts a pre-trained LDM (specifically a Diffusion Transformer, or DiT) into a pixel-space model via three main structural and strategic modifications:
- Large-Patch Tokenization: The VAE is completely discarded. Instead, raw pixel inputs are divided into large $16 \times 16$ patches. This aligns the sequence length with the original LDM's latent space, maintaining computational efficiency.
- Detailer Head: To decode the DiT representations back into dense pixel semantics and restore high-frequency details, the authors replace the original final projection layer with a lightweight U-Net architecture called the Detailer Head.
- Selective Freezing: To prevent catastrophic forgetting of the LDM's semantic knowledge, the core intermediate layers of the DiT backbone are frozen. Only the initial input projection layer, the first and last $n$ blocks of the DiT, and the Detailer Head are trained.
Scaling to Native 4K
To scale to ultra-high resolutions, L2P dynamically expands the patch size from $16 \times 16$ to $64 \times 64$ for 4K inputs, keeping the sequence length manageable. Additionally, because dense pixel correlations in 4K space prevent standard noise schedules from fully corrupting the image signal, the authors increase the noise shift parameter. This skews the schedule toward higher noise levels, forcing the model to learn robust global structures rather than trivial local reconstructions.
Datasets
Instead of curating massive real-world datasets, L2P trains exclusively on synthetic data generated by the source LDM itself:
- Synthetic Dataset Pipeline: The authors established a top-down hierarchical taxonomy containing 4 major classes, 17 sub-classes, and over 1,000 fine-grained categories expanded by an LLM.
- Training Corpus: They generated 20k high-quality synthetic images from 10k diverse prompts using the source LDM.
- Evaluation Benchmarks: The model was evaluated on DPG-Bench and GenEval for $1024 \times 1024$ generation, and UltraHR-eval4k (using a subset of the UltraHR-100K dataset) for 4K evaluation.
Key Insights and Ideas
- Fitting a Smooth Manifold: Training a pixel-space model on synthetic data from a source LDM is significantly faster and more effective than using real-world data. The synthetic data forces the pixel model to fit an already optimized, smooth data manifold constructed by the LDM, bypassing the steep optimization landscape of from-scratch training.
- Modality Isolation via Shallow Tuning: Freezing the deep layers of the DiT and tuning only the shallowest input/output layers successfully handles the latent-to-pixel modality transformation while perfectly preserving core generative knowledge.
- Overcoming Signal Leakage in 4K: Simply scaling up resolution causes standard diffusion models to fail because pixels are highly redundant. Shifting the noise schedule to be heavier is the key mechanic that forces the transformer to focus on global composition rather than pixel-level shortcuts.
Key Results
- Drastic Efficiency Gains: L2P successfully migrates latent priors using only 8 GPUs. For native 4K generation, L2P reduces single-step inference latency by 97.67% and peak GPU memory footprint by 38.81% compared to the source latent baseline.
- Benchmark Performance: On DPG-Bench, L2P scored 86.00, slightly outperforming its source LDM baseline (Z-Image-turbo at 84.86). On GenEval, it retained 93.6% of the source model's performance while demonstrating significantly higher structural and generative diversity than competing pixel-space models like Deco and PixelGen.
- 4K Superiority: L2P achieved the lowest Frechet Inception Distance (FID of 33.46) and $\text{FID}_{\text{patch}}$ (21.77) on the UltraHR-eval4k benchmark, mitigating common artifacts like over-smoothing.
Limitations and Future Work
- Upper-Bounded by the Source LDM: Because L2P relies entirely on synthetic data from the source LDM, its ultimate semantic comprehension and compositional capabilities are upper-bounded by that specific source model's priors.
- Unexplored Pixel-Space Regularizations: Operating natively in pixel space opens the door for task-specific loss functions (e.g., perceptual or physics-based constraints). The authors left the integration of these direct pixel regularizations as a promising direction for future downstream application research.
Summary updated