L2P: Unlocking Latent Potential for Pixel Generation

Zhennan Chen, Junwei Zhu, Xu Chen, Jiangning Zhang, Jiawei Chen, Zhuoqi Zeng, Wei Zhang, Chengjie Wang, Jian Yang, Ying TaiarXiv 2026 · arXiv

TL;DR

L2P (Latent-to-Pixel) is an efficient transfer learning framework that migrates the rich semantic priors of pre-trained Latent Diffusion Models (LDMs) directly into pure pixel-space models using just 8 GPUs and zero real-world data. By discarding the VAE bottleneck, L2P achieves high-quality text-to-image synthesis and seamlessly unlocks native 4K ultra-high-resolution generation.

What Problem Does This Paper Address?

Most state-of-the-art text-to-image models are Latent Diffusion Models (LDMs). They use a Variational Autoencoder (VAE) to compress images into a lower-dimensional latent space, reducing computational overhead. However, this compression comes at a cost: the VAE inevitably discards high-frequency details, leading to sub-optimal image reconstruction. Furthermore, the VAE's decoding process introduces a severe memory bottleneck, making native 4K generation practically intractable.

While pure pixel-space diffusion models avoid these VAE artifacts, training them from scratch is astronomically expensive, requiring hundreds of high-end GPUs and billions of real-world image-text pairs. L2P addresses this "cold-start" dilemma by directly transferring the world knowledge embedded in pre-trained LDMs into a pixel-space model without the prohibitive training costs.

Figure 1: By leveraging the smooth manifold of pre-trained LDMs, L2P bypasses costly from- scratch training, achieving high-quality generation with just 8 GPUs.

Methods

The L2P framework adapts a pre-trained LDM (specifically a Diffusion Transformer, or DiT) into a pixel-space model via three main structural and strategic modifications:

  1. Large-Patch Tokenization: The VAE is completely discarded. Instead, raw pixel inputs are divided into large $16 \times 16$ patches. This aligns the sequence length with the original LDM's latent space, maintaining computational efficiency.
  2. Detailer Head: To decode the DiT representations back into dense pixel semantics and restore high-frequency details, the authors replace the original final projection layer with a lightweight U-Net architecture called the Detailer Head.
  3. Selective Freezing: To prevent catastrophic forgetting of the LDM's semantic knowledge, the core intermediate layers of the DiT backbone are frozen. Only the initial input projection layer, the first and last $n$ blocks of the DiT, and the Detailer Head are trained.

Scaling to Native 4K

To scale to ultra-high resolutions, L2P dynamically expands the patch size from $16 \times 16$ to $64 \times 64$ for 4K inputs, keeping the sequence length manageable. Additionally, because dense pixel correlations in 4K space prevent standard noise schedules from fully corrupting the image signal, the authors increase the noise shift parameter. This skews the schedule toward higher noise levels, forcing the model to learn robust global structures rather than trivial local reconstructions.

Datasets

Instead of curating massive real-world datasets, L2P trains exclusively on synthetic data generated by the source LDM itself:

  • Synthetic Dataset Pipeline: The authors established a top-down hierarchical taxonomy containing 4 major classes, 17 sub-classes, and over 1,000 fine-grained categories expanded by an LLM.
  • Training Corpus: They generated 20k high-quality synthetic images from 10k diverse prompts using the source LDM.
  • Evaluation Benchmarks: The model was evaluated on DPG-Bench and GenEval for $1024 \times 1024$ generation, and UltraHR-eval4k (using a subset of the UltraHR-100K dataset) for 4K evaluation.
Figure 2: The proposed data construction pipeline. (a) Four-stage construction framework: Hi- erarchical Category Construction, General Prompt Generation, Automated Prompt Filtering, and Image Synthesis. Further details are provided in the Appendix. (b) Category distribution detailing 4 super-classes and 17 sub-classes. (c) Prompt length distribution, peaking at 200–350 characters to provide rich textual details.

Key Insights and Ideas

  • Fitting a Smooth Manifold: Training a pixel-space model on synthetic data from a source LDM is significantly faster and more effective than using real-world data. The synthetic data forces the pixel model to fit an already optimized, smooth data manifold constructed by the LDM, bypassing the steep optimization landscape of from-scratch training.
  • Modality Isolation via Shallow Tuning: Freezing the deep layers of the DiT and tuning only the shallowest input/output layers successfully handles the latent-to-pixel modality transformation while perfectly preserving core generative knowledge.
  • Overcoming Signal Leakage in 4K: Simply scaling up resolution causes standard diffusion models to fail because pixels are highly redundant. Shifting the noise schedule to be heavier is the key mechanic that forces the transformer to focus on global composition rather than pixel-level shortcuts.

Key Results

  • Drastic Efficiency Gains: L2P successfully migrates latent priors using only 8 GPUs. For native 4K generation, L2P reduces single-step inference latency by 97.67% and peak GPU memory footprint by 38.81% compared to the source latent baseline.
Efficiency comparison for 4K generation showing massive reductions in latency and memory.
  • Benchmark Performance: On DPG-Bench, L2P scored 86.00, slightly outperforming its source LDM baseline (Z-Image-turbo at 84.86). On GenEval, it retained 93.6% of the source model's performance while demonstrating significantly higher structural and generative diversity than competing pixel-space models like Deco and PixelGen.
  • 4K Superiority: L2P achieved the lowest Frechet Inception Distance (FID of 33.46) and $\text{FID}_{\text{patch}}$ (21.77) on the UltraHR-eval4k benchmark, mitigating common artifacts like over-smoothing.

Limitations and Future Work

  • Upper-Bounded by the Source LDM: Because L2P relies entirely on synthetic data from the source LDM, its ultimate semantic comprehension and compositional capabilities are upper-bounded by that specific source model's priors.
  • Unexplored Pixel-Space Regularizations: Operating natively in pixel space opens the door for task-specific loss functions (e.g., perceptual or physics-based constraints). The authors left the integration of these direct pixel regularizations as a promising direction for future downstream application research.

Summary updated

Abstract

Pixel diffusion models have recently regained attention for visual generation. However, training advanced pixel-space models from scratch demands prohibitive computational and data resources. To address this, we propose the Latent-to-Pixel (L2P) transfer paradigm, an efficient framework that directly harnesses the rich knowledge of pre-trained LDMs to build powerful pixel-space models. Specifically, L2P discards the VAE in favor of large-patch tokenization and freezes the source LDM's intermediate layers, exclusively training shallow layers to learn the latent-to-pixel transformation. By utilizing LDM-generated synthetic images as the sole training corpus, L2P fits an already smooth data manifold, enabling rapid convergence with zero real-data collection. This strategy allows L2P to seamlessly migrate massive latent priors to the pixel space using only 8 GPUs. Furthermore, eliminating the VAE memory bottleneck unlocks native 4K ultra-high resolution generation. Extensive experiments across mainstream LDM architectures show that L2P incurs negligible training overhead, yet performs on par with the source LDM on DPG-Bench and reaches 93% performance on GenEval.
BibTeX
@misc{chen2026l,
  author = {Zhennan Chen and Junwei Zhu and Xu Chen and Jiangning Zhang and Jiawei Chen and Zhuoqi Zeng and Wei Zhang and Chengjie Wang and Jian Yang and Ying Tai},
  title = {{L2P: Unlocking Latent Potential for Pixel Generation}},
  year = {2026},
  eprint = {2605.12013},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.12013},
}
Figure