ZipIR: Latent Pyramid Diffusion Transformer for High-Resolution Image Restoration

Yongsheng Yu, Haitian Zheng, Zhifei Zhang, Jianming Zhang, Yuqian Zhou, Connelly Barnes, Yuchen Liu, Wei Xiong, Zhe Lin, Jiebo LuoarXiv 2025 · DOI · arXiv

Introduction

Generative diffusion models have set new benchmarks in image restoration (IR), yet they face a significant "computational wall" when scaling to ultra-high resolutions like 2K or 4K. Standard models typically operate on a latent space with 8x compression. At high resolutions, the resulting spatial token count leads to quadratic complexity in self-attention mechanisms, making processing prohibitively slow and memory-intensive. ZipIR addresses this bottleneck by introducing a novel framework that achieves state-of-the-art restoration with unprecedented efficiency.

Architectural Innovations

ZipIR overcomes the limitations of traditional diffusion-based IR through three core innovations:

1. Latent Pyramid VAE (LP-VAE)

To reduce computational load, ZipIR employs a 32x spatial compression factor ($f=32$). To mitigate the information loss typically associated with such high compression, the authors developed a Latent Pyramid structure. This VAE captures information across multiple scales—Macro (1/32), Coarse (1/4), and Fine (original)—and concatenates them into a 64-channel latent representation. This hierarchical approach ensures that both global structure and fine-grained details are preserved.

2. Diffusion Transformer (DiT) Backbone

The 32x compression significantly reduces the token count, allowing the use of a high-capacity 3B-parameter Diffusion Transformer (DiT) instead of a traditional UNet. This shift enables the model to be trained on entire high-resolution images rather than small, local patches. Consequently, ZipIR can model holistic scene structures more effectively, avoiding the artifacts and seams common in tiling-based methods.

3. Pixel-aware Decoder

Heavy compression can sometimes distort low-level textures. ZipIR utilizes a specialized decoder equipped with skip connections from a "degradation-aware feature extractor." This allows the model to retrieve fine spatial details directly from the low-quality input, complementing the denoised latent code during the final reconstruction phase.

Experimental Evaluation

The model was trained on a massive dataset of 300 million curated stock images. For evaluation, the authors introduced Pexels-3K, a new benchmark of 3,000 2K-resolution photos, to better test ultra-high-res performance. Standard benchmarks like RealPhoto60, RealSR, and DrealSR were also used for comparison.

Key Results

ZipIR demonstrates a paradigm shift in both speed and visual fidelity for high-resolution restoration.

Efficiency and Scalability

ZipIR processes a 2K image in just 6.9 seconds on an A100 GPU, representing a 10.7x speedup over SeeSR and a 7.7x speedup over SUPIR. Furthermore, while competitors often encounter "Out of Memory" (OOM) errors at resolutions beyond 3K, ZipIR successfully scales to 4K and even 8K resolutions.

Inference time comparison at 2K resolution.

Visual Fidelity

Quantitatively, ZipIR achieves superior FID and Patch-FID scores. Qualitatively, the model excels at restoring intricate textures—such as fur, skin pores, and fabric weaves—that are often lost or blurred by other methods.

Qualitative comparison of low-quality input vs. ZipIR output.

Conclusion

By rethinking latent space compression and leveraging the scalability of Transformers, ZipIR breaks the resolution barrier in diffusion-based image restoration. Future work will explore even higher compression rates for real-time 8K restoration and the application of the Latent Pyramid approach to video restoration, where temporal consistency and efficiency are paramount.

Summary updated

Abstract

Recent progress in generative models has significantly improved image restoration capabilities, particularly through powerful diffusion models that offer remarkable recovery of semantic details and local fidelity. However, deploying these models at ultra-high resolutions faces a critical trade-off between quality and efficiency due to the computational demands of long-range attention mechanisms. To address this, we introduce ZipIR, a novel framework that enhances efficiency, scalability, and long-range modeling for high-res image restoration. ZipIR employs a highly compressed latent representation that compresses image 32x, effectively reducing the number of spatial tokens, and enabling the use of high-capacity models like the Diffusion Transformer (DiT). Toward this goal, we propose a Latent Pyramid VAE (LP-VAE) design that structures the latent space into sub-bands to ease diffusion training. Trained on full images up to 2K resolution, ZipIR surpasses existing diffusion-based methods, offering unmatched speed and quality in restoring high-resolution images from severely degraded inputs.
BibTeX
@misc{yu2025zipir,
  author = {Yongsheng Yu and Haitian Zheng and Zhifei Zhang and Jianming Zhang and Yuqian Zhou and Connelly Barnes and Yuchen Liu and Wei Xiong and Zhe Lin and Jiebo Luo},
  title = {{ZipIR: Latent Pyramid Diffusion Transformer for High-Resolution Image Restoration}},
  year = {2025},
  eprint = {2504.08591},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.08591},
}
Figure