DiT: Dynamic Diffusion Transformer for Accurate Image Generation

Weinan Jia, Mengqi Huang, Nan Chen, Lei Zhang, Zhendong Mao

TL;DR

This paper introduces D²iT, a novel two-stage diffusion model that dynamically compresses image regions based on their information density. By adapting compression levels, D²iT achieves a better balance between global consistency and local realism, leading to higher quality image generation with improved efficiency compared to existing fixed-compression methods like DiT.

What Problem Does This Paper Address?

Current state-of-the-art diffusion models, particularly those based on the Diffusion Transformer (DiT) architecture, face a fundamental limitation: they apply a uniform compression strategy across all image regions. This means that both detailed areas (like a lion's mane) and smooth areas (like a sky) are compressed and processed in the same way. This "one-size-fits-all" approach leads to a trade-off:

  • Too much compression: Loses fine-grained details, resulting in unrealistic local textures.
  • Too little compression: Increases computational cost and can compromise the global coherence of the generated image.

Essentially, these models ignore the inherent variation in information density across different parts of an image, hindering their ability to achieve both high fidelity and global consistency simultaneously.

Methods

D²iT proposes a two-stage framework to address this issue:

  1. Dynamic VAE (DVAE): This is the first stage, responsible for adaptive image compression. Unlike traditional VAEs that use a fixed downsampling rate, DVAE employs a hierarchical encoder with multiple downsampling factors. It analyzes the information density (specifically, local information entropy) of different image regions and assigns them varying levels of compression. This results in a "multi-grained latent code" where detailed regions are represented with more fidelity and smooth regions with less.
  2. Dynamic Diffusion Transformer (D²iT): This is the second stage, where image generation occurs. It builds upon the multi-grained latent codes from DVAE and introduces two key components:
  3. Dynamic Grain Transformer: This module predicts a "grain map" that explicitly encodes the spatial distribution of information complexity (i.e., which regions are coarse and which are fine). This map is derived from the DVAE's analysis.
  4. Dynamic Content Transformer: This module performs the actual diffusion process. It uses the grain map to guide a "multi-grained noise prediction." This means it predicts noise differently for coarse and fine regions. It employs a strategy of rough global noise prediction followed by fine-grained noise correction, allowing it to capture both overall structure and intricate details. A key innovation here is the "Efficient RefineNet" which uses smaller patch sizes and local attention mechanisms to efficiently refine details without incurring prohibitive computational costs.

Datasets

The paper primarily uses the following datasets for training and evaluation:

  • ImageNet: Used for class-conditional generation tasks.
  • FFHQ (Flickr-Faces-HQ): Used for unconditional generation tasks.

The authors also mention using entropy thresholds derived from natural images in the ImageNet dataset to guide their dynamic granularity strategy.

Key Insights and Ideas

  • Information Density is Key: The core insight is that images are not uniformly complex. Recognizing and leveraging this varying information density is crucial for improving generative model performance.
  • Dynamic Compression is Superior: A fixed compression strategy is inherently suboptimal. Adapting compression to local content density leads to better representation and generation.
  • Two-Stage Approach for Synergy: Combining a dynamic compression stage (DVAE) with a dynamically guided generation stage (D²iT) creates a powerful synergy, allowing for both accurate latent representation and context-aware noise prediction.
  • Global Consistency + Local Realism: The D²iT framework explicitly aims to unify these two often conflicting goals by using a coarse-to-fine noise prediction strategy guided by the grain map.
  • Efficiency Gains: By focusing computational effort on detailed regions and using simpler representations for smooth ones, the model can achieve better quality with potentially less computational overhead or faster convergence.

Key Results

  • ImageNet: D²iT-L achieved an FID score of 1.73, a 23.8% improvement over DiT-XL (FID 2.27), using only 57.1% of the training resources.
  • FFHQ: D²iT-L achieved an FID-10K score of 4.47, a 28.6% improvement over DiT-L (FID 6.26).
  • Ablation Studies:
    • Introducing the DVAE and routing module improved FID from 34.67 to 29.10.
  • Further improvements were seen with dynamic grain loss (FID 27.62).
  • Replacing standard DiT backbone layers with the Efficient RefineNet achieved the best FID of 22.11.
  • Qualitative Results: Figures 3 and 4 show generated images where the "grain maps" (visual representations of the dynamic compression strategy) align well with the generated image details, demonstrating the effectiveness of the dynamic approach.

Limitations and Future Work

  • Limited Granularities: The current D²iT model uses two granularities. The authors suggest exploring more granularities in future work to further refine the dynamic diffusion process.
  • Computational Trade-offs: While D²iT aims for efficiency, the paper notes that blindly increasing RefineNet layers can damage global consistency. Finding the optimal balance of layers between the DiT backbone and RefineNet is important.
  • Future Directions: The authors propose exploring more granularities and potentially integrating this dynamic strategy into other diffusion model architectures.

Summary updated

Abstract

Diffusion models are widely recognized for their ability to generate high-fidelity images. Despite the excellent performance and scalability of the Diffusion Transformer (DiT) architecture, it applies fixed compression across different image regions during the diffusion process, disregarding the naturally varying information densities present in these regions. However, large compression leads to limited local realism, while small compression increases computational complexity and compromises global consistency, ultimately impacting the quality of generated images. To address these limitations, we propose dynamically compressing different image regions by recognizing the importance of different regions, and introduce a novel two-stage framework designed to enhance the effectiveness and efficiency of image generation: (1) Dynamic VAE (DVAE) at first stage employs a hierarchical encoder to encode different image regions at different downsampling rates, tailored to their specific information densities, thereby providing more accurate and natural latent codes for the diffusion process. (2) Dynamic Diffusion Transformer (D$^2$iT) at second stage generates images by predicting multi-grained noise, consisting of coarse-grained (less latent code in smooth regions) and fine-grained (more latent codes in detailed regions), through an novel combination of the Dynamic Grain Transformer and the Dynamic Content Transformer. The strategy of combining rough prediction of noise with detailed regions correction achieves a unification of global consistency and local realism. Comprehensive experiments on various generation tasks validate the effectiveness of our approach. Code will be released at https://github.com/jiawn-creator/Dynamic-DiT.
BibTeX
@inproceedings{jia2025dit,
  author = {Weinan Jia and Mengqi Huang and Nan Chen and Lei Zhang and Zhendong Mao},
  title = {{DiT: Dynamic Diffusion Transformer for Accurate Image Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01200},
}
Figure