Abstract
The diffusion transformer (DiT) architecture has attracted significant attention in image generation, achieving better fidelity, performance, and diversity. However, most existing DiT-based image generation methods focus on global-aware synthesis, and regional prompt control has been less explored. In this paper, we propose a coarse-to-fine generation pipeline for regional prompt-following generation. Specifically, we first utilize the powerful large language model (LLM) to generate both high-level descriptions of the image (such as content, topic, and objects) and low-level descriptions (such as details and style). Then, we explore the influence of cross-attention layers at different depths. We find that deeper layers are always responsible for high-level content control, while shallow layers handle low-level content control. Various prompts are injected into the proposed regional cross-attention control for coarse-to-fine generation. By using the proposed pipeline, we enhance the controllability of DiT-based image generation. Extensive quantitative and qualitative results show that our pipeline can improve the performance of the generated images.
Introduction
Diffusion Transformers (DiTs) have emerged as a powerful backbone for high-fidelity image generation, utilized in state-of-the-art models like Sora and Hunyuan-DiT. Despite their success in global synthesis, these models often struggle with precise regional prompt following. When presented with complex instructions involving multiple objects and specific spatial constraints, standard DiTs frequently exhibit "attribute leakage" or ignore localized details. This paper introduces DiTPipe, a coarse-to-fine generation pipeline designed to address these limitations by providing fine-grained regional control within the DiT framework.
Methodology
The DiTPipe framework introduces a modular approach to regional control through three primary innovations:
Progressive Prompt Generation
To handle complex natural language, the system employs a Large Language Model (LLM) to decompose a single user prompt into a hierarchy of instructions. This includes high-level prompts that define the overall layout and global theme, and low-level prompts that specify regional details such as textures and colors.
Controllable Region-Attention (CRA)
The core of the architectural modification is the Controllable Region-Attention (CRA) module. Unlike standard cross-attention which applies a global prompt to the entire latent space, CRA utilizes spatial masks. It applies these masks to the image feature "Queries," allowing them to attend to specific "Keys" and "Values" derived from regional prompts. This ensures that specific areas of the image are influenced only by their corresponding instructions.
Coarse-to-Fine Layer-wise Analysis
Empirical analysis revealed a functional hierarchy within DiT layers: deeper layers primarily govern high-level structural layout, while shallower layers manage fine-grained details. DiTPipe leverages this discovery by progressively injecting prompts across different depths, aligning the prompt complexity with the layer's specialized role.
Experimental Results
The proposed pipeline was evaluated against established baselines, including SDXL (RealVisXL v5), SD-1.5 (majicMIX realistic v7), and vanilla Hunyuan-DiT.
Quantitative Performance
DiTPipe demonstrated superior performance in image reconstruction and fidelity metrics. Specifically, it achieved a PSNR of 28.2 and an rFID of 4.2, notably outperforming the SDXL-VAE baseline (24.7 PSNR and 4.4 rFID).
Qualitative Spatial Accuracy
The model's ability to adhere to spatial constraints was tested using complex multi-chunk grids. While baseline models often failed to maintain distinct attributes across regions—for instance, blending colors between adjacent objects—DiTPipe successfully generated distinct elements in their designated regions without interference.
Conclusion
DiTPipe enhances the controllability of DiT-based image generation through a progressive, coarse-to-fine pipeline. By combining LLM-based prompt decomposition with a novel regional attention mechanism, the model achieves higher fidelity and better adherence to complex spatial instructions. Future work will explore integrating additional control modalities, such as depth maps and reference images, to further refine the generation process.
Summary updated