Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis

Bingda Tang, Boyang Zheng, Xichen Pan, Sayak Paul, Saining Xie

Abstract

Recent advancements in text-to-image (T2I) synthesis have increasingly relied on Large Language Models (LLMs) to interpret complex prompts. However, most existing architectures utilize "shallow fusion," treating the LLM as a static text encoder. This paper, authored by researchers from NYU and Hugging Face, introduces a "deep fusion" approach that integrates LLMs and Diffusion Transformers (DiTs) at a fundamental architectural level. By sharing self-attention layers between a frozen LLM and a trainable DiT, the proposed FuseDiT model achieves superior text-image alignment and visual quality. Our findings suggest that the "intelligence" of the underlying LLM directly scales to image generation performance, and that simplifying traditional diffusion components—such as timestep conditioning—can lead to more efficient and effective models.

Introduction

Standard text-to-image models, including the Stable Diffusion series, typically employ a decoupled approach to conditioning. They utilize a text encoder (such as CLIP or T5) to extract features from a prompt, which are then injected into the image generator via cross-attention or global pooling. This "shallow fusion" often fails to capture the nuanced semantic relationships required for complex scene synthesis.

There is a fundamental misalignment between the next-token prediction objective of an LLM and the discriminative requirements of a diffusion model. To bridge this gap, this research explores a two-stream architecture where the LLM and DiT interact at every layer. This deep fusion allows the image generation process to leverage the internal representations of the LLM more effectively than simple feature extraction.

Methodology: The FuseDiT Architecture

The authors propose a two-stream transformer configuration designed for maximum synergy between modalities:

  1. The LLM Stream: Utilizes a frozen, pre-trained LLM (e.g., Gemma 2B) to maintain linguistic integrity.
  2. The DiT Stream: A trainable Diffusion Transformer with a configuration (hidden size, layer count, and attention heads) that mirrors the LLM.

Layer-wise Shared Self-Attention

In this architecture, text and image tokens are concatenated at every layer. The DiT stream utilizes shared self-attention to observe the LLM’s internal states. To ensure the LLM remains functional and focused on the text, a causal mask is applied: image tokens can attend to text tokens, but text tokens are prevented from attending to image tokens.

Hybrid Positional Encoding

To handle the different spatial natures of text and images, the model employs a hybrid approach to Rotary Positional Embeddings (RoPE). It uses 1D RoPE for the text sequence and 2D RoPE for the image patches, facilitating a more accurate spatial-linguistic mapping than 1D-only approaches.

Streamlined Conditioning

In a departure from standard DiT designs, the researchers experimented with removing adaLN-Zero, the traditional method for injecting timestep information. Surprisingly, they found that removing this parameter-heavy conditioning actually improved visual quality (FID) while reducing the total parameter count by approximately 20%.

Experimental Setup

The models were trained primarily on CC12M (10.9M pairs) with synthetic captions. For larger-scale evaluations, the dataset was expanded to 26M pairs by including SA-1B and JourneyDB. Performance was measured using GenEval and DPG-Bench for alignment, and FID for visual quality.

Key Insights and Results

The Impact of LLM Scaling

The study confirms that LLM scaling laws extend to image synthesis. Upgrading the base LLM from Gemma 2B to the more advanced Gemma 2 2B resulted in a significant leap in image alignment scores, suggesting that the generative capacity of the DiT is tethered to the linguistic "intelligence" of the LLM stream.

The Instruction Tuning Paradox

One of the more surprising findings was that using an "Instruction Tuned" (chat-optimized) LLM actually degraded T2I performance compared to the base model. This suggests that the internal representations of chat-tuned models may be less suited for the discriminative needs of diffusion processes than the raw representations of base models.

Performance Benchmarks

FuseDiT (2B parameters) achieved a GenEval score of 0.60, outperforming larger industrial models such as SDXL (2.6B) and academic baselines like PixArt-Σ. While shallow fusion models occasionally maintained slightly better raw visual textures (FID), deep fusion consistently dominated in prompt adherence and complex attribute binding.

Computational Efficiency

Despite the "deep" interaction, the shared-attention mechanism proved more efficient than traditional cross-attention. FuseDiT demonstrated a lower inference latency (1.66s) compared to cross-attention methods (1.86s), primarily by eliminating redundant projection layers.

Conclusion and Future Work

FuseDiT demonstrates that deep fusion is a highly effective roadmap for the next generation of T2I models. By aligning the DiT architecture with the LLM and simplifying the conditioning pipeline, the authors achieved state-of-the-art results with a relatively modest parameter count.

Future research directions include exploring modality parity for interleaved text-and-image generation and scaling the deep fusion architecture to larger LLMs (7B+). Additionally, the authors suggest investigating "decoupled" architectures where a smaller DiT might fuse with a much larger LLM through skipped-layer connections to further optimize the balance between performance and computational cost.

Summary updated

BibTeX
@inproceedings{tang2025exploring,
  author = {Bingda Tang and Boyang Zheng and Xichen Pan and Sayak Paul and Saining Xie},
  title = {{Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=bfZmFvE68r},
}
Figure