An Empirical Study and Analysis of Text-to-Image Generation Using Large Language Model-Powered Textual Representation

Zhiyu Tan, Mengping Yang, Luozheng Qin, Hao Yang, Ye Qian, Qiang Zhou, Cheng Zhang, Hao Li

Introduction

Modern text-to-image (T2I) synthesis has been revolutionized by diffusion models, yet many state-of-the-art systems remain tethered to the limitations of CLIP-based text encoders. CLIP often struggles with multilingual prompts, enforces a restrictive 77-token context window, and lacks the deep semantic understanding found in modern Large Language Models (LLMs). While some models like DALL-E 3 integrate LLMs, the computational cost of training such systems from scratch is prohibitive. This paper introduces OmniDiffusion, a framework designed to efficiently bridge the gap between powerful LLMs and existing diffusion models.

Methodology: The OmniDiffusion Pipeline

OmniDiffusion utilizes a frozen LLM (Baichuan2-7B) coupled with a lightweight, 4-layer transformer-based adapter. This adapter, representing only 4.6% of the total model parameters (487M out of 10.6B), maps textual features into the visual latent space of a diffusion model (SD-XL). The training process follows a rigorous three-stage pipeline:

  1. Multilingual Textual Alignment: The adapter is trained to align LLM features with the CLIP embedding space. A "segmented encoding" strategy is employed to bypass token limits, and a novel loss function ensures alignment of both feature direction and magnitude.
  2. End-to-End Text-Image Training: The adapter and UNet are jointly fine-tuned on a massive dataset of 43 million image-text pairs, re-captioned by multimodal LLMs for higher descriptive accuracy.
  3. High-Aesthetic Fine-tuning: A final refinement stage uses 40,000 curated, high-quality images to maximize the visual appeal of the generated outputs.

Key Insights and Technical Innovations

A critical finding of this study is the importance of the magnitude constraint. The researchers discovered that aligning only the cosine similarity (direction) of LLM and CLIP vectors was insufficient. By explicitly constraining the magnitude of feature vectors, they achieved significantly better alignment and higher CLIP-Scores.

Furthermore, the model demonstrates emergent multilingualism. Despite primary training on English and Chinese, OmniDiffusion successfully processes prompts in languages like Japanese and Spanish, leveraging the pre-existing unified representation space of the underlying LLM.

Results and Performance

OmniDiffusion demonstrates superior performance across several benchmarks. In aesthetic evaluations on Flickr-30k, it achieved a score of 7.03, significantly outperforming Alt-Diffusion (5.47) and SD-XL. Human preference studies further validate these results, with users consistently favoring OmniDiffusion for its image quality and prompt adherence.

OmniDiffusion excels at generating high-aesthetic images with complex lighting and backgrounds.

The model's ability to handle long-context descriptions is a major breakthrough. Unlike CLIP-based models that truncate text after 77 tokens, OmniDiffusion maintains fidelity to complex, multi-sentence prompts, enabling the generation of intricate scenes with multiple subjects and specific spatial relationships.

The model demonstrates robust composition and detail in complex scenes involving multiple subjects.

Conclusion

OmniDiffusion provides a scalable and efficient path for upgrading T2I models with LLM-powered textual understanding. While challenges remain regarding human evaluation subjectivity and safety biases, the framework sets a new standard for multilingual and long-context image synthesis. Future work will focus on scaling the architecture and establishing more comprehensive multilingual benchmarks.

Summary updated

BibTeX
@inproceedings{tan2024empirical,
  author = {Zhiyu Tan and Mengping Yang and Luozheng Qin and Hao Yang and Ye Qian and Qiang Zhou and Cheng Zhang and Hao Li},
  title = {{An Empirical Study and Analysis of Text-to-Image Generation Using Large Language Model-Powered Textual Representation}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-72989-8\_27},
}
Figure