GlyphDraw: Learning to Draw Chinese Characters in Image Synthesis Models Coherently

Jian Ma, Mingjun Zhao, Chen Chen, Ruichen Wang, Di Niu, Haonan Lu, Xiaodong Lin

Abstract

Recent breakthroughs in the field of language-guided image generation have yielded impressive achievements, enabling the creation of high-quality and diverse images based on user instructions. Although the synthesis performance is fascinating, one significant limitation of current image generation models is their insufficient ability to generate text coherently within images, particularly for complex glyph structures like Chinese characters. To address this problem, we introduce GlyphDraw, a general learning framework aiming to endow image generation models with the capacity to generate images coherently embedded with text for any specific language. We first sophisticatedly design the image-text dataset's construction strategy, then build our model specifically on a diffusion-based image generator and carefully modify the network structure to allow the model to learn drawing language characters with the help of glyph and position information. Furthermore, we maintain the model's open-domain image synthesis capability by preventing catastrophic forgetting by using parameter-efficient fine-tuning techniques. Extensive qualitative and quantitative experiments demonstrate that our method not only produces accurate language characters as in prompts, but also seamlessly blends the generated text into the background.

Introduction: The "Spelling Problem" in Diffusion Models

While modern text-to-image models like Stable Diffusion and DALL-E 2 can create breathtaking landscapes and surreal art, they have a notorious "spelling problem." Ask for a sign that says "Love," and you might get "Lvoe" or a string of gibberish. This challenge is magnified tenfold for languages like Chinese, which rely on intricate, two-dimensional glyph structures rather than a simple linear alphabet.

A squirrel holding a sign with accurate Chinese characters, demonstrating the model's ability to handle complex glyphs.

Current models struggle with text for two main reasons. First, they often rely on sub-word tokenizers that "blur" the individual characters of a word. Second, they lack a dedicated mechanism to handle the precise spatial layout required for legible writing. While some models use massive Large Language Models (LLMs) to improve English spelling, this approach fails for non-Latin scripts. Chinese characters, for instance, are composed of thousands of unique glyphs with complex stroke orders. Training a model to "understand" these implicitly through text prompts alone is computationally expensive and often results in "character-blind" models that produce visual nonsense.

The GlyphDraw Framework

GlyphDraw, a new framework from researchers at the OPPO Research Institute and the University of Alberta, aims to bridge this gap. By providing models with explicit "visual cheat sheets" and spatial guidance, GlyphDraw enables the seamless rendering of accurate text—even complex Chinese characters—within high-quality, open-domain images. The framework introduces three key architectural innovations built on top of Stable Diffusion:

1. Spatial and Glyph Priors

Instead of just feeding the model a text prompt, GlyphDraw modifies the input to the UNet. It concatenates the standard image latent with two extra channels: a binary location mask (telling the model where to write) and a grayscale glyph image (showing the model what the characters look like). For complex scripts like Chinese, providing the model with a visual "glyph template" directly in the latent space acts as a powerful prior that simplifies the learning task.

2. Dual-Stream Conditioning and Fusion

The model uses a pretrained CLIP text encoder for the prompt and a CLIP image encoder for the glyphs. A specialized Fusion Module, consisting of a 6-layer Transformer, blends these embeddings into a single conditional feature used by the UNet’s cross-attention layers. This allows the model to decouple content from context, learning to isolate the text-rendering task from the background-generation task.

3. Parameter-Efficient Fine-Tuning (PEFT)

To prevent "catastrophic forgetting"—where a model learns to draw text but forgets how to draw everything else—the authors only update about 3% of the total parameters. Specifically, they train the new Fusion Module and only the $W_K$ and $W_V$ projection matrices in the cross-attention blocks. This confirms that fine-tuning just the cross-attention layers is sufficient to map new visual priors to the model's existing generative "vocabulary."

4. Inference-Time Mask Prediction

Since a user won't provide a location mask during normal use, the researchers added a small MLP-based Mask Prediction Module. During the first few steps of the diffusion process, this module estimates the text's position and shape, ensuring the final render respects perspective and geometry.

Datasets and Benchmarks

The authors curated a massive, high-quality dataset for training, including a Chinese dataset of 792k images containing 3.3M characters across 4.8k unique characters, and an English dataset of 1.9M images. They also introduced the DrawTextExt Benchmark, extending existing benchmarks to include both "Spelling" (standardized templates) and "Creative" (complex, natural prompts) categories for both languages.

Experimental Results

GlyphDraw significantly outperforms existing baselines in both quantitative and qualitative measures. It achieved 74% OCR accuracy for Chinese and 75% for English. In comparison, standard Stable Diffusion models score near 0%, and even specialized models like ControlNet struggle with the precise spatial structures of Chinese characters.

Examples of GlyphDraw generating coherent Chinese and English text across various contexts, from t-shirts to stone monuments.

Despite the heavy focus on text, the model maintained a low Fréchet Inception Distance (FID), meaning the generated images remain realistic and diverse. Qualitatively, the model doesn't just "paste" text on top; it integrates it into the scene, respecting lighting, texture, and perspective.

Limitations and Future Work

While a significant step forward, GlyphDraw faces a few remaining challenges. Accuracy begins to drop as character density increases, as the spatial complexity eventually overwhelms the current architecture. Additionally, the current version is optimized for a single text bounding box; future iterations will need to handle multiple, separate text areas and improve the rendering of long paragraphs or dense blocks of text.

Summary updated

Abstract

Recent breakthroughs in the field of language-guided image generation have yielded impressive achievements, enabling the creation of high-quality and diverse images based on user instructions.Although the synthesis performance is fascinating, one significant limitation of current image generation models is their insufficient ability to generate text coherently within images, particularly for complex glyph structures like Chinese characters. To address this problem, we introduce GlyphDraw, a general learning framework aiming to endow image generation models with the capacity to generate images coherently embedded with text for any specific language.We first sophisticatedly design the image-text dataset's construction strategy, then build our model specifically on a diffusion-based image generator and carefully modify the network structure to allow the model to learn drawing language characters with the help of glyph and position information.Furthermore, we maintain the model's open-domain image synthesis capability by preventing catastrophic forgetting by using parameter-efficient fine-tuning techniques.Extensive qualitative and quantitative experiments demonstrate that our method not only produces accurate language characters as in prompts, but also seamlessly blends the generated text into the background.Please refer to our \href{https://1073521013.github.io/glyph-draw.github.io/}{project page}. \end{abstract}
BibTeX
@misc{ma2023glyphdraw,
  author = {Jian Ma and Mingjun Zhao and Chen Chen and Ruichen Wang and Di Niu and Haonan Lu and Xiaodong Lin},
  title = {{GlyphDraw: Learning to Draw Chinese Characters in Image Synthesis Models Coherently}},
  year = {2023},
  eprint = {2303.17870},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.17870},
}
Figure