STRICT: Stress-Test of Rendering Image Containing Text

Xinyu WangEMNLP 2025 · OpenReview

TL;DR

This paper introduces STRICT, a new benchmark designed to rigorously evaluate how well text-to-image models can render text within images, especially for longer text sequences and across different languages. The authors find that even state-of-the-art models still struggle with generating consistent and instruction-aligned text, highlighting a key challenge in multimodal AI.

What Problem Does This Paper Address?

Text-to-image models have become incredibly powerful, capable of generating stunningly realistic images from simple text prompts. However, a persistent Achilles' heel is their ability to render legible and accurate text within those images. Think of trying to generate a realistic image of a street sign, a product label, or a book cover – current models often produce garbled, misspelled, or nonsensical text. This is largely attributed to the "locality bias" in diffusion models, meaning they tend to focus on local details rather than maintaining long-range coherence, which is crucial for structured text. This paper aims to systematically stress-test this capability, identifying where models fail and why.

Methods

The core of this work is the STRICT benchmark. The authors developed a pipeline that involves:

  1. Text Extraction: Sourcing text snippets from Wikipedia in English, French, and Chinese, varying in length from short phrases to long paragraphs.
  2. Prompt Generation: Creating standardized prompts for text-to-image models, instructing them to generate a "plain Word document with black text on a white background, without decorative elements," embedding the extracted text.
  3. Image Generation: Feeding these prompts to various text-to-image models.
  4. OCR Text Extraction: Using the Tesseract OCR engine to extract the text rendered in the generated images.
  5. Evaluation: Comparing the OCR-extracted text against the original ground truth text using several quantitative metrics.

The key novelties here are the benchmark itself and the evaluation metrics:

  • Multilingual and Varied Lengths: STRICT covers English, French, and Chinese, and crucially, tests text lengths from 5 to 5000 characters, directly probing long-range dependency issues.
  • Quantitative Metrics:
    • Normalized Edit Distance (NED): Measures character-level dissimilarity.
  • Character Error Rate (CER) & Word Error Rate (WER): Standard metrics from speech recognition, adapted for text rendering.
  • Ratio of Not Following Instructions (RNFI): Quantifies how often models fail to render any text or render significantly less text than requested, indicating a complete failure to follow instructions.

Datasets

The primary dataset for generating text samples is Wikipedia. For evaluation, the authors use the generated image-text pairs. The benchmark itself, STRICT, is a newly introduced resource.

Key Insights and Ideas

  • Longer Text is the Enemy: The most significant finding is that model performance degrades sharply as text length increases beyond approximately 200-800 characters. This suggests a fundamental limitation in how current models handle long-range dependencies.
  • CLIP's Token Limit Might Be a Bottleneck: The authors hypothesize that the ~77-token limit of CLIP text encoders, commonly used in diffusion models, might restrict their ability to process and condition on longer instructions.
  • Multilingual Challenges: While English generally performs best, Chinese shows significantly lower accuracy, suggesting a need for more diverse multilingual training data rather than inherent difficulties with the script itself.
  • Instruction Following is Fragile: Models often fail to follow instructions for longer texts, sometimes generating entirely unrelated images or omitting text altogether. This is captured by the RNFI metric.
  • Even Top Models Struggle: While GPT-4o and Gemini 2.0 show superior performance compared to others, they still exhibit limitations, especially with longer texts and in non-English languages.

Key Results

  • GPT-4o and Gemini 2.0 Lead: These models consistently outperform others across all languages and text lengths in terms of character and word accuracy.
  • Performance Drop-off: For most models, accuracy (measured by NED, CER, WER) drops significantly as text length increases past a few hundred characters. For example, Figure 3 shows a steep rise in NED for most models beyond 200 characters.
  • RNFI Highlights Catastrophic Failures: Figure 4 shows that models like FLUX 1.1 pro and Gemini 2.0 have a higher proportion of instruction-following failures (RNFI < 0.01) as text length increases, indicating they sometimes completely ignore the text rendering instruction.
  • Chinese Language Performance: Figure 3 and Table 1 show consistently higher NED scores for Chinese across most models, indicating lower text rendering accuracy compared to English and French.

Limitations and Future Work

  • OCR Limitations: The authors acknowledge that the Tesseract OCR engine, while powerful, is not perfect and can still make errors, which can affect the evaluation.
  • Benchmark Overfitting Risk: There's a concern that future models might be specifically fine-tuned to perform well on STRICT, potentially inflating leaderboard scores without true generalization.
  • Future Work: The paper calls for architectural improvements to better handle long-range dependencies, more robust multilingual training data, and further research into structured text generation. The authors hope STRICT will guide future development in this area.

Summary updated

Abstract

While diffusion models have revolutionized text-to-image generation with their ability to synthesize realistic and diverse scenes, they continue to struggle with generating consistent and legible text within images. This shortcoming is commonly attributed to the locality bias inherent in diffusion-based generation, which limits their capacity to model long-range spatial dependencies. In this paper, we introduce \textbf{STRICT}, a benchmark designed to systematically stress-test the ability of diffusion models to render coherent and instruction-aligned text in images. Our benchmark evaluates models across multiple dimensions: (1) the maximum length of readable text that can be generated and (2) the correctness and legibility of the generated text. We assess several state-of-the-art models, including proprietary and open-source variants, and reveal persistent limitations in long-range consistency and instruction-following capabilities. Our findings provide insights into architectural bottlenecks and motivate future research directions in multimodal generative modeling.
BibTeX
@inproceedings{wang2025strict,
  author = {Xinyu Wang},
  title = {{STRICT: Stress-Test of Rendering Image Containing Text}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=Qe7je80qiy},
}
Figure