Aesthetics is Cheap, Show me the Text: An Empirical Evaluation of State-of-the-Art Generative Models for OCR

Peirong Zhang, Haowei Xu, Jiaxin Zhang, Guitao Xu, Xuhan Zheng, Zhenhua Yang, Junle Liu, Yuyi Zhang, Lianwen Jin

TL;DR

This paper introduces OCRGenBench, a comprehensive benchmark and evaluation metric (OCRGenScore) designed to rigorously assess the visual text synthesis capabilities of modern generative AI models. It highlights significant weaknesses in current models, particularly in handling complex text scenarios and non-English languages, arguing for the internalization of robust text generation as a core capability for general-purpose AI.

What Problem Does This Paper Address?

The paper tackles the growing challenge of evaluating generative AI models' ability to create and edit images containing text. While recent models have achieved impressive photorealism in natural image generation, their performance on text-heavy images (like documents, scene text, or artistic fonts) is often inconsistent and poorly understood. Existing benchmarks are too narrow in scope, focusing only on specific text types or tasks (like scene text generation or simple editing), and lack the complexity to truly stress-test model capabilities. This leaves a critical gap in understanding how well these powerful models can handle the nuanced demands of visual text synthesis, which is crucial for many real-world applications.

Methods

The authors introduce OCRGenBench, a novel benchmark that unifies three key areas of visual text synthesis:

  1. Text-to-Image (T2I) Generation: Creating images from textual descriptions.
  2. Text Editing: Modifying existing text within images based on instructions.
  3. OCR Image-to-Image Translation: Tasks like document dewarping, deblurring, and handwriting removal, which leverage OCR principles.

OCRGenBench covers five text categories (document, handwriting, scene text, artistic text, layout-rich text) and 33 distinct tasks. They also propose OCRGenScore, a unified metric that combines various task-specific evaluations (text accuracy, structural consistency, instruction following, visual quality) into a single score for holistic assessment. The benchmark includes 1,060 manually annotated samples designed to be challenging, featuring high text density, diverse aspect ratios, and bilingual content.

Datasets

The primary contribution is the introduction of OCRGenBench, a new benchmark dataset. For GT-required tasks where existing data was available, they collected from public datasets like those for document dewarping and scene text removal. However, for tasks lacking GT images (like text editing), they implemented a rigorous human-in-the-loop annotation process to create new, high-quality data.

Key Insights and Ideas

  • Text Synthesis is a Foundational Skill: The paper strongly argues that robust text generation and editing capabilities should be internalized as core functionalities of general-purpose generative models, rather than being relegated to specialized solutions.
  • Current Models Struggle with Complexity: Despite impressive general image generation, most models falter when faced with dense text, varied aspect ratios, complex fonts, or non-English languages.
  • Unified Benchmarking is Crucial: A single, comprehensive benchmark like OCRGenBench is needed to provide a holistic view of a model's text synthesis abilities across diverse scenarios, unlike fragmented existing evaluations.
  • Restorative Tasks are a Weakness: Models that excel at creative generation (T2I, editing) often perform poorly on restorative tasks (like document dewarping), indicating a trade-off between generative fidelity and precision.
  • Multilingual Support is Lacking: Most models perform significantly better on English than Chinese, with some failing entirely on Chinese text generation, highlighting an English-centric bias in training data.

Key Results

  • Overall Performance is Limited: Out of 19 evaluated models, only two (Nano Banana Pro and Flux.2-dev) achieved an OCRGenScore above 70, with most scoring below 60, indicating substantial room for improvement.
  • Specialized Models Outperform Unified Ones in Specific Areas: While unified models show high potential, specialized models often achieve better performance on challenging tasks like historical documents and handwriting.
  • Documents are the Toughest Category: Modern document tasks like dewarping and deblurring cause significant performance drops.
  • English Dominance: Models generally perform better on English text than Chinese, with some models failing to generate coherent Chinese text.
  • Fine-grained Encoding/Decoding is Lacking: Models struggle with generating small-scale text, often producing garbled but visually plausible output.

Limitations and Future Work

The authors acknowledge several limitations:

  • Dataset Size: While comparable to existing benchmarks, the 1,060 samples might not cover all rare text styles or edge cases.
  • OCR Engine Dependency: The Accuracy Rate metric relies on PP-OCRv5, meaning inherent errors or biases in the OCR engine can affect evaluation accuracy.
  • Language Scope: Currently limited to English and Chinese; expansion to other scripts (Arabic, Korean, Hindi) is needed.
  • Computational Constraints: Evaluation of exceptionally large models (>60B parameters) was not feasible.

Future work could focus on enhancing visual encoding/decoding for small-scale text, improving text localization for editing, developing more robust models for restorative tasks, and expanding language support. The authors also plan to continuously update the benchmark based on community feedback.

Summary updated

Abstract

Text image is a unique and crucial information medium that integrates visual aesthetics and linguistic semantics in modern e-society. Due to their subtlety and complexity, the generation of text images represents a challenging and evolving frontier in the image generation field. The recent surge of specialized image generators (\emph{e.g.}, Flux-series) and unified generative models (\emph{e.g.}, GPT-4o), which demonstrate exceptional fidelity, raises a natural question: can they master the intricacies of text image generation and editing? Motivated by this, we assess current state-of-the-art generative models' capabilities in terms of text image generation and editing. We incorporate various typical optical character recognition (OCR) tasks into our evaluation and broaden the concept of text-based generation tasks into OCR generative tasks. We select 33 representative tasks and categorize them into five categories: document, handwritten text, scene text, artistic text, and complex \& layout-rich text. For comprehensive evaluation, we examine six models across both closed-source and open-source domains, using tailored, high-quality image inputs and prompts. Through this evaluation, we draw crucial observations and identify the weaknesses of current generative models for OCR tasks. We argue that photorealistic text image generation and editing should be internalized as foundational skills into general-domain generative models, rather than being delegated to specialized solutions, and we hope this empirical analysis can provide valuable insights for the community to achieve this goal. This evaluation is online and will be continuously updated at our GitHub repository.
BibTeX
@misc{zhang2025aesthetics,
  author = {Peirong Zhang and Haowei Xu and Jiaxin Zhang and Guitao Xu and Xuhan Zheng and Zhenhua Yang and Junle Liu and Yuyi Zhang and Lianwen Jin},
  title = {{Aesthetics is Cheap, Show me the Text: An Empirical Evaluation of State-of-the-Art Generative Models for OCR}},
  year = {2025},
  eprint = {2507.15085},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.15085},
}
Figure