TL;DR
This paper introduces OCRGenBench, a comprehensive benchmark and evaluation metric (OCRGenScore) designed to rigorously assess the visual text synthesis capabilities of modern generative AI models. It highlights significant weaknesses in current models, particularly in handling complex text scenarios and non-English languages, arguing for the internalization of robust text generation as a core capability for general-purpose AI.
What Problem Does This Paper Address?
The paper tackles the growing challenge of evaluating generative AI models' ability to create and edit images containing text. While recent models have achieved impressive photorealism in natural image generation, their performance on text-heavy images (like documents, scene text, or artistic fonts) is often inconsistent and poorly understood. Existing benchmarks are too narrow in scope, focusing only on specific text types or tasks (like scene text generation or simple editing), and lack the complexity to truly stress-test model capabilities. This leaves a critical gap in understanding how well these powerful models can handle the nuanced demands of visual text synthesis, which is crucial for many real-world applications.
Methods
The authors introduce OCRGenBench, a novel benchmark that unifies three key areas of visual text synthesis:
- Text-to-Image (T2I) Generation: Creating images from textual descriptions.
- Text Editing: Modifying existing text within images based on instructions.
- OCR Image-to-Image Translation: Tasks like document dewarping, deblurring, and handwriting removal, which leverage OCR principles.
OCRGenBench covers five text categories (document, handwriting, scene text, artistic text, layout-rich text) and 33 distinct tasks. They also propose OCRGenScore, a unified metric that combines various task-specific evaluations (text accuracy, structural consistency, instruction following, visual quality) into a single score for holistic assessment. The benchmark includes 1,060 manually annotated samples designed to be challenging, featuring high text density, diverse aspect ratios, and bilingual content.
Datasets
The primary contribution is the introduction of OCRGenBench, a new benchmark dataset. For GT-required tasks where existing data was available, they collected from public datasets like those for document dewarping and scene text removal. However, for tasks lacking GT images (like text editing), they implemented a rigorous human-in-the-loop annotation process to create new, high-quality data.
Key Insights and Ideas
- Text Synthesis is a Foundational Skill: The paper strongly argues that robust text generation and editing capabilities should be internalized as core functionalities of general-purpose generative models, rather than being relegated to specialized solutions.
- Current Models Struggle with Complexity: Despite impressive general image generation, most models falter when faced with dense text, varied aspect ratios, complex fonts, or non-English languages.
- Unified Benchmarking is Crucial: A single, comprehensive benchmark like OCRGenBench is needed to provide a holistic view of a model's text synthesis abilities across diverse scenarios, unlike fragmented existing evaluations.
- Restorative Tasks are a Weakness: Models that excel at creative generation (T2I, editing) often perform poorly on restorative tasks (like document dewarping), indicating a trade-off between generative fidelity and precision.
- Multilingual Support is Lacking: Most models perform significantly better on English than Chinese, with some failing entirely on Chinese text generation, highlighting an English-centric bias in training data.
Key Results
- Overall Performance is Limited: Out of 19 evaluated models, only two (Nano Banana Pro and Flux.2-dev) achieved an OCRGenScore above 70, with most scoring below 60, indicating substantial room for improvement.
- Specialized Models Outperform Unified Ones in Specific Areas: While unified models show high potential, specialized models often achieve better performance on challenging tasks like historical documents and handwriting.
- Documents are the Toughest Category: Modern document tasks like dewarping and deblurring cause significant performance drops.
- English Dominance: Models generally perform better on English text than Chinese, with some models failing to generate coherent Chinese text.
- Fine-grained Encoding/Decoding is Lacking: Models struggle with generating small-scale text, often producing garbled but visually plausible output.
Limitations and Future Work
The authors acknowledge several limitations:
- Dataset Size: While comparable to existing benchmarks, the 1,060 samples might not cover all rare text styles or edge cases.
- OCR Engine Dependency: The Accuracy Rate metric relies on PP-OCRv5, meaning inherent errors or biases in the OCR engine can affect evaluation accuracy.
- Language Scope: Currently limited to English and Chinese; expansion to other scripts (Arabic, Korean, Hindi) is needed.
- Computational Constraints: Evaluation of exceptionally large models (>60B parameters) was not feasible.
Future work could focus on enhancing visual encoding/decoding for small-scale text, improving text localization for editing, developing more robust models for restorative tasks, and expanding language support. The authors also plan to continuously update the benchmark based on community feedback.
Summary updated