TL;DR
This paper introduces a novel framework for Scene Text Image Super-Resolution (STISR) using text-conditional diffusion models (DMs). By leveraging DMs' powerful generative capabilities and a new method for synthesizing realistic LR-HR text image pairs, the authors significantly improve STISR performance, especially when ground-truth text information is available.
What Problem Does This Paper Address?
Scene text recognition, a crucial technology for many applications, is severely hampered by blurred and low-resolution (LR) text images commonly found in real-world scenarios. Scene Text Image Super-Resolution (STISR) aims to enhance these LR images into high-resolution (HR) versions suitable for accurate recognition. However, training effective STISR models requires a substantial amount of paired LR-HR text images, which are expensive and difficult to collect. Existing methods for generating synthetic LR-HR pairs often use simplistic degradations that don't fully capture real-world complexities. This paper addresses the scarcity of high-quality training data for STISR and explores the potential of advanced generative models, specifically diffusion models, for this task.
Methods
The core of this work lies in adapting text-conditional diffusion models (DMs) for STISR. Unlike standard DMs used for text-to-image generation, STISR requires conditioning on the content of the text within an image, not a pre-written prompt. The authors propose a framework built upon three specialized text-conditional DMs:
- Synthesizer: Generates realistic text images from input text strings.
- Super-resolver: Takes a synthesized text image and its corresponding text as input to generate a high-resolution (HR) version.
- Degrader: Takes a synthesized text image and its corresponding text as input to generate a low-resolution (LR) version.
These three modules work together to create synthetic LR-HR paired text images. A key innovation is the use of ground-truth text priors (the actual text labels) during training, which significantly boosts performance compared to relying on text extracted by a separate recognition model. The authors also explore conditioning the DMs using cross-attention mechanisms, inserting them at the bottom of the UNet architecture to balance performance and computational cost.
Datasets
- TextZoom: This dataset is used for both training and evaluation of STISR methods. It contains real-world LR-HR paired text images.
- Scene Text Recognition Datasets: A collection of 11 real-labeled datasets (e.g., SVT, IC13, COCO) were preprocessed and used to train the Synthesizer module.
The paper introduces a method for generating augmented datasets by combining the original TextZoom dataset with synthesized LR-HR pairs.
Key Insights and Ideas
- Diffusion Models Excel at STISR: The paper demonstrates that text-conditional diffusion models, when properly adapted, can achieve state-of-the-art performance in STISR, outperforming existing methods.
- Ground-Truth Text is King: The most significant performance gains are observed when training with ground-truth text labels, highlighting the importance of accurate textual information for STISR. This suggests that even advanced generative models benefit from explicit textual guidance.
- Synthetic Data Generation is Crucial: The proposed framework for generating realistic LR-HR paired text images is vital for overcoming the data scarcity problem in STISR. The synthesized data, when combined with existing datasets, leads to substantial improvements.
- Balancing Degradation and Textual Fidelity: The authors acknowledge a trade-off where increased dataset size can sometimes decrease SSIM (a measure of image similarity) but improve recognition accuracy. This suggests that the synthesized images, while potentially less visually identical to ground truth HR images, are more effective for the downstream task of text recognition.
Key Results
- The proposed text-conditional DM achieved state-of-the-art results on the TextZoom evaluation dataset.
- Using ground-truth text priors led to a significant performance enhancement for the text-conditional DM, achieving an accuracy of 68.1% compared to 55.7% for the vanilla DDPM.
- Augmenting the TextZoom dataset with synthesized images led to a marked improvement in STISR performance for both their proposed model and the TATT method. For instance, the text-conditional DM trained on TZ+SynSTR achieved an average recognition accuracy of 55.7% on the TextZoom test set, compared to 55.0% for TZ only.
- The framework's synthesized paired images resulted in significantly higher recognition accuracy compared to those generated by simpler synthetic degradation methods like BSRGAN and Real-ESRGAN.
Limitations and Future Work
- Dependency on TextZoom: The Super-resolver and Degrader modules are trained on TextZoom. This means the framework may struggle with non-simple text images (e.g., highly artistic, curved, or rotated text) that are not present in TextZoom.
- Future Work: The authors suggest extending their work by combining it with image degradation methods that do not require paired images, which could help handle a wider variety of text image types. They also propose exploring alternative text prior generation methods and further investigating the trade-offs between image quality metrics (SSIM/PSNR) and recognition accuracy.
Summary updated