Gradient-Based Graph Attention for Scene Text Image Super-resolution

Xiangyuan Zhu, Kehua Guo, Hui Fang, Rui Ding, Zheng Wu, Gerald SchaeferAAAI 2023 · DOI

TL;DR

This paper introduces a novel "Two-Stage Attention Scheme" (TSAN) for super-resolving low-resolution scene text images. By explicitly modeling the inherent structure and repetitive patterns of text using gradient-based graph attention and group-wise attention, TSAN significantly improves the clarity of reconstructed text, leading to better downstream recognition accuracy and outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

Scene text image super-resolution (STISR) aims to enhance the quality of low-resolution images containing text, which is crucial for improving the performance of computer vision tasks like text recognition. This is particularly challenging for images captured "in the wild" due to varying lighting, angles, and image quality. While existing methods have made progress, they often overlook the unique, well-structured, and repetitive nature of text itself. This paper addresses the gap of effectively leveraging text's intrinsic structural priors to guide the super-resolution process, thereby achieving more accurate and visually appealing text reconstruction.

Methods

The core of the proposed method is the Two-Stage Attention Scheme (TSAN), which enhances feature representations of low-resolution (LR) text images.

  1. Initial Feature Extraction: A convolutional layer processes the input LR image.
  2. Group-Wise Attention Module (GWAM): This module leverages the observation that similar patterns in natural images can be exploited for reconstruction. It uses locality-sensitive hashing to group features and efficiently identify similar local image patches, enhancing generic feature representations.
  3. Gradient-Based Graph Attention Module (GGAM): This novel component focuses on refining boundary features by explicitly modeling text structure. It computes a gradient map of the LR image, uses it to construct a graph where nodes represent image patches, and then applies graph attention. Crucially, it incorporates a mechanism to filter out insignificant nodes based on similarity, ensuring that only relevant structural information contributes to the reconstruction. This is the first work to apply graph attention for embedding text structure in STISR.
  4. Cascaded Channel Attention Module (CCAM): This module further enhances features by exploiting correlations across channels, similar to existing attention mechanisms.
  5. Feature Fusion and Reconstruction: The outputs from CCAM and GGAM are concatenated and refined. Finally, this refined feature map is combined with the GWAM output and upsampled to produce the super-resolved (SR) image.

The loss function comprises a standard reconstruction loss (Lrec) and a character perceptual loss (Lcha) derived from a pre-trained text recognition model to ensure perceptual quality.

Datasets

  • TextZoom: This is the primary dataset used for training and evaluation. It's described as the first realistic STISR dataset for images "in the wild." The paper mentions it contains 21,740 image pairs with LR and HR images of sizes 16x64 and 32x128, respectively. The dataset is divided into training (17,367 pairs) and testing (4,373 pairs), further categorized into "easy," "medium," and "hard" based on focal lengths.
  • RealSR and SR-RAW: These datasets are mentioned as additional sources for training, contributing to the total of 21,740 image pairs.

Key Insights and Ideas

  • Text Structure as a Prior: The fundamental insight is that text isn't just arbitrary pixels; it has inherent, exploitable structure (strokes, layouts, repetition). Treating this structure as a prior knowledge source is key to improving STISR.
  • Gradient-Informed Graph Attention: Using gradient maps to construct graphs for attention is a novel way to explicitly capture and leverage text contour information, which is vital for clear text reconstruction.
  • Complementary Attention Modules: The paper argues that GWAM (for general feature enhancement) and GGAM (for specific structural/boundary refinement) are complementary, with GGAM having the most significant impact on performance.
  • Importance of Patch Size in GGAM: The choice of patch size for graph construction in GGAM is critical, with 7x7 found to yield the best results, suggesting a sweet spot for capturing local stroke-level patterns.

Key Results

  • State-of-the-Art Performance: The proposed TSAN model achieves state-of-the-art (SOTA) performance on the TextZoom dataset, outperforming 19 other methods (including generic SR and specialized STISR techniques).
  • Recognition Accuracy: Using ASTER as the evaluation metric, TSAN achieves accuracies of 53.0% (CRNN), 62.7% (MORAN), and 64.1% (ASTER) on the overall test set, surpassing the previous best method (TATT) by a notable margin. It also leads in most sub-groups.
  • Image Quality: The paper reports SOTA SR image quality in terms of PSNR and SSIM, as shown in Table 2.
  • Ablation Studies: Removing GWAM leads to a 0.6% drop, CCAM improves accuracy by 0.4%, and GGAM provides the most significant gain of 1.6%. This highlights the critical role of GGAM.

Limitations and Future Work

  • Inference Efficiency: The authors acknowledge that future work will focus on improving inference efficiency, suggesting that the current model might be computationally intensive.
  • Generalizability: While tested on TextZoom, the paper doesn't extensively discuss performance on other "in the wild" datasets or different types of text.
  • Parameter Tuning: The ablation studies highlight the importance of parameters like the number of groups (M) in GWAM and patch size in GGAM, implying that careful tuning might be necessary for optimal performance.

Summary updated

BibTeX
@inproceedings{zhu2023gradient,
  author = {Xiangyuan Zhu and Kehua Guo and Hui Fang and Rui Ding and Zheng Wu and Gerald Schaefer},
  title = {{Gradient-Based Graph Attention for Scene Text Image Super-resolution}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i3.25499},
}
Figure