Hiren: Towards Higher Supervision Quality Forbetter Scene Text Image Super-Resolution

Minyi Zhao, Yi Xu, Bingjia Li, Jie Wang, Jihong Guan, Shuigeng ZhouSSRN 2024 · DOI

TL;DR

This paper introduces HiREN, a novel framework that tackles the problem of low-quality "ground truth" high-resolution (HR) images in scene text super-resolution (STISR). By first enhancing the quality of these HR images to create high-quality (HQ) versions and then using these HQ images for supervision, HiREN significantly boosts the performance of existing STISR methods.

What Problem Does This Paper Address?

Scene text recognition (STR) is crucial for understanding images, but its performance degrades significantly when dealing with low-resolution (LR) text images. Scene text image super-resolution (STISR) aims to improve LR images to aid recognition. Current STISR methods rely on high-resolution (HR) images for supervision during training. However, these HR images, often captured in real-world scenarios, suffer from various quality issues like blurring and low contrast due to uncontrollable factors like camera equipment and focus. This poor quality of HR "ground truth" can lead to erroneous supervision signals, ultimately limiting the effectiveness of STISR models. The paper addresses this critical gap by proposing a method to improve the quality of the HR supervision images themselves.

Methods

HiREN introduces a two-branch framework:

  1. LR Recovery Branch: This branch can be any existing STISR method. It takes the LR image and aims to produce a super-resolved (SR) image.
  2. HR Enhancement Branch: This is the core novelty. It takes the potentially low-quality HR image and enhances it to produce a high-quality (HQ) image. This branch consists of:
  3. Kernel Predictor: Estimates the degradation kernel (e.g., blur) applied to the original high-quality text image.
  4. Kernel-Guided Enhancement Network: Uses the predicted kernel to guide the enhancement process, aiming to restore details and improve clarity.
  5. Loss Functions: Employs a recognition loss (using a pre-trained scene text recognizer) to ensure the HQ image is recognizable, and a style loss (MSE) to preserve the original image's characteristics.

Additionally, a Quality Estimation (QE) Module is introduced. This module evaluates the quality of the generated HQ images and uses this assessment to weight the losses applied to the LR recovery branch. This helps to suppress the impact of any remaining erroneous supervision signals from imperfectly enhanced HQ images. HiREN can be used in an "online" (integrated into training) or "offline" (pre-generating HQ images) manner.

Datasets

  • TextZoom: A dataset of 21,740 LR-HR text image pairs, collected from real-world scenarios. It's divided into "easy," "medium," and "hard" settings based on camera focal length.
  • Regular STR Datasets: IC15-352, SVT, and SVTP were used to evaluate the generalization capabilities of HiREN when adapted to other datasets.

Key Insights and Ideas

  • The "Ground Truth" is Flawed: The paper convincingly argues that the quality of HR images used for STISR supervision is a major bottleneck, often being of insufficient quality itself.
  • Enhance Supervision First: Instead of solely focusing on improving LR images, HiREN prioritizes improving the quality of the supervision data (HR images) to make the entire STISR process more effective.
  • Kernel-Guided Enhancement: Leveraging a predicted degradation kernel is a clever way to guide the enhancement of HR images, allowing the network to better handle various degradations like blur.
  • Weak Supervision is Powerful: Using a pre-trained recognizer and text-level annotations as weak supervision for the HR enhancement branch is an effective strategy when direct pixel-level supervision for HQ images is unavailable.
  • Quality-Aware Loss Weighting: The QE module is crucial for mitigating the impact of imperfect HR enhancement, ensuring that only high-quality supervision signals contribute significantly to training.

Key Results

  • HiREN significantly boosts the recognition accuracy of various existing STISR methods on the TextZoom dataset, often by several percentage points. For example, with the TG method, accuracy increased from 49.6% to 51.1% on average.
  • The framework demonstrates universality, working well with different STISR methods and recognizers (CRNN, MORAN, ASTER, SEED, ABINet).
  • HiREN also shows improved performance on other STR datasets (IC15-352, SVT, SVTP).
  • The offline deployment of HiREN incurs no additional inference cost, and the online version adds negligible computational overhead.
  • Qualitative examples show HiREN effectively recovering blurry pixels and improving the clarity of text in challenging cases.

Limitations and Future Work

  • Noise Introduction: HiREN may introduce minor noise to HR images, though this is generally outweighed by the benefits.
  • Weak Supervision Dependency: The HR enhancement branch still requires some weak supervision (recognizer feedback and text annotations).
  • Future Directions:
    • Further improving the HR enhancement branch or addressing vocabulary reliance in STISR.
  • Exploring self-supervised or unsupervised settings for HiREN.
  • Extending the quality enhancement idea to develop new noisy learning algorithms for STISR.

Summary updated

BibTeX
@misc{zhao2024hiren,
  author = {Minyi Zhao and Yi Xu and Bingjia Li and Jie Wang and Jihong Guan and Shuigeng Zhou},
  title = {{Hiren: Towards Higher Supervision Quality Forbetter Scene Text Image Super-Resolution}},
  year = {2024},
  doi = {10.2139/ssrn.4822234},
  howpublished = {SSRN},
}
Figure