Image Super-Resolution with Text Handling Via Generative Adversairal Network

Jaepil Yu, Hyeongseok Son, Sunghyun Cho, Seungyong LeeKIISE 2018 · DOI

TL;DR

This paper tackles the challenge of super-resolving images containing text, where traditional methods struggle to render sharp and legible characters. By adapting Generative Adversarial Networks (GANs) to specifically learn text characteristics alongside general image features, the authors achieve significantly improved text clarity in super-resolved images.

What Problem Does This Paper Address?

Super-resolution (SR) aims to reconstruct a high-resolution (HR) image from a low-resolution (LR) input. While deep learning, particularly Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs), has dramatically advanced SR performance, existing methods primarily focus on learning the general statistical properties of natural images. This leads to a significant degradation in the quality of text regions within images. Text has distinct characteristics, such as sharp edges and uniform internal colors, which differ from natural textures. Standard SR models often blur these sharp transitions and fail to reconstruct legible text, rendering them unsuitable for applications where text clarity is crucial, like document analysis or signage enhancement.

Methods

The core of this paper's contribution lies in enhancing a state-of-the-art GAN-based SR framework (specifically, SRGAN by Ledig et al.) to handle text regions more effectively. The key innovation is the introduction of a dual-discriminator architecture.

  1. Dual Discriminators: Instead of a single discriminator that learns to distinguish between generated and real images, this approach uses two discriminators:
  2. Discriminator 1 (D_n): Learns to distinguish general image features, similar to standard SRGAN.
  3. Discriminator 2 (D_t): Specifically trained to distinguish features of text regions.
  4. Separate Training Stages: The training process is divided into two stages, interleaved within each batch update:
  5. Stage 1 (General Image Training): The network is trained on standard natural images to improve general SR quality.
  6. Stage 2 (Text-Specific Training): To train the text discriminator, the authors augment their dataset by adding synthetic text to natural images. They also use a corresponding mask (M_t) that highlights the text regions. This mask is dilated to account for potential blending of text and background pixels in the generated output. This stage allows the network to learn the specific characteristics of text.
  7. Loss Function: The paper adopts the perceptual loss function from SRGAN, which utilizes a pre-trained VGG network to capture high-level image features, in addition to the adversarial loss. This helps in generating more photorealistic details.

Datasets

  • Training: The authors used the DIV2K dataset, a common benchmark for image super-resolution. They augmented this dataset by programmatically adding synthetic text. The text was generated using three serif and three sans-serif font types, with random font sizes and limited to white and black colors.
  • Evaluation: The 2015 IDCAR Focused Scene Text dataset was used for testing. This dataset comprises images where text is a primary subject.

Key Insights and Ideas

  • Text is Different: The fundamental insight is that text regions possess distinct statistical properties compared to natural image textures, necessitating specialized handling in SR.
  • Dual Discriminators for Specialization: Employing separate discriminators for general image features and text features allows the GAN to learn and enforce distinct quality criteria for each.
  • Augmentation is Key: Synthetically adding text to existing datasets is an effective way to create training data for text-specific SR without requiring entirely new, large-scale text-centric SR datasets.
  • Masking for Precision: Using a dilated mask for text regions during training helps the text discriminator focus on the relevant pixels and their immediate surroundings, improving the accuracy of text reconstruction.

Key Results

The paper demonstrates qualitative improvements in text super-resolution.

  • Qualitative: The proposed method produces sharper and more legible text compared to the baseline SRGAN. The blurring artifacts around text characters observed in the baseline are significantly reduced.
  • Quantitative: While specific PSNR values are not explicitly detailed for the text regions in the abstract, the paper states that the PSNR of the proposed method is higher than the baseline for text-containing images. Figure 4 shows a visual comparison where the proposed method (b) clearly renders text more crisply than the SRGAN result (a). Figure 5 shows a failure case with very bold text, indicating room for improvement.

Limitations and Future Work

  • Bold Text: The authors acknowledge that the current method struggles with very bold text (as shown in Figure 5). The training data primarily used text with a certain stroke width, and thicker fonts are not reconstructed as clearly.
  • Font Variety: While multiple font types were used, the system is limited by the specific fonts and colors included in the training data.
  • Future Work: The authors suggest future research directions:
  • Automatic Text Detection: Instead of relying on pre-generated text masks, a more robust approach would involve automatically detecting text regions within an image and applying specialized processing.
  • Broader Font and Style Handling: Expanding the training data to include a wider variety of fonts, styles, and colors would improve generalization.
  • Real-world Text Complexity: Addressing more complex scenarios like curved text, text with shadows, or text in highly textured backgrounds.

Summary updated

Abstract

본 논문은 딥러닝 모델 중 하나인 generative adversarial network (GAN)을 사용하여 글자 특성이 고려된 초해상도 영상 복원 방법을 제시한다. 기존의 초해상도 영상 복원 방법은 일반적인 영상에 대한 특징들을 주로 학습하기 때문에 글자 영역의 복원에 대해서는 부족한 성능을 보인다. 글자 영상이 가지고 있는 특징은 일반 영상의 특징과 구분되므로 이를 별도로 처리하기 위한 과정이 필요하다. 따라서 본 논문에서는 기존의 데이터셋에 글자를 추가하고, 일반 영상에 대한 학습과 글자 영상에 대한 학습을 나누어 수행하여 글자 영역에 대해 개선된 초해상도 복원 방법을 제시한다. 실험 결과를 통해 본 논문에서 제안한 알고리즘이 글자가 포함된 영상에 대하여 복원의 품질이 향상되는 것을 보인다.
BibTeX
@inproceedings{yu2018image,
  author = {Jaepil Yu and Hyeongseok Son and Sunghyun Cho and Seungyong Lee},
  title = {{Image Super-Resolution with Text Handling Via Generative Adversairal Network}},
  booktitle = {KIISE},
  year = {2018},
  doi = {10.5626/ktcp.2018.24.8.405},
}
Figure