TL;DR
This paper tackles the challenge of improving scene text image super-resolution (STISR) by addressing two key issues: the instability of explicit text priors and the overconfidence problem in joint training. They introduce Non-Categorical Prior (NCAP), which uses implicit representations from a recognizer's penultimate layer, and a novel loss function that combines hard and soft labels to mitigate overconfidence, leading to significant performance gains.
What Problem Does This Paper Address?
Recognizing text in images (scene text recognition) is crucial, but low-resolution images often hinder this process. Scene Text Image Super-Resolution (STISR) aims to enhance these images. While using "text priors" (information extracted from pre-trained text recognizers) has boosted STISR performance, two major problems persist:
- Unstable Text Priors: Explicit text priors, like predicted character sequences, can be incorrect. If the prior is wrong, it can negatively impact the STISR task, leading to worse results.
- Overconfidence in Joint Training: To bridge the gap between low-resolution (LR) and high-resolution (HR) images, STISR models are often trained jointly with text recognizers. However, this joint training, especially when using ground-truth "hard" labels, can lead to an "overconfidence" issue where the model becomes too certain about its predictions, even when they are wrong, ultimately degrading performance.
Methods
The paper proposes two main innovations:
- Non-Categorical Prior (NCAP): Instead of relying on explicit, potentially erroneous text logits from a pre-trained recognizer, NCAP utilizes the penultimate layer representations of the recognizer. These representations are considered "category-free" and richer in information. They are then processed by small, learnable "adapters" to create a more stable and informative prior. This approach requires minimal additional parameters (around 0.3%).
- Mixed Hard and Soft Labels Loss Function: To combat the overconfidence issue arising from joint training, the authors propose a novel loss function that linearly combines ground-truth "hard" labels with "soft" labels. Soft labels, often derived from a teacher model or through techniques like label smoothing, provide a more nuanced probability distribution, helping to regularize the training and prevent overconfidence. The paper provides theoretical analysis of Kullback-Leibler (KL) divergence and cross-entropy losses to justify this approach.
Datasets
The paper uses the following datasets:
- TextZoom [33]: A widely used dataset for STISR, which creates LR-HR image pairs from existing super-resolution datasets. It includes subsets for "easy," "medium," and "hard" degradation levels.
- Scene Text Recognition (STR) Datasets: To evaluate generalization, the authors use IIIT5K [23], ICDAR2015 [16], SVT [32], and SVTP [25]. These datasets are used with manually degraded images.
Key Insights and Ideas
- Implicit Priors are More Robust: Explicit text priors, while seemingly informative, are fragile. Using richer, implicit representations from earlier layers of a recognizer (NCAP) offers a more stable and less error-prone prior for STISR.
- The Danger of Overconfidence: Jointly training STISR and recognition models is beneficial for bridging the domain gap, but it can lead to models that are overly confident in their incorrect predictions, especially when trained with hard labels.
- The Best of Both Worlds: A simple linear combination of hard and soft labels effectively balances the need for precise supervision (from hard labels) with the regularization benefits of soft labels, mitigating overconfidence without sacrificing performance.
- Generalizability: The proposed NCAP and the mixed loss function are designed to be compatible with existing TP-guided STISR networks, suggesting broad applicability.
Key Results
- TextZoom Performance: The proposed method achieves a 3.5% improvement on the TextZoom dataset.
- Generalization: The method significantly enhances generalization performance by 14.8% across four text recognition datasets (IIIT5K, ICDAR2015, SVT, SVTP) under degradation.
- Module Effectiveness: Ablation studies show that both NCAP and the mixed loss function contribute to performance gains. Applying both leads to the best results.
- Computational Efficiency: NCAP adds only a minimal overhead (around 0.3%) in terms of trainable parameters.
- Visual Quality vs. Readability: While PSNR and SSIM scores might slightly decrease, the SR images produced by their method show improved readability, indicating a successful trade-off for the primary goal of text recognition.
Limitations and Future Work
- Visual Metrics: The paper acknowledges that while readability and accuracy improve, traditional visual quality metrics like PSNR and SSIM might slightly decrease. This is attributed to the focus on noise reduction for text clarity, which can sometimes conflict with overall image fidelity.
- Future Work: The paper suggests that their method is compatible with all STISR networks using explicit priors, implying potential for integration with a wider range of models. Further exploration into optimizing the balance between hard and soft labels and investigating more advanced adapter architectures for NCAP could be future directions.
Summary updated