Challenges of Deep Learning-based Text Detection in the Wild

Zobeir Raisi, Mohamed A. Naiel, Paul Fieguth, Steven Wardell, John ZelekJournal of Computational Vision and Imaging Systems 2021 · DOI

Abstract

The reported accuracy of recent state-of-the-art text detection methods, mostly deep learning approaches, is in the order of 80% to 90% on standard benchmark datasets. These methods have relaxed some of the restrictions of structured text and environment (i.e., "in the wild") which are usually required for classical OCR to properly function. Even with this relaxation, there are still circumstances where these state-of-the-art methods fail. Several remaining challenges in wild images, like in-plane-rotation, illumination reflection, partial occlusion, complex font styles, and perspective distortion, cause exciting methods to perform poorly. In order to evaluate current approaches in a formal way, we standardize the datasets and metrics for comparison which had made comparison between these methods difficult in the past. We use three benchmark datasets for our evaluations: ICDAR13, ICDAR15, and COCO-Text V2.0. The objective of the paper is to quantify the current shortcomings and to identify the challenges for future text detection research.

Introduction

Deep learning has revolutionized scene text detection, pushing performance on standard benchmarks to impressive heights. However, these high scores often mask a significant fragility when models encounter the unconstrained reality of "text in the wild." Unlike traditional OCR, which operates on structured documents, wild text detection must contend with perspective distortion, motion blur, and complex backgrounds. This paper investigates the gap between benchmark performance and real-world utility by standardizing evaluation across four state-of-the-art detectors: EAST, Pixellink, CRAFT, and PMTD.

Evaluation Framework

The study categorizes and evaluates three main architectural approaches:

  • Regression-based (RB): Models like EAST predict bounding boxes directly. They are efficient but struggle with irregular shapes.
  • Segmentation-based (SB): Models like Pixellink perform pixel-level classification, offering flexibility for varied shapes but requiring complex post-processing.
  • Hybrid & Character-level: CRAFT detects individual characters to handle artistic fonts, while PMTD (Pyramid Mask Text Detector) uses soft pyramid labels for robust mask generation.

These models were tested against three datasets of increasing difficulty: ICDAR 2013 (horizontal text), ICDAR 2015 (multi-oriented text), and COCO-Text V2.0 (small, blurry text in complex backgrounds).

Quantitative Analysis: The Precision Cliff

A critical finding is the "precision cliff." As the Intersection over Union (IoU) threshold increases—demanding more precise localization—performance for all models drops sharply. This drop is particularly pronounced in the COCO-Text dataset, where even the best model, PMTD, falls from nearly 90% on ICDAR datasets to just 60.40%. Interestingly, simpler models like EAST can outperform more complex ones like PMTD at very high precision thresholds on specific datasets (e.g., ICDAR13) due to better scale-invariant properties.

H-mean performance across varying IoU thresholds for ICDAR13, ICDAR15, and COCO-Text.

Qualitative Challenges in the Wild

Beyond raw metrics, qualitative analysis reveals specific failure modes. Models struggle with in-plane rotation (IPR), illumination variations (IV), and perspective distortion (PD). Character-based models like CRAFT excel at artistic fonts but often fail at scale consistency. The generalization gap is also significant; for instance, Pixellink saw a 20% performance drop when moving from ICDAR15 to the similar ICDAR13, suggesting overfitting to specific dataset distributions.

Qualitative comparison of text detection models across various environmental challenges.

Conclusion

While PMTD emerges as the most robust overall performer, the significant performance drop on the COCO-Text dataset highlights the limitations of current state-of-the-art methods. Future progress requires moving beyond simple bounding box metrics toward more descriptive annotations that categorize environmental challenges—such as rotation or occlusion—enabling models to better generalize across diverse real-world scenarios.

Summary updated

Abstract

The reported accuracy of recent state-of-the-art text detection methods, mostly deep learning approaches, is in the order of 80% to 90% on standard benchmark datasets. These methods have relaxed some of the restrictions of structured text and environment (i.e., "in the wild") which are usually required for classical OCR to properly function. Even with this relaxation, there are still circumstances where these state-of-the-art methods fail. Several remaining challenges in wild images, like in-plane-rotation, illumination reflection, partial occlusion, complex font styles, and perspective distortion, cause exciting methods to perform poorly. In order to evaluate current approaches in a formal way, we standardize the datasets and metrics for comparison which had made comparison between these methods difficult in the past. We use three benchmark datasets for our evaluations: ICDAR13, ICDAR15, and COCO-Text V2.0. The objective of the paper is to quantify the current shortcomings and to identify the challenges for future text detection research.
BibTeX
@article{raisi2021challenges,
  author = {Zobeir Raisi and Mohamed A. Naiel and Paul Fieguth and Steven Wardell and John Zelek},
  title = {{Challenges of Deep Learning-based Text Detection in the Wild}},
  journal = {Journal of Computational Vision and Imaging Systems},
  year = {2021},
  doi = {10.15353/jcvis.v6i1.3543},
}
Figure