ARTS: Eliminating Inconsistency between Text Detection and Recognition with Auto-Rectification Text Spotter

Humen Zhong, Jun Tang, Wenhai Wang, Zhibo Yang, Cong Yao, Tong LuarXiv 2021 · OpenReview

TL;DR

This paper tackles the critical "inconsistency problem" in end-to-end text spotting, where text detection and recognition modules don't align well. They introduce the Auto-Rectification Text Spotter (ARTS) with a novel differentiable module (ARM) that allows recognition losses to directly improve text detection, leading to state-of-the-art performance and efficiency on arbitrarily-shaped text spotting tasks.

What Problem Does This Paper Address?

Text spotting, the task of both detecting and recognizing text in images, has seen significant progress. However, a persistent challenge is the disconnect between the text detection and text recognition components. This inconsistency manifests in two key ways:

  1. Feature Inconsistency: Text recognition features are often extracted using ground-truth bounding boxes during training but predicted bounding boxes during testing. This discrepancy leads to different feature distributions, making the recognition model less effective at inference time.
  2. Optimization Target Mismatch: Text detectors are typically trained to maximize Intersection over Union (IoU) with ground truth. However, a detection box with a high IoU isn't always ideal for recognition. Sometimes, a slightly less precise box might contain more accurate character information, leading to better recognition. This mismatch means the detector might produce boxes that are "good enough" for detection metrics but suboptimal for the subsequent recognition task.

These issues collectively hinder the performance of end-to-end text spotting systems, especially for challenging cases like arbitrarily-shaped text.

Methods

The core innovation of ARTS is its Auto-Rectification Module (ARM) and a new training strategy.

  • Auto-Rectification Module (ARM): This is a differentiable module designed to bridge the gap between detection and recognition. It takes predicted control points from the detection branch and rectifies the feature maps, aligning curved text into a straight representation for the recognition module. Crucially, ARM is differentiable, allowing gradients from the recognition loss to flow back to the detection branch.
  • Rectification Control Points Detection (RCPD): The detection branch of ARTS uses a one-stage, anchor-free approach to densely predict control points that define the shape of text lines. This is a departure from methods that rely solely on bounding boxes.
  • Joint Optimization: The key training strategy is to jointly optimize both the detection and recognition losses. By using the differentiable ARM, the recognition loss (Lrec) can be back-propagated to the RCPD head. This means the detection branch is not only trained to produce accurate bounding boxes (Ldet) but also to produce boxes that are more suitable for recognition.
  • Training Strategy: Instead of solely relying on ground-truth annotations for feature extraction during training, ARTS uses predicted detection results. This further helps in aligning the training process with the inference scenario.

Datasets

The paper evaluates ARTS on several challenging datasets:

  • Total-Text: A benchmark for arbitrarily-shaped text spotting.
  • CTW1500: Another dataset focused on long, curved text instances.
  • ICDAR2015: A dataset for multi-oriented text spotting.

For pre-training, a mixed dataset including SynthText150k, Total-Text, and MLT was used.

Key Insights and Ideas

  • The "Inconsistency Problem" is Real and Quantifiable: The paper provides experimental evidence (Figure 1) showing how feature distributions shift between training and testing and how detection quality (IoU) doesn't perfectly correlate with recognition accuracy. This formalizes a long-suspected issue.
  • Recognition Loss is Valuable for Detection: The most significant insight is that allowing recognition loss to influence the detection branch dramatically improves performance. This suggests that the "ideal" detection box for recognition might differ from the one optimized purely for IoU.
  • Differentiable Rectification is Key: The ARM's differentiability is crucial. It enables the direct flow of information from recognition back to detection, allowing for a more holistic optimization of the entire text spotting pipeline.
  • Arbitrarily-Shaped Text Benefits Greatly: The method's ability to handle curved text and rectify it before recognition is a major advantage, leading to substantial gains on datasets like Total-Text and CTW1500.

Key Results

  • State-of-the-Art Performance: ARTS-S (ResNet50) achieves an end-to-end text spotting F-measure of 77.1% on Total-Text, outperforming previous methods by a significant margin (e.g., 7.6 points over ABCNet-MS).
  • Competitive Speed: The ARTS-S model achieves this accuracy at 10.5 FPS, demonstrating a strong trade-off between accuracy and inference speed.
  • Real-time Capabilities: A real-time version (ARTS-RT) achieves 65.9% F-measure at 28.0 FPS, which is faster than previous state-of-the-art methods.
  • Effectiveness of ARM: Ablation studies show that using the ARM and enabling recognition loss back-propagation leads to substantial improvements in both detection and end-to-end F-measures compared to methods that don't allow this feedback.

Limitations and Future Work

  • Computational Cost of ARM: While ARTS is efficient, the ARM still adds some computational overhead. The paper shows that using a parallel recognizer significantly reduces the time cost for recognition, making the overall system faster.
  • Future Work: The paper implicitly suggests that further exploration of how to best leverage recognition feedback for detection could lead to even more robust text spotting systems. The current work focuses on arbitrarily-shaped text, and extending these principles to other challenging text scenarios could be a future direction.

Summary updated

Abstract

Recent approaches for end-to-end text spotting have achieved promising results. However, most of the current spotters were plagued by the inconsistency problem between text detection and recognition. In this work, we introduce and prove the existence of the inconsistency problem and analyze it from two aspects: (1) inconsistency of text recognition features between training and testing, and (2) inconsistency of optimization targets between text detection and recognition. To solve the aforementioned issues, we propose a differentiable Auto-Rectification Module (ARM) together with a new training strategy to enable propagating recognition loss back into detection branch, so that our detection branch can be jointly optimized by detection and recognition targets, which largely alleviates the inconsistency problem between text detection and recognition. Based on these designs, we present a simple yet robust end-to-end text spotting framework, termed Auto-Rectification Text Spotter (ARTS), to detect and recognize arbitrarily-shaped text in natural scenes. Extensive experiments demonstrate the superiority of our method. In particular, our ARTS-S achieves 77.1% end-to-end text spotting F-measure on Total-Text at a competitive speed of 10.5 FPS, which significantly outperforms previous methods in both accuracy and inference speed.
BibTeX
@misc{zhong2021arts,
  author = {Humen Zhong and Jun Tang and Wenhai Wang and Zhibo Yang and Cong Yao and Tong Lu},
  title = {{ARTS: Eliminating Inconsistency between Text Detection and Recognition with Auto-Rectification Text Spotter}},
  year = {2021},
  eprint = {2110.10405},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.10405},
}
Figure