Deformable Kernel Expansion Model for Efficient Arbitrary-shaped Scene Text Detection

Tao He, Sheng Huang, Wenhao Tang, Bo LiuarXiv 2023 · OpenReview

TL;DR

This paper introduces the Deformable Kernel Expansion (DKE) model, a novel approach for detecting arbitrary-shaped text in images. DKE combines the strengths of segmentation-based methods for initial text region identification and contour-based methods for precise boundary refinement, achieving a strong balance between accuracy and efficiency.

What Problem Does This Paper Address?

Detecting text in natural images (scene text detection) is a notoriously difficult computer vision problem. This is primarily due to the wide variety of text shapes (from horizontal to curved and even irregular), scales, and aspect ratios encountered in real-world scenes. Existing methods often fall into two camps: segmentation-based approaches that excel at modeling arbitrary shapes but require complex post-processing, and contour-based methods that are faster but can struggle with initial boundary accuracy. This paper aims to bridge this gap by developing a method that is both accurate in handling diverse text shapes and efficient in its inference speed.

Methods

The DKE model is built upon two core modules:

  • Text Kernel Generation (TKG): This module leverages a segmentation-based approach to identify a "text kernel." This kernel is essentially a shrunken version of the text region, acting as a robust initial boundary that is less susceptible to background noise and shape variations compared to directly segmenting the full text region.
  • Deformable Contour Expansion (DCE): This module takes the contour of the text kernel and expands it to form the final text boundary. Instead of complex pixel-level post-processing, DCE uses a learnable regression network to predict vertex-wise offsets. A key innovation here is the Optimal Bipartite Graph Matching Loss (OBGML). This loss function intelligently pairs predicted kernel contour vertices with ground truth boundary vertices by formulating it as a bipartite graph matching problem, ensuring a globally optimal alignment and minimizing deformation cost. This approach allows for a single-step, efficient contour deformation.

Datasets

The paper uses the following datasets for training and evaluation:

  • SynthText: Used for pre-training the model.
  • MLT-2017: Used for pre-training the model.
  • ICDAR2015: A dataset for evaluating quadrangular text detection.
  • MSRA-TD500: A multi-language dataset for evaluating text-line detection.
  • Total-Text: A dataset featuring curved text instances.
  • CTW1500: Another dataset for curved text instances.

Key Insights and Ideas

  • Text Kernel as a Robust Initial Boundary: Using a shrunken "text kernel" from segmentation provides a more stable and less noisy starting point for contour deformation compared to directly using the full text region or its boundary.
  • Single-Step Contour Expansion: The DCE module's ability to deform the kernel contour to the final boundary in a single, learnable step significantly speeds up inference compared to iterative refinement methods.
  • Optimal Bipartite Graph Matching Loss (OBGML): This novel loss function is crucial for accurate contour deformation. By treating vertex pairing as a global optimization problem, it ensures better alignment and prevents issues like vertex mismatch or loss of shape information that can plague simpler matching strategies.
  • Trade-off Between Accuracy and Efficiency: The DKE model demonstrates that it's possible to achieve state-of-the-art accuracy on challenging datasets while maintaining competitive inference speeds, making it suitable for real-world applications.

Key Results

  • DKE achieves a good trade-off between accuracy and efficiency on various benchmarks.
  • The OBGML loss contributes to performance gains over simpler contour deformation losses (DML, NNML).
  • The model demonstrates competitive or superior performance compared to other top methods on datasets like Total-Text and CTW1500. For instance, on Total-Text, DKE achieves competitive efficiency with DBNet++ but outperforms it by 2.6% and 1.9% in F-measure (with ResNet18 and ResNet50 backbones, respectively).
  • The model achieves state-of-the-art results on MSRA-TD500 and shows strong performance on ICDAR2015.
  • The proposed method offers a significant speedup (e.g., 2.4x faster than TextBPN) while maintaining comparable or better accuracy.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the OBGML is effective, it relies on accurate kernel generation. If the initial segmentation of the text kernel is poor, the subsequent contour expansion might also be affected. The paper also notes that while the model is efficient, further optimization for extremely low-latency applications might be explored.
  • Future Work: The authors suggest that their method can be further improved by exploring more advanced segmentation backbones or incorporating attention mechanisms for better feature representation. Additionally, extending the framework to handle more complex text scenarios, such as very dense text or text with extreme distortions, could be a future direction.

Summary updated

Abstract

Scene text detection is a challenging computer vision task due to the high variation in text shapes and ratios. In this work, we propose a scene text detector named Deformable Kernel Expansion (DKE), which incorporates the merits of both segmentation and contour-based detectors. DKE employs a segmentation module to segment the shrunken text region as the text kernel, then expands the text kernel contour to obtain text boundary by regressing the vertex-wise offsets. Generating the text kernel by segmentation enables DKE to inherit the arbitrary-shaped text region modeling capability of segmentation-based detectors. Regressing the kernel contour with some sampled vertices enables DKE to avoid the complicated pixel-level post-processing and better learn contour deformation as the contour-based detectors. Moreover, we propose an Optimal Bipartite Graph Matching Loss (OBGML) that measures the matching error between the predicted contour and the ground truth, which efficiently minimizes the global contour matching distance. Extensive experiments on CTW1500, Total-Text, MSRA-TD500, and ICDAR2015 demonstrate that DKE achieves a good tradeoff between accuracy and efficiency in scene text detection.
BibTeX
@misc{he2023deformable,
  author = {Tao He and Sheng Huang and Wenhao Tang and Bo Liu},
  title = {{Deformable Kernel Expansion Model for Efficient Arbitrary-shaped Scene Text Detection}},
  year = {2023},
  eprint = {2303.15737},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.15737},
}

Loading related content...

Figure