Pyramid Mask Text Detector

Jingchao Liu, Xuebo Liu, Jie Sheng, Ding Liang, Xin Li, Qingjie LiuarXiv 2019 · arXiv

Abstract

The Pyramid Mask Text Detector (PMTD) advances scene text detection by moving beyond simple binary masks. By supervising the network to predict "soft" pyramid-shaped masks and using a novel 3D plane clustering algorithm to fit text boxes, PMTD achieves state-of-the-art accuracy and shows remarkable robustness to imprecise bounding box predictions.

Introduction

Detecting text in natural scenes—such as street signs, product labels, or logos—is a fundamental challenge in computer vision due to variations in orientation, scale, and environmental clutter. While instance segmentation frameworks like Mask R-CNN have been adapted for this task, they often suffer from three critical flaws: over-simplified binary supervision, imprecise manual labels, and error propagation from initial bounding box proposals. PMTD addresses these issues by redefining the mask prediction task as a geometric regression problem.

Methodology

PMTD introduces two primary innovations to the standard Mask R-CNN architecture: Soft Pyramid Masks and 3D Plane Clustering.

Soft Pyramid Masks

Unlike traditional methods that use binary (0/1) masks, PMTD employs "soft" masks where each pixel is assigned a value between 0 and 1 based on its position within the text region. Imagine a 3D pyramid sitting on top of the text: the center of the text region (the apex) has a value of 1, while the values decrease linearly toward 0 at the boundaries. This location-aware supervision forces the network to learn the internal geometric structure of the text, making it significantly more resilient to noisy boundary labels.

PMTD detection results on multi-oriented and logo-style text.

3D Plane Clustering

During inference, the predicted soft mask is treated as a 3D point cloud, where the pixel's confidence score serves as the $z$-coordinate. PMTD then applies a Plane Clustering algorithm to fit four planes representing the sides of the pyramid using Robust Least Squares. The final quadrilateral text box is derived from the intersection of these planes with the base plane ($z=0$).

Because this algorithm considers the "mass" of the entire soft mask rather than just the boundary pixels, it can regress an accurate text box even if the initial bounding box proposal was poorly placed. To further enhance performance, the authors replaced the standard mask head's initial convolution layers with dilated convolutions, expanding the receptive field to capture more global context.

Evaluation

The PMTD framework was evaluated on three major benchmarks: ICDAR 2017 MLT, ICDAR 2015, and ICDAR 2013.

Results

PMTD established new state-of-the-art performance metrics at the time of publication:

  • ICDAR 2017 MLT: Achieved an F-measure of 80.13%, demonstrating strong performance across multi-lingual and multi-script environments.
  • ICDAR 2015: Reached an F-measure of 89.33%.
  • High-Precision Localization: The model's superior localization was most evident at strict evaluation thresholds. On ICDAR 2015, PMTD showed a 30.93% relative improvement in F-measure over the baseline at a strict IoU threshold of 0.9.

Conclusion

By treating classifier confidence as a geometric coordinate, PMTD transforms text detection from a simple classification task into a robust regression problem. While the current implementation assumes a quadrilateral geometry, the "soft mask" concept provides a foundation for future work targeting more complex, curved text shapes through more sophisticated 3D modeling.

Summary updated

BibTeX
@misc{liu2019pyramid,
  author = {Jingchao Liu and Xuebo Liu and Jie Sheng and Ding Liang and Xin Li and Qingjie Liu},
  title = {{Pyramid Mask Text Detector}},
  year = {2019},
  eprint = {1903.11800},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1903.11800},
}
Figure