Fused Text Segmentation Networks for Multi-oriented Scene Text Detection

Yuchen Dai, Zheng Huang, Yuting Gao, Youxuan Xu, Kai Chen, Jie Guo, Weidong Qiu

Abstract

In this paper, we introduce a novel end-to-end framework for multi-oriented scene text detection from an instance-aware semantic segmentation perspective. We present Fused Text Segmentation Networks (FTSN), which combine multi-level features during feature extraction, as text instances may rely on finer feature expression compared to general objects. It detects and segments text instances jointly and simultaneously, leveraging merits from both semantic segmentation and region proposal-based object detection tasks. Without involving any extra pipelines, our approach surpasses current state-of-the-art benchmarks: ICDAR2015 Incidental Scene Text and MSRA-TD500, reaching Hmean scores of 84.1% and 82.0%, respectively. Moreover, we report a baseline on Total-Text containing curved text, suggesting the effectiveness of the proposed approach.

Introduction

Detecting text in natural scenes is a complex task due to extreme variations in orientation, font, and scale, often compounded by challenging backgrounds and lighting. Traditional object detection methods typically rely on axis-aligned bounding boxes, which are ill-suited for the multi-oriented or tilted text common in "the wild." While rotated quadrilateral methods exist, they often require complex, multi-stage pipelines that are difficult to optimize.

Furthermore, text instances are often smaller and more detailed than general objects like cars or pedestrians. Standard deep learning backbones may discard the fine-grained feature representations necessary for accurate character-level distinction. To address these challenges, we propose Fused Text Segmentation Networks (FTSN), which treat text detection as an instance-aware semantic segmentation problem.

Methodology

The FTSN framework leverages the strengths of both region-proposal-based detection and semantic segmentation within an end-to-end trainable pipeline.

Feature Extraction and Fusion

Using a ResNet-101 backbone, the model extracts features at multiple stages. A key innovation is the fusion of low-level features (capturing fine details and textures) with high-level semantic features. This "FusedMap" ensures the network maintains the resolution required to distinguish small text characters from noisy backgrounds.

Joint Prediction and Mask-NMS

Built upon the Fully Convolutional Instance-aware Semantic Segmentation (FCIS) architecture, the model simultaneously predicts a classification score, a pixel-level mask, and a bounding box for each text instance.

The FTSN architecture, highlighting the feature fusion process and the joint prediction of classification, masks, and boxes.

To handle the overlap issues inherent in multi-oriented text, the authors introduce Mask-NMS. Unlike standard NMS which uses bounding box Intersection over Union (IoU), Mask-NMS utilizes Maximum-Intersection (MMI) based on actual pixel masks. This prevents the erroneous suppression of valid, closely-packed lines of tilted text.

Experimental Results

The model was evaluated on several challenging benchmarks, including ICDAR 2015 (incidental text), MSRA-TD500 (multi-lingual line-level text), and Total-Text (curved text).

Qualitative results showing the original images, predicted pixel masks, and final bounding boxes for multi-oriented and multi-lingual text.

FTSN achieved state-of-the-art performance at the time of publication:

  • ICDAR 2015: Reached an Hmean (F1-score) of 84.1%, outperforming previous methods like EAST and SegLink.
  • MSRA-TD500: Achieved an Hmean of 82.0%.
  • Curved Text: On the Total-Text dataset, the model provided a strong baseline (81.3% Hmean), demonstrating that a segmentation-based approach naturally handles curved text better than traditional box-regression methods.

Conclusion

Fused Text Segmentation Networks provide a robust solution for multi-oriented scene text detection by framing the problem as instance-aware segmentation. By fusing multi-level features and employing Mask-NMS, the system accurately handles diverse text shapes and orientations. Future work will focus on optimizing the architecture for real-time performance and improving segmentation in extremely dense text environments.

Summary updated

Abstract

In this paper, we introduce a novel end-end framework for multi-oriented scene text detection from an instance-aware semantic segmentation perspective. We present Fused Text Segmentation Networks, which combine multi-level features during the feature extracting as text instance may rely on finer feature expression compared to general objects. It detects and segments the text instance jointly and simultaneously, leveraging merits from both semantic segmentation task and region proposal based object detection task. Not involving any extra pipelines, our approach surpasses the current state of the art on multi-oriented scene text detection benchmarks: ICDAR2015 Incidental Scene Text and MSRA-TD500 reaching Hmean 84.1% and 82.0% respectively. Morever, we report a baseline on total-text containing curved text which suggests effectiveness of the proposed approach.
BibTeX
@misc{dai2017fused,
  author = {Yuchen Dai and Zheng Huang and Yuting Gao and Youxuan Xu and Kai Chen and Jie Guo and Weidong Qiu},
  title = {{Fused Text Segmentation Networks for Multi-oriented Scene Text Detection}},
  year = {2017},
  eprint = {1709.03272},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1709.03272},
}
Figure