Abstract
In this paper, we introduce a novel end-to-end framework for multi-oriented scene text detection from an instance-aware semantic segmentation perspective. We present Fused Text Segmentation Networks (FTSN), which combine multi-level features during feature extraction, as text instances may rely on finer feature expression compared to general objects. It detects and segments text instances jointly and simultaneously, leveraging merits from both semantic segmentation and region proposal-based object detection tasks. Without involving any extra pipelines, our approach surpasses current state-of-the-art benchmarks: ICDAR2015 Incidental Scene Text and MSRA-TD500, reaching Hmean scores of 84.1% and 82.0%, respectively. Moreover, we report a baseline on Total-Text containing curved text, suggesting the effectiveness of the proposed approach.
Introduction
Detecting text in natural scenes is a complex task due to extreme variations in orientation, font, and scale, often compounded by challenging backgrounds and lighting. Traditional object detection methods typically rely on axis-aligned bounding boxes, which are ill-suited for the multi-oriented or tilted text common in "the wild." While rotated quadrilateral methods exist, they often require complex, multi-stage pipelines that are difficult to optimize.
Furthermore, text instances are often smaller and more detailed than general objects like cars or pedestrians. Standard deep learning backbones may discard the fine-grained feature representations necessary for accurate character-level distinction. To address these challenges, we propose Fused Text Segmentation Networks (FTSN), which treat text detection as an instance-aware semantic segmentation problem.
Methodology
The FTSN framework leverages the strengths of both region-proposal-based detection and semantic segmentation within an end-to-end trainable pipeline.
Feature Extraction and Fusion
Using a ResNet-101 backbone, the model extracts features at multiple stages. A key innovation is the fusion of low-level features (capturing fine details and textures) with high-level semantic features. This "FusedMap" ensures the network maintains the resolution required to distinguish small text characters from noisy backgrounds.
Joint Prediction and Mask-NMS
Built upon the Fully Convolutional Instance-aware Semantic Segmentation (FCIS) architecture, the model simultaneously predicts a classification score, a pixel-level mask, and a bounding box for each text instance.
To handle the overlap issues inherent in multi-oriented text, the authors introduce Mask-NMS. Unlike standard NMS which uses bounding box Intersection over Union (IoU), Mask-NMS utilizes Maximum-Intersection (MMI) based on actual pixel masks. This prevents the erroneous suppression of valid, closely-packed lines of tilted text.
Experimental Results
The model was evaluated on several challenging benchmarks, including ICDAR 2015 (incidental text), MSRA-TD500 (multi-lingual line-level text), and Total-Text (curved text).
FTSN achieved state-of-the-art performance at the time of publication:
- ICDAR 2015: Reached an Hmean (F1-score) of 84.1%, outperforming previous methods like EAST and SegLink.
- MSRA-TD500: Achieved an Hmean of 82.0%.
- Curved Text: On the Total-Text dataset, the model provided a strong baseline (81.3% Hmean), demonstrating that a segmentation-based approach naturally handles curved text better than traditional box-regression methods.
Conclusion
Fused Text Segmentation Networks provide a robust solution for multi-oriented scene text detection by framing the problem as instance-aware segmentation. By fusing multi-level features and employing Mask-NMS, the system accurately handles diverse text shapes and orientations. Future work will focus on optimizing the architecture for real-time performance and improving segmentation in extremely dense text environments.
Summary updated