MR-FPN: Multi-Level Residual Feature Pyramid Text Detection Network Based on Self-Attention Environment

Jianjun Kang, Mayire Ibrayim, Askar HamdullaSensors 2022 · DOI

TL;DR

This paper introduces MR-FPN, a novel deep learning network for detecting text in complex scenes. By enhancing a Feature Pyramid Network (FPN) with multi-level residuals and a self-attention module, MR-FPN significantly improves the accuracy and robustness of text detection, especially for challenging cases like adjacent or curved text.

What Problem Does This Paper Address?

Detecting text in natural scenes is a surprisingly difficult task for computers. Images often contain text with diverse shapes, sizes, and orientations, making it hard for algorithms to accurately locate and distinguish individual text instances. Existing methods struggle with:

  • Complex backgrounds: Text can blend in with its surroundings, leading to false positives or missed detections.
  • Adjacent text: Differentiating between closely spaced text instances is a common failure point.
  • Arbitrary text shapes: Many algorithms are optimized for straight text and fail on curved or irregularly shaped text.
  • Information loss in deep networks: As information passes through multiple layers of a deep neural network, fine-grained details crucial for text detection can be lost.

This paper aims to tackle these challenges by developing a more robust and accurate text detection system.

Methods

The proposed MR-FPN builds upon the PSENet architecture, introducing several key improvements:

  • Multi-Level Residual Feature Pyramid Network (MR-FPN): This is the core innovation. It enhances the standard Feature Pyramid Network (FPN) by incorporating multi-level residuals. This helps to mitigate the loss of semantic information that occurs during the top-down propagation of features in FPNs, ensuring that richer contextual information is preserved.
  • Self-Attention Module (SAM): A novel self-attention module is introduced. This module combines channel and spatial attention to better capture long-range dependencies within the feature maps. It adaptively focuses on important textual features while suppressing irrelevant background information, thereby improving feature representation and reducing noise.
  • Multi-Scale Enhancement Module (MEM): This module is designed to extract and fuse rich semantic information from both global and local perspectives. It leverages features from different scales to create more detailed representations, further enhancing the network's ability to detect text of varying sizes and complexities.

The overall architecture consists of a backbone (ResNet50), the MR-FPN for feature fusion, and a progressive scale expansion network for segmentation.

Datasets

The MR-FPN was evaluated on four challenging public datasets:

  • CTW1500: A dataset specifically designed for long-curved text detection, using polygonal annotations.
  • ICDAR2015: A widely used dataset featuring street view elements, primarily straight text, with challenges like blur and complex backgrounds.
  • Total-Text: A dataset containing natural scene text in English, with a focus on horizontal, multi-directional, and curved text.
  • MSRA-TD500: A multilingual dataset with natural scene images, known for its challenging text and background complexity.

The paper does not introduce new benchmark datasets.

Key Insights and Ideas

  • The power of multi-level residuals: The paper highlights how residual connections, when applied across multiple levels of a feature pyramid, can effectively combat the information degradation inherent in top-down feature propagation, leading to more informative features for detection.
  • Self-attention for context: The integration of a self-attention mechanism proves crucial for enhancing contextual understanding. By allowing the network to weigh the importance of different feature locations and channels, it can better isolate text from complex backgrounds and resolve ambiguities between adjacent text instances.
  • Fusion is key: The experiments demonstrate that combining multi-scale features, residual connections, and attention mechanisms leads to synergistic improvements, outperforming individual module contributions.

Key Results

The MR-FPN achieved state-of-the-art or competitive results on the evaluated datasets.

  • On the Total-Text dataset, the proposed method achieved an F-measure of 83.31%, outperforming the baseline PSENet by 5.01% and TextSnake by 4.81%.
  • On the CTW1500 dataset, MR-FPN achieved an F-measure of 80.65%, a significant improvement over the baseline.
  • On ICDAR2015, the model achieved an F-measure of 80.92%, showing good performance on multi-oriented text.
  • On MSRA-TD500, the F-measure was 78.24%, demonstrating its capability on multilingual text.

Ablation studies confirmed the effectiveness of each proposed module (MEM, SAM, MR-FPN), with the combined MR-FPN + SAM + MEM achieving the best overall performance.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Environmental Impact: Images in datasets like ICDAR2015 are heavily influenced by environmental factors (blur, lighting, distance). The model's performance can be affected by these conditions, and future work should focus on mitigating these effects.
  • Small Target Detection: While robust for close-up text, the model shows weaknesses in detecting distant text due to smaller text areas and weaker features. Enhancing feature extraction for small targets is a key area for improvement.
  • Computational Cost: The increased depth of the model to incorporate these enhancements has led to higher computational time. Future work will focus on optimizing the model for efficiency without sacrificing accuracy.
  • Image Compression: The paper notes that text can be distorted by image compression, and future research could explore better handling of such scenarios.

Summary updated

BibTeX
@article{kang2022mr,
  author = {Jianjun Kang and Mayire Ibrayim and Askar Hamdulla},
  title = {{MR-FPN: Multi-Level Residual Feature Pyramid Text Detection Network Based on Self-Attention Environment}},
  journal = {Sensors},
  year = {2022},
  doi = {10.3390/s22093337},
}
Figure