Predictive Ensemble Learning with Application to Scene Text Detection

Danlu Chen, Xu-Yao Zhang, Wei Zhang, Yao Lu, Xiuli Li, Tao MeiarXiv 2019 · arXiv

Introduction

Ensemble learning has long been a cornerstone of robust machine learning, typically involving the combination of multiple models to improve overall performance. In simple classification tasks, this is often achieved through straightforward techniques like majority voting or averaging probability scores. However, for "complex tasks" such as scene text detection, speech recognition, or machine translation, the outputs are structured and variable-length, making traditional fusion methods like Non-Maximum Suppression (NMS) both computationally expensive and prone to performance degradation.

This paper introduces Predictive Ensemble Learning (PEL), a novel framework that reframes the ensemble problem. Instead of attempting to merge disparate and often conflicting outputs, PEL employs a "selector" neural network to predict which single model in an ensemble is best suited for a given input. By treating ensemble learning as a classification problem, PEL achieves state-of-the-art results while maintaining high inference efficiency.

The Limitations of Output Fusion

In scene text detection, different models—such as anchor-based, segmentation-based, or instance-segmentation-based architectures—produce bounding boxes with varying shapes, orientations, and confidence levels. The standard approach to ensembling these models is to aggregate all predicted boxes and apply NMS to filter overlaps.

The authors argue that this "lazy" fusion is fundamentally flawed. It frequently introduces a high volume of false positives and can lead to "downhill" performance, where the ensemble performs worse than its best individual component. Furthermore, executing every model in an ensemble for every inference request is prohibitively slow for real-time applications.

The PEL Framework

The PEL approach consists of a structured four-stage pipeline designed to identify and utilize the "expert" model for any specific scene.

The four stages of the Predictive Ensemble Learning (PEL) framework: (a) Base model training, (b) Evaluation/Labeling, (c) Selector training, and (d) Two-stage testing.
  1. Base Model Training: A diverse pool of $K$ expert models is trained. Diversity is encouraged by using different training data, augmentations, architectures, and loss functions.
  2. Evaluation and Labeling: All base models are run against a training dataset. For each image $x_i$, the model that achieves the highest F-score is identified. This generates a new dataset where the "label" $y_i$ is the index of the best-performing model.
  3. Selector Training: A dedicated classification network, the "Selector," is trained on this new dataset. The authors utilized a VGG16 backbone for the selector. Since multiple models might perform equally well on a single image, this is treated as a multi-label classification task.
  4. Two-stage Inference: During testing, the Selector first analyzes the image to pick the optimal model index ($idx$). Subsequently, only that specific base model is executed to generate the final output.

Key Insights: Selection Over Fusion

The primary innovation of PEL is the shift from output fusion to model selection. This bypasses the geometric complexities of merging rotated or irregular bounding boxes.

Model Complementarity

Different model architectures possess distinct "blind spots" and strengths. For instance, a segmentation-based model might excel at detecting curved text, while an anchor-based model is more reliable for long, horizontal text lines. PEL leverages these specific strengths by learning the domain expertise of each model.

Visualization of model complementarity, showing how different models specialize in different visual domains or text styles.

Efficiency and the "Oracle" Gap

Unlike traditional ensembles that require $K$ full model passes, PEL only requires the overhead of a lightweight selector plus a single base model pass. This significantly reduces the computational footprint.

However, the authors note an "Oracle" gap. While PEL outperforms individual models, there is still a performance difference between the current selector (which achieves ~80% accuracy) and a theoretical "Oracle" that always picks the perfect model. This suggests that the primary bottleneck for further improvement lies in the classification accuracy of the selector network itself.

Experimental Results

The researchers evaluated PEL against three major benchmarks: ICDAR 2013, ICDAR 2015, and SCUT-FORU. The base model pool included SSTD, PixelLink, and Mask TextSpotter.

  • ICDAR 2013: PEL achieved an F-score of 92.0, surpassing previous state-of-the-art methods like FEN (91.8) and vastly outperforming traditional NMS fusion (82.7).
  • ICDAR 2015: PEL reached an F-score of 85.5, consistently beating all individual base models.
  • SCUT-FORU: The framework achieved an F-score of 83.2, demonstrating robustness across English and Chinese text datasets.

In every benchmark, PEL successfully avoided the performance degradation often seen with NMS fusion, proving that "choosing the right tool" is more effective than "merging all tools."

Conclusion and Future Work

Predictive Ensemble Learning offers a scalable and efficient alternative to traditional ensemble methods for complex tasks. By transforming the problem into one of expert selection, it achieves superior accuracy and faster inference. While currently applied to scene text detection, the authors suggest that PEL could be equally transformative for other domains with structured outputs, such as speech recognition and machine translation. Future research will likely focus on improving selector accuracy through more advanced architectures like ResNet or attention-based models.

Summary updated

Abstract

Deep learning based approaches have achieved significant progresses in different tasks like classification, detection, segmentation, and so on. Ensemble learning is widely known to further improve performance by combining multiple complementary models. It is easy to apply ensemble learning for classification tasks, for example, based on averaging, voting, or other methods. However, for other tasks (like object detection) where the outputs are varying in quantity and unable to be simply compared, the ensemble of multiple models become difficult. In this paper, we propose a new method called Predictive Ensemble Learning (PEL), based on powerful predictive ability of deep neural networks, to directly predict the best performing model among a pool of base models for each test example, thus transforming ensemble learning to a traditional classification task. Taking scene text detection as the application, where no suitable ensemble learning strategy exists, PEL can significantly improve the performance, compared to either individual state-of-the-art models, or the fusion of multiple models by non-maximum suppression. Experimental results show the possibility and potential of PEL in predicting different models' performance based only on a query example, which can be extended for ensemble learning in many other complex tasks.
BibTeX
@misc{chen2019predictive,
  author = {Danlu Chen and Xu-Yao Zhang and Wei Zhang and Yao Lu and Xiuli Li and Tao Mei},
  title = {{Predictive Ensemble Learning with Application to Scene Text Detection}},
  year = {2019},
  eprint = {1905.04641},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.04641},
}
Figure