Ensemble Learning for Vietnamese Scene Text Spotting in Urban Environments

Hieu Nguyen, Cong-Hoang Ta, Phuong-Thuy Le-Nguyen, Minh-Triet Tran, Trung-Nghia LearXiv 2024 · DOI · arXiv

TL;DR

This paper introduces a simple yet effective ensemble learning framework designed to tackle the unique challenges of Vietnamese scene text spotting in messy urban environments. By strategically combining the outputs of multiple state-of-the-art models, the authors achieved a significant 5% boost in accuracy on the VinText benchmark, demonstrating that the "wisdom of the crowd" can effectively overcome the diacritic-heavy complexities of the Vietnamese script.

What Problem Does This Paper Address?

Scene text spotting—the simultaneous detection and recognition of text in natural images—is a cornerstone of technologies like autonomous driving and intelligent navigation. However, Vietnamese presents a specific set of hurdles. The language uses a rich set of diacritics that create visually similar characters (e.g., ‘a’ vs. ‘ă’ or ‘o’ vs. ‘ô’), which often trip up individual OCR models.

Furthermore, urban environments in Vietnam are visually "noisy." Text is frequently obscured by trees, power lines, or traffic signs, and subjected to harsh perspective shifts and variable lighting. Existing single-model solutions often struggle to maintain robustness across these diverse and occluded contexts.

Fig. 1: Scene text spotting in Vietnamese urban envi- ronments poses various challenges, such as obscured by trees and perspective-shifted.

Methods

The authors propose an ensemble framework that aggregates predictions from several base models to mitigate individual weaknesses. The workflow consists of three main stages:

  1. Data Converter: Raw images from the VinText dataset are transformed into formats suitable for different base models (e.g., cropping images into single-word patches for recognition-only models).
  2. Base Models: The framework integrates both two-stage (separate detection and recognition) and end-to-end (joint detection and recognition) architectures. Models used include DB++, SAST, and EAST for detection, and SPIN, ABINet, and SRN for recognition.
  3. Meta-Model (The Merger): This is the core of the ensemble. It uses an Intersection over Union (IoU) logic to combine results:
  4. Non-overlapping boxes: Simply combined into the final set.
  5. Overlapping boxes: If the IoU $\ge$ 0.5, the boxes are merged (using the intersection). If the IoU < 0.5, they are treated as distinct words, and the overlapping region is subtracted to keep them separate.
The proposed ensemble learning workflow.

Datasets

  • VinText: The primary benchmark, containing 56,000 text instances across 2,000 images of Vietnamese urban scenes.
  • Pre-training Datasets: The models were pre-trained on standard large-scale datasets including ICDAR 2015, Total-Text, MJSynth, and SynthText before being fine-tuned on VinText.

Key Insights and Ideas

  • Complementary Strengths: The authors found that different models fail in different ways. For instance, DB++ excels at high precision, while EAST is better at recall. By ensembling them, the framework captures text that individual models might miss or misidentify.
  • Selective Ensembling: A key "aha" moment was realizing that more models isn't always better. Adding too many models or including those with high overlap errors (like RobustScanner in certain configurations) actually degraded performance. The most effective ensemble was a carefully curated pair of high-performing model combinations.
  • The Power of Fine-Tuning: The results underscored that while pre-training on generic English datasets is helpful, fine-tuning on language-specific data (VinText) is non-negotiable for achieving high accuracy in Vietnamese.

Key Results

The ensemble approach yielded impressive quantitative gains on the VinText dataset:

  • Accuracy Boost: The best ensemble configuration (combining DB++/SPIN and SAST/ABINet) achieved a Character Accuracy of 58.75% and an F-measure of 65.21%, representing a roughly 5% improvement over the best individual model pairs.
  • Qualitative Improvement: The ensemble successfully resolved instances where individual models were confused by diacritics or partial occlusions.
Visualization showing the ensemble method (c) correctly identifying text where individual models (a, b) failed.

Limitations and Future Work

While effective, the authors acknowledge two main drawbacks:

  1. Computational Complexity: Running multiple base models in parallel significantly increases the inference time and resource requirements, making real-time deployment challenging.
  2. Suboptimal Pairings: Not all model combinations are synergistic; some can introduce noise that lowers the overall F-measure.

Future Directions: The team plans to investigate methods to reduce the computational overhead of the ensemble and explore post-processing techniques, such as language-model-based spelling correction, to further refine word recognition accuracy.

Summary updated

Abstract

This paper presents a simple yet efficient ensemble learning framework for Vietnamese scene text spotting. Leveraging the power of ensemble learning, which combines multiple models to yield more accurate predictions, our approach aims to significantly enhance the performance of scene text spotting in challenging urban settings. Through experimental evaluations on the VinText dataset, our proposed method achieves a significant improvement in accuracy compared to existing methods with an impressive accuracy of 5%. These results unequivocally demonstrate the efficacy of ensemble learning in the context of Vietnamese scene text spotting in urban environments, highlighting its potential for real world applications, such as text detection and recognition in urban signage, advertisements, and various text-rich urban scenes.
BibTeX
@misc{nguyen2024ensemble,
  author = {Hieu Nguyen and Cong-Hoang Ta and Phuong-Thuy Le-Nguyen and Minh-Triet Tran and Trung-Nghia Le},
  title = {{Ensemble Learning for Vietnamese Scene Text Spotting in Urban Environments}},
  year = {2024},
  eprint = {2404.00852},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.00852},
}
Figure